Meta's Code Llama project has moved from a specialized Llama 2 code model into a stronger family of tools for software generation, debugging and completion. The latest release, Code Llama 70B, gives developers and researchers a larger open model aimed directly at programming work.
Code Llama 70B Changes the Benchmark Story
Meta has released Code Llama 70B as a powerful open-source LLM for code generation. It comes in two variants: CodeLlama-70B-Python and CodeLlama-70B-Instruct.
The headline result is the benchmark score. The 70B-instruct-version scored 67.8 on HumanEval, ahead of GPT-4 and Gemini Pro for prompts without examples, also described as zero-shot. The first version of Code Llama achieved up to 48.8 points.
That matters because HumanEval is one of the benchmark names used in the source article to compare coding models. In this case, the result places Code Llama 70B in a different position from the earlier Code Llama 34B story, where the model was described as being on par with GPT-3.5 but far behind GPT-4 in Human Eval.
Meta says Code Llama 70B is suitable for both research and commercial projects, with the usual Llama licenses applying. The article also notes that Code Llama 70B and other Llama models are available after asking Meta, with more information available on Github.
What Code Llama Was Built to Do
Code Llama began as Meta's refined Llama 2 variant for code generation. According to Meta, it is an evolution of Llama 2 trained further on 500 billion code tokens and code-related tokens from Llama 2's code-specific datasets.
The basic purpose is straightforward: improve programming tasks compared with a general Llama 2 model. Code Llama can generate code from natural language prompts, complete code and debug code. The source gives an example prompt: "Write me a function that outputs the Fibonacci sequence."
The model supports popular programming languages including Python, C++, Java, PHP, Typescript, C#, Bash and others. That breadth is important because code assistants are useful only if they can work across the practical mix of languages developers already use.
In plain terms, Code Llama is not presented as a general chatbot with some coding ability. It is a model family trained and released around software tasks, with variants aimed at different developer needs.
Model Sizes, Context and Variants
The original Code Llama release included three sizes: 7 billion, 13 billion and 34 billion parameters. The 34-billion-parameter variant was described as offering the highest code quality, which made it suitable as a code assistant.
The smaller models were positioned differently. They were optimized for real-time code completion, with lower latency, and were trained to fill-in-the-middle by default.
Another notable feature was the context window. Code Llama's context window is 100,000 tokens, which makes it relevant for working with large amounts of code at the same time.
"When developers need to debug a large chunk of code, they can pass the entire code length to the model," Meta AI writes.
Meta also released a Python-focused Code Llama variant trained with an additional 100 billion Python code tokens. Alongside it, Meta released an Instruct variant optimized with code tasks and their sample solutions. Meta recommended that Instruct version for code generation because it was best at following instructions.
Why the Open Model Angle Matters
The source article describes Code Llama as outperforming Llama 2, which is not optimized for code, as well as other open-source models tested. With Code Llama 70B, the performance claim becomes stronger because the instruct version is reported ahead of GPT-4 and Gemini Pro in zero-shot HumanEval prompts.
Meta argues that Code Llama's performance makes it an ideal basis for refining code generation models and should help the open-source community as a whole. The article points to the 34B version of Code Llama as an example, saying it had already been significantly improved by the open-source community and brought up to GPT-4 levels.
That is the logic behind the larger model's significance. If the 34B model was already improved substantially by the community, Code Llama 70B may provide more room for refinement. The source frames that as likely, not as a guaranteed outcome.
There is also a licensing debate around the phrase open source. Meta releases Code Llama under the same Llama license as Llama 2 on Github, and the application and the content it generates can be used for scientific and commercial purposes.
At the same time, the Open Source Initiative criticized Meta for marketing the models as open source. Its objection, as summarized in the source, is that the license restricts commercial use and certain areas of application and therefore does not fully meet the open-source definition.
The Practical Takeaway
Code Llama 70B is the strongest Code Llama release described in the source article. Its CodeLlama-70B-Instruct variant scored 67.8 on HumanEval and moved ahead of GPT-4 and Gemini Pro for zero-shot prompts.
For developers, the broader Code Llama family is aimed at code generation, code completion and debugging. For researchers and commercial teams, Meta says Code Llama 70B is available under the usual Llama licenses, with access after asking Meta and more information on Github.
The result is a clearer competitive picture for open code models. Code Llama started as a code-specialized Llama 2 variant, and Code Llama 70B now gives the open model community a larger base model with a stronger benchmark result to build on.