Today, we’re releasing Code Llama, a large language model (LLM) that can use text prompts to generate and discuss code. Code Llama is state-of-the-art for publicly available LLMs on coding tasks. It has the potential to make workflows faster and more efficient for developers and lower the barrier to entry for people who are learning to code. Code Llama has the potential to be used as a productivity and educational tool to help programmers write more robust, well-documented software.
We believe an open approach to AI is best for developing new AI tools that are innovative, safe and responsible, so we’re releasing Code Llama for both research and commercial use under the same community license as Llama 2.
Code Llama is a code-specialized version of Llama 2 that was created by further training Llama 2 on its code-specific datasets, sampling more data from that same dataset for longer. Essentially, Code Llama features enhanced coding capabilities. It can generate code and natural language about code, from both code and natural language prompts (e.g., “Write me a function that outputs the fibonacci sequence”). It can also be used for code completion and debugging. It supports many of the most popular programming languages used today, including Python, C++, Java, PHP, Typescript (Javascript), C#, Bash and more.
We are releasing three sizes of Code Llama with 7B, 13B and 34B parameters respectively. Each of these models is trained with 500B tokens of code and code-related data. The 7B and 13B base and instruct models have also been trained with fill-in-the-middle (FIM) capability, allowing them to insert code into existing code, meaning they can support tasks like code completion right out of the box.
The three models address different serving and latency requirements. The 7B model, for example, can be served on a single GPU. The 34B model returns the best results and allows for better coding assistance, but the smaller 7B and 13B models are faster and more suitable for tasks that require low latency, like real-time code completion.
We also further fine-tuned two additional variations of Code Llama: Code Llama – Python and Code Llama – Instruct.
Code Llama – Python is a language specialized variation of Code Llama, further fine-tuned on 100B tokens of Python code. Because Python is the most benchmarked language for code generation, and because Python and PyTorch play an important role in the AI community – we believe a specialized model provides additional utility.
Code Llama – Instruct is an instruction fine-tuned and aligned variation of Code Llama. Instruction tuning continues the training process, but with a different objective. The model is fed a natural language instruction input and the expected output. This makes it better at understanding what people expect out of their prompts. We recommend using Code Llama – Instruct variants whenever using Code Llama for code generation since Code Llama – Instruct has been fine-tuned to generate helpful and safe answers in natural language.
Programmers are already using LLMs to assist in a variety of tasks. The goal is to make developer workflows more efficient so that they can focus on the most human-centric aspects of their job, rather than repetitive tasks. We believe that AI models, and LLMs for coding in particular, benefit most from an open approach, both in terms of innovation and safety. Publicly available, code-specific models can facilitate the development of new technologies that improve peoples’ lives. By releasing code models like Code Llama, the entire community can evaluate their capabilities, identify issues and fix vulnerabilities.
Code Llama is designed to support software engineers in all sectors — including research, industry, open source projects, NGOs and businesses. But there are still many more use cases to support. We hope Code Llama will inspire others to leverage Llama 2 to create new innovative tools for research and commercial products.
Learn more about Code Llama on our AI blog or download the Code Llama model.