The Llama 3 Herd of Models

The Llama 3 Herd of Models. Llama Team, AI @ Meta. A detailed contributor list can be found in the appendix of this paper. Modern artificial intelligence (AI) systems are powered by foundation models.

Foundation models are general models of language, vision, speech, and/or other modalities that are designed to support a large variety of AI tasks. The development of modern foundation models consists of two main stages: (1) a pre-training stage in which the model is trained at massive scale using straightforward tasks such as next-word prediction or captioning, and (2) a post-training stage in which the model is tuned to follow instructions, align with human preferences, and improve specific capabilities.

In this paper, we present a new set of foundation models for language, called Llama 3. The Llama 3 Herd of models natively supports multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters, processing information in a context window of up to 128K tokens.

We believe there are three key levers in the development of high-quality foundation models: data, scale, and managing complexity. Data. We improved both the quantity and quality of the data we use for pre-training and post-training. We pre-train Llama 3 on a corpus of about 15T multilingual tokens, compared to 1.8T tokens for Llama 2. Scale. Our flagship language model was pre-trained using $3.8 \times 10^{25}$ FLOPs, almost 50× more than the largest version of Llama 2; we pre-trained a flagship model with 405B trainable parameters on 15.6T text tokens. Managing complexity. We opt for a standard dense Transformer model architecture rather than a mixture-of-experts model to maximize training stability, and we adopt a relatively simple post-training procedure based on supervised finetuning (SFT), rejection sampling (RS), and direct preference optimization (DPO).

The development of our Llama 3 language models comprises two main stages. Language model pre-training: we pre-train a model with 405B parameters on 15.6T tokens using a context window of 8K tokens; this is followed by a continued pre-training stage that increases the supported context window to 128K tokens. Language model post-training: the pre-trained language model has a rich understanding of language but it does not yet follow instructions or behave in the way we would expect an assistant to. We align the model with human feedback in several rounds, each of which involves supervised finetuning (SFT) on instruction tuning data and Direct Preference Optimization (DPO).

We also perform experiments in which we add image, video, and speech capabilities to Llama 3 using a compositional approach. The approach we study comprises three additional stages: multi-modal encoder pre-training, vision adapter training, and speech adapter training.

We create our dataset for language model pre-training from a variety of data sources containing knowledge until the end of 2023. We apply several de-duplication methods and data cleaning mechanisms on each data source to obtain high-quality tokens. We remove domains that contain large amounts of personally identifiable information (PII), and domains with known adult content.

Web Data Curation. We implement filters designed to remove data from websites that are likely to contain unsafe content or high volumes of PII. We process the raw HTML content with a custom parser that optimizes for precision in boilerplate removal and content recall; we carefully preserve mathematics and code structure and maintain image alt attribute text. We find that markdown is harmful to performance and remove all markdown markers.

De-duplication. We apply several rounds of de-duplication at the URL, document, and line level. URL-level: we keep the most recent version for each URL. Document-level: global MinHash de-duplication across the entire dataset to remove near duplicates. Line-level: we remove lines that appeared more than 6 times in each bucket of 30M documents (similar to ccNet).

Heuristic filtering. We use duplicated n-gram coverage ratio to remove repeated content; "dirty word" counting to filter adult sites not covered by block lists; and token-distribution KL divergence to filter documents with excessive outlier tokens.

Model-based quality filtering. We use fasttext-based classifiers trained to recognize text that would be referenced by Wikipedia, as well as Roberta-based classifiers trained on Llama 2 predictions; for efficiency we use DistilRoberta to score each document. We also build domain-specific pipelines for code and reasoning, trained on web data annotated by Llama 2.

Multilingual data. Our multilingual text processing pipeline has several unique features: we use a fasttext-based language identification model to categorize documents into 176 languages; we perform document-level and line-level de-duplication within data for each language; we apply language-specific heuristics and model-based filters to remove low-quality documents; we perform quality ranking of multilingual documents using a multilingual Llama 2-based classifier.

Data Mix. Our main tools in determining this data mix are knowledge classification and scaling law experiments. Our final data mix contains roughly 50% of tokens corresponding to general knowledge, 25% of mathematical and reasoning tokens, 17% code tokens, and 8% multilingual tokens.

Annealing Data. We find that annealing on small amounts of high-quality code and mathematical data can boost the performance of pre-trained models on key benchmarks; annealing improved the performance of a pre-trained Llama 3 8B model on the GSM8k and MATH validation sets by 24.0% and 6.4%, respectively. However, the improvements on the 405B model are negligible.

Llama 3 uses a standard, dense Transformer architecture. It does not deviate significantly from Llama and Llama 2 in terms of model architecture; our performance gains are primarily driven by improvements in data quality and diversity as well as by increased training scale. We make a few small modifications compared to Llama 2: we use grouped query attention (GQA) with 8 key-value heads to improve inference speed and to reduce the size of key-value caches during decoding; we use an attention mask that prevents self-attention between different documents within the same sequence; we use a vocabulary with 128K tokens (100K from tiktoken plus 28K additional non-English tokens), improving compression rates from 3.17 to 3.94 characters per token; we increase the RoPE base frequency hyperparameter to 500,000 to better support longer contexts.

Llama 3 405B uses an architecture with 126 layers, a token representation dimension of 16,384, and 128 attention heads.

Scaling Laws. We develop scaling laws to determine the optimal model size for our flagship model given our pre-training compute budget. We implement a two-stage methodology to develop scaling laws that accurately predict downstream benchmark performance: (1) we first establish a correlation between the compute-optimal model's negative log-likelihood on downstream tasks and the training FLOPs; (2) next, we correlate the negative log-likelihood on downstream tasks with task accuracy, utilizing both the scaling law models and older models trained with higher compute FLOPs.

We construct our scaling laws by pre-training models using compute budgets between $6 \times 10^{18}$ FLOPs and $10^{22}$ FLOPs. We assume a power-law relation between compute budget $C$ and the optimal number of training tokens $N^\star(C)$:

Extrapolation of the resulting scaling law to $3.8 \times 10^{25}$ FLOPs suggests training a 402B parameter model on 16.55T tokens. An important observation is that IsoFLOPs curves become flatter around the minimum as the compute budget increases — this implies that performance of the flagship model is relatively robust to small changes in the trade-off between model size and training tokens. Based on this observation, we ultimately decided to train a flagship model with 405B parameters.

Initial Pre-Training. We pre-train Llama 3 405B using AdamW with a peak learning rate of $8 \times 10^{-5}$, a linear warm up of 8,000 steps, and a cosine learning rate schedule decaying to $8 \times 10^{-7}$ over 1,200,000 steps. We use a lower batch size early in training to improve training stability, and increase it subsequently to improve efficiency. Specifically, we use an initial batch size of 4M tokens and sequences of length 4,096, and double these values to a batch size of 8M sequences of 8,192 tokens after pre-training 252M tokens. We double the batch size again to 16M after pre-training on 2.87T tokens.

Long Context Pre-Training. In the final stages of pre-training, we train on long sequences to support context windows of up to 128K tokens. We do not train on long sequences earlier because the compute in self-attention layers grows quadratically in the sequence length. We increase the supported context length in increments, pre-training until the model has successfully adapted to the increased context length. We assess successful adaptation by measuring whether (1) model performance on short-context evaluations has recovered completely and (2) the model perfectly solves "needle in a haystack" tasks up to that length. In Llama 3 405B pre-training, we increased context length gradually in six stages, starting from the original 8K context window and ending in the final 128K context window. This long-context pre-training stage was performed using approximately 800B training tokens.

Annealing. During pre-training on the final 40M tokens, we linearly annealed the learning rate to 0, maintaining a context length of 128K tokens. During this annealing phase, we also adjusted the data mix to upsample data sources of very high quality. Finally, we compute the average of model checkpoints (Polyak averaging) during annealing to produce the final pre-trained model.

We produce the aligned Llama 3 models by applying several rounds of post-training, or aligning the model with human feedback on top of a pre-trained checkpoint. Each round of post-training involves supervised finetuning (SFT) followed by Direct Preference Optimization (DPO) on examples collected either via human annotations or generated synthetically. The backbone of our post-training strategy is a reward model and a language model. We first train a reward model on top of the pre-trained checkpoint using human-annotated preference data. We then finetune pre-trained checkpoints with supervised finetuning (SFT), and further align the checkpoints with Direct Preference Optimization (DPO).

Reward Modeling. The training objective is the same as Llama 2 except that we remove the margin term in the loss, as we observe diminishing improvements after data scaling. Annotations also create a third "edited response" for some prompts, where the chosen response from the pair is further edited for improvement. Hence, each preference ranking sample has two or three responses with clear ranking (edited > chosen > rejected). The standard reward model objective is the Bradley–Terry pairwise loss:

where $(y_w, y_l)$ is a preferred / rejected pair and $\sigma$ is the logistic function. We concatenate the prompt and multiple responses into a single row during training with responses randomly shuffled — this approximates the standard scenario but improves training efficiency without a loss in accuracy.

Supervised Finetuning (SFT). The reward model is then used to perform rejection sampling on our human annotation prompts. Together with this rejection-sampled data and other data sources (including synthetic data), we finetune the pre-trained language model using a standard cross entropy loss on the target tokens (while masking loss on prompt tokens). We refer to this stage as supervised finetuning even though many of the training targets are model-generated. Our largest models are finetuned with a learning rate of $10^{-5}$ over the course of 8.5K to 9K steps.

Direct Preference Optimization (DPO). We further train our SFT models with DPO for human preference alignment. For training, we primarily use the most recent batches of preference data collected using the best performing models from the previous alignment rounds. We also explored on-policy algorithms such as PPO, but found that DPO required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval. For Llama 3, we use a learning rate of $10^{-5}$ and set the $\beta$ hyper-parameter to be $0.1$. The DPO objective is:

We apply two algorithmic modifications to DPO. Masking out formatting tokens in DPO loss: we mask out special formatting tokens (header and termination tokens) from both chosen and rejected responses in the loss; we observe that having these tokens contribute to the loss may lead to undesired model behaviors such as tail repetition or abruptly generating termination tokens. Regularization with NLL loss: we add an additional negative log-likelihood loss term with a scaling coefficient of $0.2$ on the chosen sequences, which helps stabilize DPO training by maintaining desired formatting and preventing the decrease of log probability of chosen responses.

Model Averaging. Finally, we average models obtained from experiments using various versions of data or hyperparameters at each RM, SFT, or DPO stage. Iterative Rounds. Following Llama 2, we apply the above methods in six rounds. In each cycle, we collect new preference annotations and SFT data, sampling synthetic data from the latest models.

Preference Data. Our preference data annotation process is similar to Llama 2. We deploy multiple models for annotation after each round and sample two responses from two different models for each user prompt. We ask annotators to rate the strength of their preference by categorizing it into one of four levels (significantly better, better, slightly better, marginally better). We also incorporate an editing step after preference ranking to encourage annotators to further improve the preferred response. Consequently, a portion of our preference data has three responses ranked (edited > chosen > rejected).

In each round of post-training, we use all the preference data that is available at the time for reward modeling, while only using the latest batches from various capabilities for DPO training. For both reward modeling and DPO, we use samples that are labeled as the chosen response being significantly better or better than the rejected counterpart for training and discard samples with similar responses.

SFT Data. Our finetuning data is largely comprised of the following sources: (1) Prompts from our human annotation collection with rejection-sampled responses; (2) Synthetic data targeting specific capabilities; (3) Small amounts of human-curated data. We refer to this stage as supervised finetuning even though many of the training targets are model-generated. Our finetuning data is meticulously curated to ensure quality.

Data Processing and Quality Control. Given that most of our training data is model-generated, it requires careful cleaning and quality control. We perform topic classification, quality scoring (using both reward-model scores and Llama-based rubric scores), difficulty scoring, and semantic de-duplication (using RoBERTa-based embeddings + clustering) to keep only high-quality, diverse examples.

Code. We train an expert via continued pre-training on a code-heavy data mix, then run multiple rounds of SFT on it. We use the expert to generate high-quality SFT data for code and to back-translate code documentation and tests. We also use execution feedback to filter out generations that fail to compile, run, or pass unit tests.

Reasoning. For mathematical reasoning, we (1) generate step-by-step solutions and verify them with answer-checking, (2) use Llama 3 itself as a reward signal to filter reasoning chains, and (3) sample long chains-of-thought during rejection sampling.

Tool Use. We train Llama 3 to use search engines, a Python interpreter, and Wolfram Alpha. We generate synthetic data for tool use, including single-turn and multi-turn function calling. We use multiple agents to generate domains, APIs, user queries, API calls, and responses, while also ensuring that the generated data covers a set of diverse domains and realistic APIs.

Factuality. We follow the principle that post-training should align the model to "know what it knows" rather than add knowledge. Our primary approach involves generating data that aligns model generations with subsets of factual data present in the pre-training data: extract pre-training snippets → generate factual questions about them → sample answers → score correctness (Llama-as-judge against the original context) → score informativeness → generate refusals for "confidently wrong" outputs.

Steerability. For Llama 3, we focus on enhancing its steerability through system prompt with natural language instructions, especially around response length, format, tone and character/persona. We collect steerability preference samples by asking annotators to design different system prompts and evaluate conversations against them.

We performed an extensive series of evaluations of Llama 3, investigating the performance of: (1) the pre-trained language model, (2) the post-trained language model, and (3) the safety characteristics of Llama 3. The evaluations cover eight top-level categories: commonsense reasoning; knowledge; reading comprehension; math, reasoning, and problem solving; long context; code; adversarial evaluations; and aggregate evaluations.

Llama 3 8B outperforms competing models in virtually every category. Llama 3 70B outperforms its predecessor Llama 2 70B by a large margin on most benchmarks, with the exception of commonsense benchmarks that are likely saturated. Our flagship Llama 3 405B model performs on par with leading language models such as GPT-4 across a variety of tasks, and is close to matching the state-of-the-art on some benchmarks.

Selected benchmarks for Llama 3 405B Instruct vs GPT-4 (0125) / GPT-4o / Claude 3.5 Sonnet: MMLU (5-shot) 87.3 / 85.1 / 89.1 / 89.9; MMLU-Pro (5-shot, CoT) 73.3 / 64.8 / 74.0 / 77.0; IFEval 88.6 / 84.3 / 85.6 / 88.0; HumanEval 89.0 / 86.6 / 90.2 / 92.0; GSM8K 96.8 / 94.2 / 96.1 / 96.4; MATH 73.8 / 64.5 / 76.6 / 71.1; ARC-C 96.9 / 96.4 / 96.7 / 96.7; GPQA 51.1 / 41.4 / 53.6 / 59.4; MGSM 91.6 / 85.9 / 90.5 / 91.6.

For long context: Llama 3 405B reaches 95.2 on ZeroSCROLLS/QuALITY, 83.4 on InfiniteBench/En.MC, and 98.1 on NIH/Multi-needle. We also conduct human evaluations comparing Llama 3 against competing models.

Safety. We conducted extensive measurement and mitigation on a wide variety of risks to safe usage of Llama 3. However, no testing can be guaranteed to be exhaustive in identifying every possible risk. Llama 3 may still generate harmful content due to training on various datasets, particularly for languages beyond English and when prompt engineered by skilled adversarial red teamers. We released Llama Guard for input and output safety, and a Prompt Guard classifier for jailbreak / prompt-injection detection (jailbreaks TPR 99.9%, indirect injections TPR 71.4%).

Inference. We investigate two main techniques to make inference with the Llama 3 405B model efficient: pipeline parallelism and FP8 quantization. When using a BF16 number representation for the model parameters, Llama 3 405B does not fit in the GPU memory of a single machine with 8 Nvidia H100 GPUs. To address this issue, we parallelize model inference using BF16 precision across 16 GPUs on two machines. Within each machine, the high NVLink bandwidth enables the use of tensor parallelism. Across nodes, however, connectivity has lower bandwidth and higher latency, so we use pipeline parallelism instead.

For FP8 quantization, we apply FP8 quantization to most matrix multiplications inside the model. In particular, we quantize most parameters and activations in the feedforward network layers in the model, which account for roughly 50% of the inference compute time. We do not quantize parameters in the self-attention layers of the model. We have publicly released our implementation of FP8 quantization.

We perform a series of experiments in which we incorporate visual-recognition capabilities into Llama 3 via a compositional approach that consists of two main stages. First, we compose a pre-trained image encoder and the pre-trained language model by introducing and training a set of cross-attention layers between the two models on a large number of image-text pairs. Second, we introduce temporal aggregator layers and additional video cross-attention layers that operate on a large collection of video-text pairs to learn the model to recognize and process temporal information from videos.

A compositional approach has several advantages: (1) it enables us to parallelize the development of the vision and language modeling capabilities; (2) it circumvents complexities of joint pre-training on visual and language data; (3) it guarantees that model performance on text-only tasks is not affected by the introduction of visual-recognition capabilities; (4) the cross-attention architecture ensures that we do not have to expend compute passing full-resolution images through the increasingly large LLM backbones, making it more efficient during inference.

Image Data Pipeline. Our image encoder and adapter are trained on image-text pairs. We construct this dataset via a complex data processing pipeline that consists of four main stages: (1) quality filtering (using CLIP-score based heuristics), (2) perceptual de-duplication (using an internal SSCD copy-detection model), (3) resampling (n-gram based to favor richer descriptions), and (4) optical character recognition. We also apply a series of safety mitigations, including PhotoDNA scanning for CSAM, NSFW filtering, and face blurring.

Annealing data. We create an annealing dataset by resampling the image-caption pairs to ∼350M examples using n-grams, augmented with ∼150M examples from five additional sources: visual grounding (linking noun phrases to bounding boxes / masks), screenshot parsing (HTML rendering ↔ code), question-answer pairs, synthetic captions, and synthetically-generated structured images (charts, tables, math equations).

Speech encoder. Our speech encoder is trained using a self-supervised approach that masks out parts of the speech inputs and tries to reconstruct the masked out parts via a discrete-token representation. As a result, the model learns the structure of speech signals.

Speech adapter. We integrate the speech encoder into the model via an adapter that converts speech encodings into token representations that can be fed directly into the finetuned language model. The parameters of the adapter and encoder are jointly updated in a supervised finetuning stage to enable high-quality speech understanding. We do not change the language model during speech adapter training. We also integrate a text-to-speech system.

The development of Llama 3 builds on a large body of prior work studying foundation models for language, images, videos, and speech. A comprehensive overview of that work is outside the scope of this paper. Llama 3 inherits ideas from PaLM, Chinchilla scaling laws, the GQA family for efficient inference, RoPE positional embeddings, RLHF / DPO for alignment, Flamingo-style cross-attention for vision-language models, and self-supervised audio representations for speech.

In many ways, the development of high-quality foundation models is still in its infancy. Our experience in developing Llama 3 suggests that substantial further improvements of these models are on the horizon. Throughout the development of the Llama 3 model family, we found that a strong focus on high-quality data, scale, and simplicity consistently yielded the best results. The open release of Llama 3 reaffirms the importance of making such models broadly available to the research community to accelerate progress on AI in a responsible way.