Training language models to follow instructions

Making language models bigger does not inherently make them better at following a user's intent: large LMs can produce outputs that are untruthful, toxic, or unhelpful. We align language models with user intent by fine-tuning with human feedback. Starting from labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of desired model behavior and fine-tune GPT-3 with supervised learning; we then collect a dataset of rankings of model outputs and further fine-tune with reinforcement learning from human feedback. The resulting models, called InstructGPT, are preferred by labelers over GPT-3: outputs from the 1.3B InstructGPT are preferred to those from the 175B GPT-3, despite a 100× parameter gap. InstructGPT also shows improvements in truthfulness and reduced toxic generation while incurring only minimal performance regressions on public NLP datasets, suggesting fine-tuning with human feedback is a promising direction for alignment.

Large language models can be "prompted" to perform a range of NLP tasks, but they often express unintended behaviors: making up facts, generating biased or toxic text, or simply not following user instructions. The root cause is a misalignment between the LM training objective (predict the next token on internet text) and the practical objective ("follow the user's instructions helpfully and safely"). We aim to align LMs with user intentions — both explicit (instruction-following) and implicit (truthful, unbiased, non-toxic, non-harmful) — and adopt the Askell et al. framing of helpful, honest, harmless. To do so, we use reinforcement learning from human feedback (RLHF) to fine-tune GPT-3 on a broad class of written instructions. Our quantitative outcome is striking: a 1.3B InstructGPT beats the 175B GPT-3 in labeler preference on our API distribution, demonstrating that alignment fine-tuning can dominate two orders of magnitude of pretraining scale.

Our methodology follows Ziegler et al. (2019) and Stiennon et al. (2020): we hire a team of about 40 contractors selected by a screening test, collect human-written demonstrations on prompts submitted to OpenAI API (plus some labeler-written prompts) to train an SFT baseline, then collect comparison data between model outputs and train a reward model (RM) to predict labeler preferences. Finally we use the RM as a reward function and fine-tune the supervised baseline with PPO to maximize that reward, illustrated in Figure 2. The procedure aligns GPT-3 to the stated preferences of a specific group of people — mostly our labelers and researchers — rather than to any broader notion of "human values"; we revisit this caveat in §5.2. We train three model sizes (1.3B, 6B, 175B) on the GPT-3 architecture, and call the resulting models InstructGPT.

Our main findings are: (i) labelers significantly prefer InstructGPT over GPT-3 — 175B InstructGPT outputs win against 175B GPT-3 85±3% of the time, and 71±4% against few-shot 175B GPT-3; (ii) InstructGPT improves on truthfulness — on TruthfulQA it produces truthful and informative answers about twice as often as GPT-3, and on closed-domain tasks it halves the hallucination rate (41% → 21%); (iii) toxicity drops modestly (about 25% fewer toxic outputs under a "respectful" prompt), but bias on Winogender / CrowS-Pairs barely moves; (iv) RLHF fine-tuning induces an alignment tax — performance regressions on SQuAD, DROP, HellaSwag, WMT — which we largely remove by mixing pretraining gradients into PPO updates (PPO-ptx); (v) the models generalize to held-out labelers (similar preference rates) and to instructions outside the fine-tuning distribution (non-English, coding), though they still make simple mistakes such as hedging, hallucinating, or accepting false premises.

We build on prior RLHF work originally developed for simple robots and Atari games (Christiano et al. 2017; Ibarz et al. 2018) and more recently applied to fine-tune LMs for summarization (Ziegler et al. 2019; Stiennon et al. 2020; Wu et al. 2021), dialogue, translation, semantic parsing, story generation, and evidence extraction. Conceptually our work is a direct application of RLHF to aligning language models on a broad distribution of language tasks rather than a single task. A second line of related work is cross-task instruction tuning (FLAN, T0, etc.), which fine-tunes LMs on a mix of public NLP datasets prefixed with natural-language instructions and evaluates on held-out tasks; we compare InstructGPT head-to-head with FLAN- and T0-finetuned GPT-3 in §4. A third line concerns evaluating and mitigating harms — toxicity (RealToxicityPrompts), bias (CrowS-Pairs, Winogender), data leakage and misuse — much of which informs both our evaluation and our labeling rubric.

Our methodology follows Ziegler et al. (2019) and Stiennon et al. (2020). Starting from a pretrained LM (GPT-3), a prompt distribution, and a team of trained human labelers, we apply three steps (Figure 2). Step 1 (SFT): labelers provide demonstrations of desired behavior on the input prompt distribution; we fine-tune the pretrained GPT-3 on these demonstrations with supervised learning. Step 2 (RM): we collect a dataset of comparisons between model outputs for the same prompt, where labelers indicate which output they prefer; we train a reward model $r_\theta(x, y)$ to predict the human-preferred output. Step 3 (PPO): we treat the RM output as a scalar reward and fine-tune the SFT policy with PPO to maximize it. Steps 2 and 3 can be iterated: collect more comparisons on the current best policy, train a new RM, then a new policy.

Our prompt dataset comes primarily from text prompts submitted to the OpenAI API Playground on an earlier InstructGPT model (trained only on demonstration data); users were informed via a recurring notification that their data could be used for further training, and we do not use prompts from production API traffic. We heuristically deduplicate by long common prefixes, cap to 200 prompts per user-ID, and split train/val/test by user-ID so that no user appears in two splits. PII is filtered out of the training split. To bootstrap the very first models, labelers also wrote three kinds of prompts themselves: plain (arbitrary tasks with diversity), few-shot (one instruction plus several query/response pairs), and user-based (use-cases drawn from API waitlist applications). From these prompts we derive three datasets: the SFT dataset (~13k training prompts), the RM dataset (~33k training prompts), and the PPO dataset (~31k training prompts, unlabeled). Use-case mix on the RM dataset: Generation 45.6%, Open QA 12.4%, Brainstorming 11.2%, Chat 8.4%, Rewrite 6.6%, Summarization 4.2%, Classification 3.5%, Other 3.5%, Closed QA 2.6%, Extract 1.9%. The dataset is over 96% English. Labelers are instructed to infer user intent, and to consider implicit intentions such as truthfulness and harm avoidance.

We hired about 40 contractors on Upwork and ScaleAI to produce demonstrations and comparisons and to perform our main evaluations. Compared to prior work that collected human preferences on a single task such as summarization, our inputs span a much broader range of tasks and can include controversial or sensitive topics, so we designed a screening test to measure each candidate's sensitivity to different demographic groups and their ability to flag potentially harmful outputs; only those who scored well were retained. During training we instruct labelers to prioritize helpfulness to the user; during the final evaluations we instruct them to prioritize truthfulness and harmlessness, since this matches what we care about in deployment. We onboard labelers, write detailed per-task instructions (Appendix B.2), and run a shared chat room to answer questions in real time. As a generalization check we also hire a separate set of "held-out" labelers (same vendors, no screening test) who produce none of the training data. Inter-annotator agreement is high despite task complexity — 72.6±1.5% among training labelers and 77.3±1.3% among held-out labelers (Stiennon et al. 2020 reported 73±4% researcher-researcher agreement).

Supervised fine-tuning (SFT). We fine-tune GPT-3 on labeler demonstrations using supervised learning: 16 epochs, cosine learning-rate decay, residual dropout 0.2. Final SFT model selection is based on RM score on the validation set. Similar to Wu et al. (2021), SFT overfits on validation loss after 1 epoch, but training longer continues to improve both RM score and human preference ratings — so we accept that overfitting.

Reward modeling (RM). Starting from the SFT model with its final unembedding layer removed, we add a linear head that outputs a single scalar reward for a (prompt, completion) pair. We use only 6B RMs in this paper because 175B RM training was unstable and less suitable as a value function during RL. To accelerate labeling we present labelers with $K \in [4, 9]$ responses per prompt to rank, producing $\binom{K}{2}$ comparisons per prompt. Critically, we train all $\binom{K}{2}$ comparisons from one prompt as a single batch element rather than as independent samples; this requires only one RM forward pass per completion (instead of $K\!-\!1$), and — because each completion is no longer reused across many gradient updates in one epoch — avoids overfitting. The loss is

where $r_\theta(x, y)$ is the scalar output of the RM with parameters $\theta$, $y_w$ is the preferred completion of the pair $(y_w, y_l)$, and $\mathcal{D}$ is the dataset of human comparisons. Because the RM loss is invariant to reward shifts, we add a bias before RL so that labeler demonstrations have mean reward 0.

Reinforcement learning (PPO & PPO-ptx). Following Stiennon et al. (2020), we fine-tune the SFT model with PPO. The environment is a bandit that presents a random customer prompt and expects a response; given (prompt, response) the RM emits a reward and the episode ends. We additionally add a per-token KL penalty from the SFT model at each token, to mitigate over-optimization of the RM. The value function is initialized from the RM. We call these models "PPO". We also mix the pretraining gradients into the PPO gradients to fix performance regressions on public NLP datasets, calling these "PPO-ptx". The combined RL objective is

where $\pi_\phi^{\mathrm{RL}}$ is the learned RL policy, $\pi^{\mathrm{SFT}}$ is the supervised model, and $\mathcal{D}_{\mathrm{pretrain}}$ is the pretraining distribution. The KL coefficient $\beta$ controls the per-token KL penalty (and hence drift away from $\pi^{\mathrm{SFT}}$); the pretraining-mix coefficient $\gamma$ controls how strongly we keep the language-modeling objective. For "PPO" models $\gamma = 0$; for "PPO-ptx" $\gamma > 0$. Unless stated otherwise, "InstructGPT" in this paper refers to the PPO-ptx models.

Baselines. We compare the PPO models to SFT and to GPT-3, both raw and with a hand-crafted few-shot prefix that nudges GPT-3 into an instruction-following mode ("GPT-3-prompted"). We additionally compare InstructGPT to 175B GPT-3 fine-tuned on the FLAN (Wei et al. 2021) and T0 (Sanh et al. 2021) datasets — instruction-tuned compilations of public NLP tasks — using approximately 1M examples each and picking the checkpoint with the highest validation RM score.

Following Leike et al. (2018) and Askell et al. (2021), we operationalize "alignment" as helpful, honest, harmless. For helpful, the metric is labeler preference on a held-out set of API prompts. For honest, since we can't probe a model's "belief", we measure truthfulness via (i) hallucination rate on closed-domain API tasks and (ii) the TruthfulQA dataset. For harmless, we use proxy criteria evaluated by labelers (e.g., inappropriate for a customer assistant, denigrates a protected class, contains sexual/violent content) plus public benchmarks (RealToxicityPrompts, CrowS-Pairs, Winogender). Evaluations split into two parts: (a) on the API distribution, using win-rates against a fixed baseline (we pick the 175B SFT model since its performance sits in the middle of the pack) and 1–7 Likert overall-quality ratings; (b) on public NLP datasets — both safety-oriented and traditional zero-shot tasks like QA, reading comprehension, summarization — plus human evaluations of toxicity on RealToxicityPrompts.

4.1 Results on the API distribution. Across model sizes, labelers significantly prefer InstructGPT over GPT-3 (Figure 1). The ordering is consistent: raw GPT-3 < GPT-3-prompted < SFT < PPO ≈ PPO-ptx. Concretely, 175B InstructGPT wins 85±3% against 175B GPT-3 and 71±4% against few-shot 175B GPT-3; even the 1.3B PPO-ptx beats 175B GPT-3 in pairwise preference. The same ordering holds when evaluated on prompts that were submitted to GPT-3 itself, not to InstructGPT (Figure 3), refuting a "we just rigged the prompt distribution" objection. Along finer-grained metadata (Figure 4), PPO models are more often attempting the correct instruction, more often follow explicit constraints (e.g., "answer in two paragraphs"), are rated more appropriate for a customer assistant, and hallucinate roughly half as often in closed-domain tasks (21% vs. 41% for GPT-3).

Generalization to held-out labelers and to FLAN/T0. Held-out labelers — who produced none of the training data — show preference patterns essentially identical to training labelers (Figure 3), and a 5-fold cross-validation RM achieves 69.6±0.9% on held-out groups vs. 72.4±0.4% on its training groups, indicating the reward signal generalizes beyond the training annotators. In head-to-head comparisons, 175B InstructGPT outputs are preferred over 175B GPT-3-FLAN 78±4% of the time and over 175B GPT-3-T0 79±4% of the time (Figure 5). We attribute this to two factors: (i) public NLP datasets are dominated by classification / QA / summarization that together account for only ~18% of API use cases, while open-ended generation and brainstorming together account for ~57%; (ii) public NLP datasets struggle to match the input diversity that real users actually want — though the broadest instruction-following model should clearly combine both.

4.2 Public NLP datasets — truthfulness, toxicity, bias, alignment tax. On TruthfulQA, PPO models produce truthful-and-informative outputs roughly twice as often as GPT-3 (Figure 6), even without being explicitly instructed to tell the truth; the exception is the 1.3B PPO-ptx, which is slightly worse than 1.3B GPT-3. With a "respectful" prompt on RealToxicityPrompts, InstructGPT generates about 25% fewer toxic outputs than GPT-3 (Figure 7); without the respectful prompt the gap disappears, and when explicitly asked to be toxic, InstructGPT can actually be more toxic than GPT-3. On Winogender and CrowS-Pairs (bias) we see no significant improvement. RLHF causes regressions on SQuAD, DROP, HellaSwag, and WMT 2015 fr→en — the classic alignment tax; mixing pretraining gradients into PPO (PPO-ptx) recovers most of these losses without hurting labeler preference scores.

4.3 Qualitative results. InstructGPT shows promising generalization to instructions outside the fine-tuning distribution: it can follow code-summarization prompts, answer questions about code, and sometimes follow instructions in non-English languages, despite these inputs being very rare in the fine-tuning data. By contrast, GPT-3 can perform such tasks with careful prompting but does not reliably follow instructions in these domains. InstructGPT still makes simple mistakes: failing to follow instructions, fabricating facts, giving long hedging answers to simple questions, and failing to detect false premises in the user query. These failures motivate the discussion in §5.

5.1 Implications for alignment research. The cost of aligning is modest relative to pretraining: training our 175B SFT requires 4.9 petaflops/s-days and our 175B PPO-ptx requires 60 petaflops/s-days, against 3,640 petaflops/s-days for GPT-3 itself. Yet RLHF makes 1.3B models more preferred than 175B GPT-3 in labeler win-rate — currently a more cost-effective investment than scaling further. We observe encouraging cross-distribution generalization (non-English, code) and validate alignment ideas on production-grade models. Finally, by mostly removing the alignment tax via PPO-ptx, we make RLHF a low-tax alignment technique, which is essential if highly capable future systems are to be aligned without a competitive disadvantage.

5.2 Who are we aligning to? The literature often invokes "human preferences" or "human values", but the operational reality is narrower. We align to (i) the demonstrations and preferences of our ~40 training labelers (mostly English speakers in the US or Southeast Asia hired through Upwork/ScaleAI; inter-annotator agreement ~73%); (ii) the researchers' preferences embedded in the labeling instructions, edge-case answers in the chat room, and the very choice of evaluation metrics; (iii) the OpenAI API customer base whose prompts populate the training data and who pay for the model; (iv) OpenAI's own commercial constraints, which exclude certain content. None of these correspond to "humanity at large", and any of them can be questioned. The right framing is not "the right preferences to align to" but "a process for designing a fair, transparent, and accountable alignment pipeline" — and our work is only a first concrete instance.

Methodology. InstructGPT's behavior is determined in part by human feedback obtained from contractors, and the inter-annotator disagreement (~27%) plus the explicit instruction to prioritize helpfulness during training vs. truthfulness and harmlessness during evaluation can create internal conflicts. We did not explore active learning or careful labeler-pool sizing; with more labelers from broader backgrounds the resulting model could be meaningfully different. We also did not explore alternatives to PPO (e.g., expert iteration, advantage-weighted regression) or alternatives to the per-token KL penalty for controlling drift from $\pi^{\mathrm{SFT}}$.

Models. Despite improvements, InstructGPT models are still neither fully aligned nor fully safe. They still generate toxic or biased outputs, make up facts, and produce sexual or violent content when prompted; they may fail to generate reasonable outputs at all for some inputs. The biggest failure of the current models is following user instructions even when those instructions would lead to harm in the real world — e.g., compliance with "write a misleading article" requests; ideally the models would refuse. The right tradeoff between user helpfulness and overall harmlessness is a values question that engineers alone cannot answer.