Continual Pre-Training of Large Language Models: How to (re)warm your model?

Large language models (LLMs) are routinely pre-trained on billions of tokens, only to restart the process over again once new data becomes available. A much cheaper and more efficient solution would be to enable the continual pre-training of these models, i.e. updating pre-trained models with new data instead of re-training them from scratch. However, the distribution shift induced by novel data typically results in degraded performance on past data. Taking a step towards efficient continual pre-training, in this work, we examine the effect of different warm-up strategies. Our hypothesis is that the learning rate must be re-increased to improve compute efficiency when training on a new dataset.

We study the warmup phase of models pre-trained on the Pile (upstream data, 300B tokens) as we continue to pre-train on SlimPajama (downstream data, 297B tokens), following a linear warmup and cosine decay schedule. We conduct all experiments on the Pythia 410M language model architecture and evaluate performance through validation perplexity. We experiment with different pre-training checkpoints, various maximum learning rates, and various warmup lengths. Our results show that while rewarming models first increases the loss on upstream and downstream data, in the longer run it improves the downstream performance, outperforming models trained from scratch — even for a large downstream dataset.

Large pre-trained models have enabled massive performance improvements for many downstream tasks in vision (Kirillov et al., 2023; Oquab et al., 2023) and language (Brown et al., 2020; Zhao et al., 2023). However, training these foundation models is prohibitively expensive. Existing works aim to reduce the cost of large-scale model development by enabling low-cost hyperparameter optimization (Yang et al., 2022) or providing guidelines for maximizing performance under a given compute budget (Hoffmann et al., 2022). However, these works assume that models will be trained from scratch. As the amount of data available for pre-training is ever-growing, new and improved datasets (e.g. RedPajama and SlimPajama) will continue to become available. Should practitioners always combine existing datasets (e.g. Pile) and train from scratch to obtain the best performance? Doing so would quickly become prohibitively expensive and fails to leverage existing pre-trained models.

Our approach circumvents the need for complete re-training by continuing to pre-train existing models on new data. We refer to this as "continual pre-training" (CPT) and the goal is to minimize the loss on new data while maintaining low loss on previous data. CPT is a critical challenge since it can lead to catastrophic forgetting (French, 1999). Moreover, the potential long sequence of training stages may make common continual learning techniques such as replay (Rebuffi et al., 2017; Ostapenko et al., 2022) or regularisation (Kirkpatrick et al., 2017; Farajtabar et al., 2020) not compute efficient enough (Lesort et al., 2023). A simple and — from a compute cost perspective — scalable solution to limit forgetting in such situations is to (only) progressively decrease the learning rate every time new data becomes available (Mirzadeh et al., 2020; Winata et al., 2023). However, this solution is limited because repeatedly decreasing the learning rate would cause it to eventually become too small if the number of training stages becomes high.

In this work, we take a step towards efficient continual pre-training by studying how to re-increase a small learning rate to keep training a pre-trained language model on new data. We refer to this as re-warming the model. Re-warming the model should improve learning efficiency by avoiding a vanishing learning rate. We study warm-up strategies on Pythia 410M model with various amounts of data, maximum learning rates and different pre-trained checkpoints. This would allow a model trained initially on a large dataset to benefit from resuming training on a newer large dataset without having to retrain from scratch. In order to simulate this setting, we fix our initial pre-training dataset to be Pile and the newer dataset to be SlimPajama. We hope that this may guide the adaptation of existing LLMs to future new datasets.

Our results show that: (1) progressively increasing the learning rate to warm-up is not necessary, but starting directly from the maximum learning rate creates an initial large spike in the loss (chaotic phase, a.k.a. stability gap) with no consequences later. (2) Adjusting the maximum learning rate can help trade off between upstream and downstream performance: increasing the maximum learning rate leads to stronger adaptation to the downstream dataset (SlimPajama), while smaller learning rates preserve more performance on the upstream dataset (Pile). (3) Continual pre-training with the latest pre-trained checkpoint improves performance.

Crucially, we find that even though re-warmed models first suffer higher loss on both upstream and downstream data, in the longer run they outperform models trained from scratch — even when the downstream dataset is itself very large. This is strong evidence that CPT, not retraining from scratch, should be the default response to newly available pre-training data — provided the LR is re-warmed properly.

In our setup, the upstream (or pre-training) dataset is the Pile (Gao et al., 2020). The downstream (or fine-tuning) dataset is SlimPajama (Soboleva et al., 2023). SlimPajama is an extensively deduplicated version of RedPajama (Together.xyz, 2023) which is built based on the LLaMA dataset (Touvron et al., 2023). In this work, we use "fine-tuning" and downstream continual pre-training interchangeably. However, in our continual pre-training setting, we note that the downstream dataset is on the scale of the previous pre-training dataset (i.e. very large, unlike many fine-tuning datasets). The SlimPajama dataset is built from similar sources as the Pile but with a higher quantity of data. Therefore, some upstream data may be repeated during downstream pre-training. Our experimental setup is comparable to the setup of Ash & Adams (2020), where they train a classifier on half of the samples of a dataset first, and fine-tune it later on all samples. They show that warm starting for image classification is challenging. Using a model pre-trained on the Pile and continuing the pre-training on SlimPajama, we follow an analogous setup for causal language modeling.

Datasets. We use the Pile with the same weights as Black et al. (2022) for validation. We shuffle and randomly sample the SlimPajama dataset (Soboleva et al., 2023) to form the ~297B-token training dataset and ~316M validation token dataset. The training mix is: StackExchange 2.0% (9.95B train, 13.08M val), Arxiv 2.5% (13.77B / 22.73M), Wikipedia 4.5% (11.78B / 15.79M), Book 4.5% (14.22B / 22.04M), Github 4.5% (15.41B / 22.42M), C4 15.0% (78.49B / 72.49M), CommonCrawl 67.0% (153.25B / 147.28M) — totaling 100% / 296.86B train / 315.83M validation tokens. We do not use replay. We use the same tokenizer as Black et al. (2022), which was trained specifically on the Pile.

Model. We use the 410M Pythia model pre-trained on the Pile (Biderman et al., 2023), i.e. GPT-NeoX (Black et al., 2022) models. We do not use flash attention (Dao et al., 2022).

Hyperparameters. We use the AdamW optimizer with $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 10^{-8}$, and a weight decay of $0.1$. The maximum learning rate is varied in our experiments: $\{1.5{\times}10^{-4},\ 3{\times}10^{-4},\ 6{\times}10^{-4}\}$. We use cosine learning rate decay to a minimum of $0.1 \cdot \eta_\max$. All warmup lengths are calculated based on the full downstream dataset size (297B tokens). We note that our cosine decay schedule reaches the minimum learning rate at 240B tokens and is constant thereafter. We set gradient clipping to $1.0$. Training is conducted at half-precision (FP16), without dropout.

Three LR strategies in the CPT literature. LLMs are usually trained with Adam (GPT-3, BLOOM, Gopher, Pythia) or AdamW (Chinchilla, LLaMA); in all of these models, the canonical schedule is a linear warm-up followed by cosine decay to 10% of $\eta_\max$. In continual learning (CL), models are trained on sequences of datasets, the data is not i.i.d., and the model can lose plasticity or forget; in such situations, particular monitoring of the LR schedule can be beneficial. Three families have been tried in CL of LMs: constant LR (Ke et al., 2023a; Scialom et al., 2022), progressive decrease across stages (Winata et al., 2023), and warm-up then decrease at each new stage (Caccia et al., 2021). To the best of our knowledge, no existing work has studied specifically the influence of the warm-up phase in LLM CPT — this is our entry point.

How long to warm up? (§4.1) In the literature, warm-up is usually conducted on at most 1% of the data (Zhao et al., 2023). In this experiment, we investigate if the results are sensitive to this hyper-parameter. Setup: we experiment with different warm-up lengths for a schedule of 297B tokens: $\{0\%, 0.5\%, 1\%, 2\%\}$ of the data, and measure performance after the first 50B tokens. From a different perspective, we could see this as running a 1% warm-up on different amounts of data. We hypothesize that warming up for a larger number of iterations could lead to a smoother transition with subsequent performance improvements.

Results — warm-up length is a free knob. The results (Fig. 1) show that the amount of data used for warming up the learning rate does not significantly influence the perplexity on the downstream task (learning) or the upstream task (forgetting). These results invalidate our hypothesis that using more tokens for warm-up can smooth the transition, and show that linear warmup is essentially useless in this setting. Nevertheless, the model trained without any progressive warm-up experiences an initial "chaotic phase" causing a spike in the loss in its first few iterations of training — this phenomenon is also referred to as the stability gap (Lange et al., 2023; Caccia et al., 2022). Takeaway 1: the length of the warm-up phase does not appear to have a significant effect on the Pile and SlimPajama validation losses.

How high to warm up? (§4.2) One objective of re-warming the learning rate is to enable compute-efficient CPT. A learning rate that is too small may lead to inefficient learning on the downstream dataset, whereas a learning rate that is too large may lead to catastrophic forgetting of the upstream dataset. One important aspect of re-warming the LR is to decide how high to increase it. Setup: we fix the length of the warm-up phase to the default amount of 1% of the training data and vary the maximum learning rate. We experiment with the default value of $3{\times}10^{-4}$ used for pre-training Pythia 410M, $1.5{\times}10^{-4}$, and $6{\times}10^{-4}$. For the post-warmup cosine decay phase, we set the final LR to 10% of $\eta_\max$. The LR schedule decays to the minimum at 240B tokens and is constant thereafter. The runs are reported to the end of 240B tokens (the end of decay period).

Results — $\eta_\max$ is the trade-off knob (Figs. 2–4). We observe, at the end of training, that larger $\eta_\max$ improves performance on downstream data while hurting performance on upstream data. Conversely, a smaller $\eta_\max$ improves performance on upstream data while limiting adaptation to downstream data — causing decreased downstream performance. These findings show that altering $\eta_\max$ can be an effective way to trade off downstream vs upstream. We also observe a general trend: fine-tuning on SlimPajama causes the model to forget what has been learned on the Pile, leading to an increase in the Pile validation perplexity. Finally, employing early stopping on a model trained from a constant learning rate (similar to traditional fine-tuning) is an economical way of adapting to the new distribution while retaining strong upstream performance. Takeaway 2: rewarming then decaying the learning rate appears necessary to learn well on the downstream task; while keeping a constant LR is initially advantageous on Pile, this advantage vanishes when training long enough on SlimPajama. Moreover, a model that only learns on SlimPajama performs worse on SlimPajama than models pre-trained on Pile — despite being optimised solely for the downstream task — highlighting positive transfer between the two datasets.

Why mixing matters in CPT. The "new corpus" in CPT is rarely a single homogeneous source — it is itself a weighted mixture of sub-domains. The CPT outcome therefore depends on two mixing decisions: (a) the internal mix of the new corpus, which determines what new capabilities the model picks up; and (b) the proportion of old corpus replayed, which determines how strongly catastrophic forgetting is buffered (covered in §5). This paper fixes (a) to the standard SlimPajama mix and turns off (b) entirely — making it the cleanest possible isolation of the LR-schedule axis. We describe (a) here in detail.

SlimPajama training mix (this paper). The 297B-token SlimPajama training set is sampled with the following weights, almost a 1:1 mirror of the original SlimPajama release: CommonCrawl 67.0% (153.25B), C4 15.0% (78.49B), Github 4.5% (15.41B), Book 4.5% (14.22B), Wikipedia 4.5% (11.78B), Arxiv 2.5% (13.77B), StackExchange 2.0% (9.95B). Roughly $82\%$ of the corpus is general web text (CommonCrawl + C4), while the remaining $18\%$ is code / STEM / encyclopedic / Q&A content. The validation split (~316M tokens) follows the same proportions.

Implicit "soft replay" through source overlap. Because SlimPajama and Pile draw from the same families of sources (CommonCrawl, Wikipedia, GitHub, Books, Arxiv, StackExchange, etc.), CPT on SlimPajama already passively revisits a non-trivial fraction of the Pile distribution. The authors call this out explicitly: "the SlimPajama dataset is built from similar sources as the Pile but with a higher quantity of data. Therefore, some upstream data may be repeated during downstream pre-training." This is, in effect, free implicit replay through provenance overlap — and it is the most plausible explanation for the positive transfer observed in §4.3.

Positive transfer in numbers. A model pre-trained on Pile and then CPT'd on SlimPajama outperforms, at convergence, a model trained only on SlimPajama from scratch on the very same SlimPajama validation set — even though the latter is optimised purely for the downstream task. This is one of the most striking results of the paper: under similar-provenance distribution shift, the Pile-pretrained prior is net positive for a downstream task evaluation, not just an artefact to be unlearned. The conclusion only inverts for the constant low LR baseline: it gives up on adaptation and is overtaken by from-scratch after ~200B tokens.

What this paper does not sweep. The authors fix this mix and do not study how varying it (e.g. up-weighting code, or injecting a math / legal / biomedical domain subset) affects forgetting and downstream gains. This is the gap later filled by domain-CPT works such as DAPT (Gururangan et al., 2020), code-CPT, and math-CPT — and by the mid-training recipes in Llama-3 / DeepSeek-V3 / Qwen-2.5, which simultaneously shift the new-corpus mix toward the target domain and add explicit replay of the original pre-training distribution. Takeaway. CPT data design is a two-axis decision: new-corpus internal mix (capability tilt) + old-corpus replay ratio (forgetting buffer). This paper isolates the LR-schedule axis by holding both data axes fixed; consider this paper a clean unit-test on top of which all later mixing/replay work composes.

What replay is. Replay (a.k.a. rehearsal) is the canonical CL trick: when training on a new stage, mix a small fraction of old data into each batch so the model never fully drops the previous distribution. Classical references are Rebuffi et al. (2017, iCaRL) and Ostapenko et al. (2022, latent replay). In LLMs it shows up as "mix some general pre-training data back into your domain CPT". Mathematically, the per-step batch becomes a mixture $B_t = (1-\rho)\,B_{\text{new}} + \rho\,B_{\text{old}}$, where $\rho$ is the replay ratio; gradient is taken on the mixture instead of on $B_{\text{new}}$ alone, so the optimizer cannot move into a region that catastrophically increases loss on the old distribution.

This paper's stance: no replay, on purpose. The authors explicitly write "We do not use replay." in §2, and motivate it in §1: in a potentially long sequence of CPT stages, the bookkeeping and compute overhead of replay (Lesort et al., 2023) can become a bottleneck, and existing CL techniques like replay or regularisation "may make … not compute efficient enough" in that regime. They therefore isolate the LR-schedule effect, and crucially demonstrate that re-warm alone — without any replay — already beats from-scratch training. This is a strong claim: it says the simplest LR trick is competitive with, and often better than, the entire combinatorial space of replay-based CL methods at LLM scale.

Standard production sweep: 1% / 5% / 10% replay. In real LLM pipelines (Llama-3 mid-training, DeepSeek-V3 / V3.1 long-context CPT, Qwen-2.5 code-CPT, etc.) the typical replay ratio is swept in $\{1\%, 5\%, 10\%\}$ of total batch tokens drawn from the original pre-training distribution. 1% negligibly slows downstream learning while clearly slowing forgetting — the highest cost / benefit point. 5% is the common default for mid-training and domain-CPT — there is visible recovery of upstream PPL at a small downstream cost. 10% almost eliminates forgetting but already dilutes downstream gains; this is reserved for the case where upstream capability is a hard product requirement (e.g. preserving instruction-following or safety alignment after a heavy CPT).

Replay ↔ LR re-warm interaction. Replay and re-warm are complementary, not redundant. Re-warm fixes "LR too small to actually learn the new distribution"; replay fixes "the gradient direction is biased entirely toward the new distribution". Re-warm without replay (this paper) still beats from-scratch but pays a transient upstream loss spike. Replay without re-warm (constant-LR + 5% old data) avoids the spike but learns the new data too slowly. The full combination is what production systems converge to: $\eta_\max \approx 1/3$ of the original peak, 1% linear warm-up (or skip and accept the spike), cosine decay to $0.1\,\eta_\max$, plus $1\%\text{–}5\%$ replay of the original mix.

Takeaway. This paper proves LR re-warm is sufficient to beat from-scratch even without replay; in production, adding a few-percent replay on top is essentially free insurance against catastrophic forgetting — and is what every large-scale CPT recipe in 2024–2025 ends up doing. As a corollary: if you only have budget to tune one knob, tune $\eta_\max$ (this paper, §4.2); if you have budget for two, add a small replay ratio on top; only then consider checkpoint selection or regularisation.

The forgetting picture in CPT. When the Pythia-410M model is CPT'd on SlimPajama, the validation perplexity on Pile — the upstream dataset it was originally trained on — increases during the warm-up and early decay phases, then partially recovers. The size of this increase scales monotonically with $\eta_\max$ — the same knob that controls downstream gains. This is the textbook signature of catastrophic forgetting (French, 1999) in CPT. Importantly the from-scratch baseline, which is being trained on SlimPajama from random init, consistently improves its performance on Pile during training — showing the significant synergy between the two datasets and that "forgetting" is specifically the loss of the prior CPT model's advantage, not a violation of the synergy itself.

Two candidate causes. Two plausible mechanisms could explain the upstream loss spike. (a) Distribution shift: SlimPajama's mix is not identical to Pile's, so optimizing on it pulls the parameters away from Pile's optimum. (b) Optimization dynamics: simply re-warming a previously decayed learning rate can perturb any minimum, regardless of whether the data distribution actually changed. The two stories make different predictions for the case "CPT a Pile-pretrained model back on Pile itself" — distribution-shift theory predicts no spike, dynamics theory predicts the spike persists. The authors disentangle them with a clean ablation.

Result of the ablation (§4.4). Setup: replace the downstream data with 50B tokens of Pile (the same dataset the model was already pre-trained on), apply the same re-warm policy as §4.2, and watch the Pile validation loss. Result: even with zero distribution shift, the Pile validation loss still spikes up during re-warm and the model never fully recovers to its pre-CPT minimum (Figs. 5–6). Therefore at least part of the forgetting in §4.2 is not caused by SlimPajama-vs-Pile differences, but by re-warm itself: it pushes the parameters out of a local minimum, and cosine decay does not land back at the exact same point. Takeaway 3: rewarming the learning rate appears to be a significant cause of the performance degradation; and the models do not appear able to recover from the rewarm-induced hit, even when training on the same dataset.

Interesting fingerprint. When fine-tuning on Pile, Pile-PPL and SlimPajama-PPL move along a near-linear trajectory (Fig. 6); when fine-tuning on SlimPajama, they do not (Fig. 4). A plausible interpretation: same-distribution re-warm climbs out and returns to the same basin (linear in PPL space — the two PPL measures co-vary), whereas distribution-shifted re-warm lands in a different basin (non-linear, asymmetric trade-off curve). This is a useful diagnostic — if you see a non-linear Pile-PPL vs new-PPL curve during CPT, that is direct evidence that the new data is pulling you into a genuinely different optimum, not just temporarily perturbing the current one.

Implications for forgetting mitigation. Because forgetting has both a data component and an optimization component, no single fix is sufficient: (i) shrink $\eta_\max$ to reduce the optimization-dynamics piece; (ii) add replay to attack the distribution-shift piece; (iii) avoid earlier (less-converged) checkpoints — §4.5 confirms they hurt both upstream and downstream. The recipe is multi-pronged, and this paper makes the case that the LR axis alone is already first-order important. The Lange et al. (2023) and Caccia et al. (2022) "stability gap" framing is the deeper theoretical backdrop here: any transition between training regimes provokes a transient capability drop, and the magnitude of that drop is mostly governed by how aggressively the optimizer is perturbed.

Result 1 — warm-up length is a free knob (Fig. 1). Sweeping warm-up over $\{0\%, 0.5\%, 1\%, 2\%\}$ of the 297B-token budget leaves the final Pile and SlimPajama validation losses essentially unchanged. The only visible artefact at $0\%$ is an initial loss spike (the stability gap / chaotic phase), which is gone within a few hundred steps. Takeaway 1: "how long to warm up" is not where the leverage is — pick anything from 0% (and accept a brief spike) to 1%–2% (clean curves), it does not matter for the eventual loss.

Result 2 — $\eta_\max$ is the trade-off knob (Figs. 2–4). At a fixed 1% warm-up, increasing $\eta_\max$ from $1.5{\times}10^{-4}$ to $6{\times}10^{-4}$ monotonically improves SlimPajama PPL and monotonically degrades Pile PPL. The Pareto frontier in (downstream-PPL, upstream-PPL) space is precisely traced out by $\eta_\max$. The constant low LR baseline ($3{\times}10^{-5}$, the decay-tail value) is best on Pile early on but is overtaken on SlimPajama after ~100B tokens. Takeaway 2: if you only have budget to tune one thing in CPT, tune $\eta_\max$ — it is the master plasticity-vs-stability dial.

Result 3 — CPT beats from-scratch even when downstream is huge (Figs. 2–3). A model trained from random init on the same 297B-token SlimPajama with the same cosine schedule loses to every re-warmed CPT model on SlimPajama at convergence — even though the downstream dataset is on the same scale as the upstream pre-training and overlaps with it in source. After ~200B tokens, the constant low-LR CPT model is even overtaken by from-scratch, confirming that "just sit at the decay-tail LR" is the wrong choice in the long run. This is the practical headline of the paper: do not retrain from scratch when new data arrives — CPT with re-warmed LR is strictly better, even at full pre-training scale.

Result 4 — same-data re-warm still hurts (Figs. 5–6). Replicating the §4.2 setup on 50B tokens of Pile (no distribution shift) produces a similar upstream loss spike as in §4.2's SlimPajama setting, and the model never fully recovers to its pre-CPT minimum. Moreover, Pile-PPL and SlimPajama-PPL move along a near-linear trajectory during this same-data re-warm — which was not the case during SlimPajama-CPT. A possible explanation: models trained on Pile climb out of a minimum during warm-up and return towards the same minimum as the LR is decayed (yielding the linear trend); under distribution shift, decay lands them in a different basin. This is the smoking-gun evidence that re-warm is partly an optimization-dynamics effect, independent of distribution shift.

Result 5 — use the latest checkpoint (Fig. 7, §4.5). Three checkpoints are compared as CPT starting points: the fully-converged Pythia-410M at iter 143k, an early checkpoint near the maximum Pile validation loss attained across all Fig. 1 runs (~2.5), and a third one in between. Starting CPT from earlier (less-converged) Pile checkpoints gives no improvement on SlimPajama and actively hurts Pile. The pre-trained model has not lost plasticity at convergence; there is no reason to back off to an earlier checkpoint just for CPT. Takeaway 4: using an earlier checkpoint when pretraining on the Pile does not lead to learning faster on SlimPajama. Practical guidance: always start CPT from the latest, fully-decayed pre-training checkpoint, and re-warm the LR from there.

Data similarity and overlapping. In our experimental setup, upstream and downstream data have a high similarity, notably because of data overlap — Pile and SlimPajama share CommonCrawl / C4 / Wikipedia / Arxiv / GitHub sources. Since in continual learning, different types of shifts can lead to variations in performance (Lesort et al., 2021), our results may not generalize to setups with different distribution shifts, such as language domain adaptation pre-training setups (Xu et al., 2019; Gururangan et al., 2020; Ke et al., 2023a; Chakrabarty et al., 2019; Ke et al., 2023b). Nevertheless, comparing Fig. 4 and Fig. 6, we see that the results are not identical when fine-tuning on the Pile vs on SlimPajama. A possible explanation is that even a slight shift in data distribution can lead to a significant perturbation of the learning dynamics — as also observed for image classification, where Igl et al. (2020) show that a sudden transition of 10%–20% of the labels in the dataset can have a significant impact on downstream performance.

Experiments scale. As described in §2, our investigation explores models of size 410M and a fine-tuning dataset of size 297B tokens. While this is a preliminary study, in future work, we plan to verify whether our conclusions hold at different model scales (e.g. 3B and 7B) and different dataset scales (e.g. 100B and 600B). Moreover, we plan to test our models throughout using benchmarks such as HELM (Liang et al., 2022) or Harness (Gao et al., 2021) instead of only loss or perplexity, as these benchmarks can provide important insight into the evolution of model capabilities. This is non-trivial: perplexity averages over all of a corpus and tends to hide domain-specific capability drops (e.g. coding accuracy, multi-step reasoning) that a benchmark suite would catch immediately.

Conclusion. Our experiments demonstrate that warming up to higher maximum learning rates helps models pre-trained on the Pile adapt to SlimPajama, while a smaller maximum learning rate preserves performance on the Pile. In both cases, however, models that are rewarmed improve over models trained from scratch. These results motivate the use of continual pre-training on new datasets rather than restarting training from scratch. More research is needed, however, to establish similar results for larger model scales, different distribution shifts, and to verify that this strategy can be applied repeatedly to update models.

Reading-time editorial. The lasting contribution of this paper is not any single number; it is the conceptual decoupling of three CPT knobs that the community had been conflating: (i) warm-up length (free), (ii) $\eta_\max$ (the Pareto trade-off), (iii) checkpoint choice (always latest). Once decoupled, the practical rule of thumb collapses to: "pick $\eta_\max$ on the Pareto frontier you care about, give it any 0–2% warm-up, decay cosine to $0.1\,\eta_\max$, start from the latest pre-training checkpoint." Subsequent work — including Ibrahim et al. 2024 "Simple and Scalable Strategies to Continually Pre-train LLMs" (a direct follow-up by the same group), Llama-3 mid-training, DeepSeek-V3 long-context CPT, and Qwen-2.5 code-CPT — adopts this decomposition essentially verbatim and adds explicit replay on top.