Training Compute-Optimal Large Language Models

The question nobody had answered cleanly

By 2022 the field had a working recipe: take more compute, spend almost all of it on a bigger model, train on whatever token budget happened to be convenient. GPT-3 was 175B parameters on roughly 300B tokens; Gopher was 280B on 300B; Megatron-Turing NLG was 530B on a similar token count. The parameter counts climbed by an order of magnitude while the training data barely moved. Nobody had asked the sharp version of the question: given a fixed compute budget $C$, how should you split it between making the model bigger and training it on more tokens? The two are coupled — training a model of $N$ parameters on $D$ tokens costs approximately $C \approx 6ND$ FLOPs — so for any budget there is a frontier of $(N, D)$ pairs you can afford, and the recipe of the day had quietly assumed the answer was "spend it on $N$."

Three ways to ask, one answer

The paper attacks the allocation question three different ways and gets the same answer from all three, which is what makes the result hard to dismiss. The first approach fixes a set of model sizes and, for each, trains on several token budgets, then reads off the loss-minimizing point for every compute level. The second holds compute fixed and sweeps model size along each iso-FLOP slice, finding the bottom of each valley. The third fits a parametric form for the loss as a function of $N$ and $D$ and solves for the optimum analytically. Across hundreds of training runs spanning several orders of magnitude in compute, the three methods converge on the same rule of thumb.

The rule: model size and training tokens should grow in lockstep. As the compute budget rises, the optimal parameter count $N_{\text{opt}}$ and the optimal token count $D_{\text{opt}}$ each scale as roughly the square root of compute — both proportional to about $C^{0.5}$ — so doubling your budget means making the model about $1.4\times$ bigger and training it on about $1.4\times$ more tokens, not pouring everything into one of the two. A convenient way to remember the implied ratio is that the compute-optimal token count sits at roughly 20 tokens per parameter. The exact crossover depends on the constants fitted from the data, but the equal-scaling shape of the answer is the durable part.

Gopher was undertrained, and so was almost everything else

The uncomfortable corollary is that the large models of the era were sitting in the wrong place on the frontier. They had spent their compute on parameters and starved themselves of tokens. At Gopher's compute budget the compute-optimal configuration was not a 280B model on 300B tokens — it was a model roughly a quarter the size trained on more than four times the data. The same diagnosis applied up and down the list of contemporary giants: they were big because big was the lever everyone reached for, but at their compute they were far from the loss they could have reached by trading parameters for tokens.

Chinchilla: the prediction, cashed in

To prove the analysis rather than merely assert it, DeepMind trained a model exactly where the laws said the optimum was. Chinchilla is 70B parameters trained on 1.4T tokens — a quarter of Gopher's size, more than four times its data, and the same total training compute. If the scaling analysis were wrong, Chinchilla would underperform the model it was derived against. Instead it beat Gopher, and by a wide and consistent margin: across language modeling, reading comprehension, common-sense reasoning, and the MMLU knowledge benchmark, the smaller, longer-trained model came out ahead almost everywhere. The headline was not subtle — a model less than a third the size of the previous flagship was simply better, because it had been fed properly.

Why the result reshaped the field

The practical consequences cut two ways. First, a compute-optimal model is dramatically cheaper to serve: Chinchilla has a quarter of Gopher's parameters, so every forward pass at inference reads a quarter of the weights and runs faster on less memory. The training-time allocation that the paper optimizes for also happens to hand you a smaller artifact to deploy, which matters because inference cost is paid on every query for the model's entire lifetime. Second, the result rerouted where the field spent its effort. After Chinchilla, the token budget stopped being an afterthought. Data collection, filtering, and deduplication — the unglamorous work of assembling trillions of high-quality tokens — became as central as the architecture, because the analysis said tokens were exactly as load-bearing as parameters.

The influence shows up directly in what came next. LLaMA took the Chinchilla framing and pushed past it deliberately: if you care about inference cost rather than training cost, it pays to overtrain a small model well beyond the compute-optimal token count, because the extra training is a one-time expense and the smaller model is permanently cheaper to run. LLaMA-7B trained on 1T tokens — far more than the roughly 20-tokens-per-parameter the compute-optimal point would suggest — is the clearest descendant of this idea. By the time models were training on 15T tokens, the Chinchilla correction had been fully internalized: token count was no longer the variable you left to convenience.

Limitations and what came after

The specific coefficients are fitted from one model family, one architecture, and one data distribution; the equal-scaling shape generalizes well, but the exact crossover point and the 20-tokens-per-parameter figure are not universal constants. The analysis also optimizes a single objective — minimizing pretraining loss at a fixed training-compute budget — and is silent about inference cost, which is exactly the gap LLaMA exploited by intentionally moving off the compute-optimal point. Later work refined the loss-fitting methodology and re-examined some of the parametric estimates, and the rise of heavy post-training (instruction tuning, preference optimization, reasoning RL) means pretraining loss is no longer the only number that matters. But the core lesson has held up: scale data and parameters together, and treat a token budget that lags the parameter count as a model you have not finished training.