Quadrillion-parameter transformers and how to train them

The 1,000,000x sample efficiency gap between AI and humans is one of the central mysteries of the field, with no good explanation. One clue is model size: Chinchilla’s prescription for 30T tokens is a ~1.5T-parameter model, a tokens per parameter of 20:1 (Hoffmann et al., 2022). Humans, by contrast, have ~100T parameters (counting each synapse as a weight) and learn from just ~200M tokens, a tokens per parameter of 1:500,000. The tokens per parameter gap between AI and humans is thus 10,000,000x, and an important step forward for the sample efficiency gap may be to close it. On 30T tokens, we should be training quintillion-parameter neural networks. Our current networks are way too small, by seven orders of magnitude.

Our new paper (Baron et al., 2026) makes progress towards this vision. Extrapolating its scaling law to a dataset of 5T tokens, the compute-optimal parameter count reaches ~7,000x the standard Chinchilla prescription at 10²⁹ FLOPs. Given enough compute, we give prescriptions for training quadrillion-parameter networks.

Our new paper (Baron et al., 2026) finds that when the dataset is held fixed, the compute-optimal model size keeps growing with the compute budget. In this regime, we also want to increase the number of epochs and the number of ensemble members with the compute budget. By contrast, Chinchilla prescribes a fixed model size with a fixed dataset: 20 tokens per parameter. Extrapolating our paper’s prescriptions to 5T tokens, we find that the ensemble holds over 300 parameters per token at 10²⁹ FLOPs, nearly 7,000x Chinchilla’s parameters-per-token ratio, and the count keeps growing with compute. We should be increasing parameter counts by orders of magnitude to achieve optimal performance in the multi-epoch regime.

Compute is growing much faster than the supply of high quality language training data (Villalobos et al., 2024), so language models will be increasingly trained in a multi-epoch data-constrained regime. To study this regime directly, we train transformers on fixed datasets, across a range of data sizes, model sizes, and epoch count, and fit how loss depends on each. Each run also uses snapshot ensembling, averaging the predictions of checkpoints saved during training (Huang et al., 2017; Mandal et al., 2026), which lowers loss at no extra training compute. While snapshot ensembling is a bit unusual, we find that ensembling is an important technique to integrate in a first-class way within the data-constrained regime.

Concretely, we fit a single scaling law that predicts the loss $L$ from four quantities: the number of unique tokens in the dataset $U$, the model size $N$, the number of epochs $R$, and the number of ensemble members $M$. The law extends the existing data-constrained scaling laws (Muennighoff et al., 2023; Lovelace et al., 2026) with one new term for ensembling:

The first three terms come from Chinchilla: we have an irreducible cost, a term for the size of the model, and a term for the size of the dataset. The fourth term is an overfitting penalty following Lovelace et al. (2026), and the fifth is the gain from ensembling. It is zero for a single model and grows with each added model towards a ceiling. The constants and exponents are fit to our runs.

Prior data-constrained scaling laws hold weight decay fixed across scales (Lovelace et al., 2026). These pre-existing laws predict loss increasing with compute, and don't consider that we can use larger models with more regularization as part of the scaling law. Existing transfer rules don’t fix this problem. MuP covers learning rate, but not weight decay (Yang et al., 2022). The weight decay heuristics contradict each other (Wang and Aitchison, 2025; Bergsma et al., 2025; Qiu et al., 2025), and none account for epochs.

So we fit power laws for how these hyperparameters should scale in terms of unique data, model size, and epochs. We find that optimal weight decay grows steeply with epochs and with model size. With larger compute budgets and fixed datasets, we should train larger models for more epochs. This follows the finding from Kim et al. (2026), but goes further by giving concrete prescriptions for the training parameters.

Furthermore the law extrapolates well. We find that it is near-optimal at 16x of the fitted compute, and within 0.01 nats of the optimal loss on every axis we test. We find that each prior heuristic would miss on at least one axis.

Extrapolated to 5T tokens, the parameter count our data-constrained scaling law prescribes is ~7,000x larger than Chinchilla would predict at 10²⁹ FLOPs, and the models train for several epochs each.

While our models are much bigger than Chinchilla would prescribe, we would like to scale parameter counts further in order to improve sample efficiency. We explore ensembling through the snapshot ensembles of q0 (Mandal et al., 2026), and through deep ensembles (Lakshminarayanan et al., 2017).

With snapshot ensembling, we save checkpoints from different points in a single training run and average their predictions (Huang et al., 2017; Mandal et al., 2026). Since the checkpoints come from one run, this costs no extra training compute. To make the checkpoints diverse, we train with a cyclic learning rate in the second half of training, which repeatedly raises and lowers the learning rate in one-epoch cycles. At the lowest learning rate of each cycle, we observe that the model converges to good solutions. We find that cycling the learning rate is better than holding the learning rate at its peak, and averaging the snapshots’ predictions lowers loss further. We hypothesize that this procedure works because of mode connectivity (Garipov et al., 2018): the good solutions are joined by low-loss paths, so short cycles can reach different strong models without much training.

We also include deep ensembling in our scaling laws. Deep ensembling trains independent models with different seeds and averages their predictions (Lakshminarayanan et al., 2017), so each member costs a full training run. The snapshots from each run are pooled into one ensemble. Unlike adding parameters, which increases overfitting, adding ensemble members reduces it, so we give ensembling its own term in the data-constrained scaling law. The term is zero for a single model, and its gain grows toward a ceiling as members are added. This law fits and extrapolates better than the alternatives, including on runs from Kim et al. (2026).

We find that deep ensembling becomes compute-optimal when the compute budget is large relative to the dataset. Past this point, the optimal number of members grows roughly linearly with compute. So the prescription with limited data is to grow model size and epoch count first with scaled weight decay, including snapshot ensembling in the multi-epoch training. Then we allocate the extra compute to deep ensemble members. In this way, loss continues to improve with more compute, whereas prior recipes eventually get worse.

In order to close the sample efficiency gap between AI and humans, we are motivated to train larger networks than today’s models. From comparison with the brain, we may want to increase the size of current models by seven orders of magnitude. In our latest paper (Baron et al., 2026), we provide prescriptions that increase the size of the network by four orders of magnitude on limited datasets, and give prescriptions showing how to train them. We think we can improve data efficiency by going further, but it will require new algorithms, hardware and more. We’ll have more to share on this soon.