Appendix
APretraining Loss in Detail
Tables A1–A3 give the pretraining loss at every budget for each of the three steps behind Figure 3: our tuned single pass, tuned repetition and our full recipe. Where a domain’s published baseline was trained at the same budgets, we list it too, and the molecular table also includes the ablations of Section 5. Unlike Figure 3, these tables measure multipliers against our tuned single pass, so they show what repetition and the recipe add on top of tuning. Against the published baselines, the multipliers are larger.
For molecules (Table A1), our recipe at 200M unique tokens (0.5676 nats) beats our tuned single pass at 800M (0.5694) and nearly matches it at 1.5B (0.5665). For DNA (Table A2), the recipe is worth 7.5–8.9× the single pass’s data from 10M onward, but only 3.0× at 5M, where its last snapshots start to overfit. For crystals (Table A3), our recipe at 100M (0.3326) beats our tuned single pass trained on the entire 694M-token corpus (0.3387).
| $U$ | Tuned Single Pass | Tuned Repetition | Our Recipe | − Ensembling | − Distillation | − Looping |
|---|---|---|---|---|---|---|
| 5M | 0.7197 | 3.00× | 0.6295 (5.68×) | 0.6414 (3.98×) | 0.6330 (5.12×) | 0.6322 (5.23×) |
| 10M | 0.6745 | 2.85× | 0.6086 (5.45×) | 0.6175 (4.06×) | 0.6115 (4.87×) | 0.6109 (4.96×) |
| 20M | 0.6412 | 2.63× | 0.5935 (5.44×) | 0.6001 (3.98×) | 0.5955 (4.87×) | 0.5950 (4.97×) |
| 50M | 0.6106 | 2.67× | 0.5799 (5.40×) | 0.5842 (3.84×) | 0.5808 (5.00×) | 0.5813 (4.81×) |
| 100M | 0.5949 | 2.48× | 0.5715 (6.20×) | 0.5757 (3.82×) | 0.5717 (6.08×) | 0.5729 (5.25×) |
| 150M | — | 2.30× | 0.5689 (5.95×) | 0.5731 (3.42×) | 0.5689 (5.91×) | 0.5700 (4.98×) |
| 200M | 0.5835 | 2.08× | 0.5676 (5.85×) | 0.5713 (3.19×) | 0.5678 (5.68×) | 0.5688 (4.52×) |
| $U$ | Tuned Single Pass | Tuned Repetition | Our Recipe | Evo 2 (Scaled Down) |
|---|---|---|---|---|
| 5M | 1.3172 | 1.3039 (3.2×) | 1.3047 (3.0×, 2.5–3.4) | 1.3129 |
| 10M | 1.3116 | 1.2906 (5.7×) | 1.2814 (7.5×, 7.1–8.1) | 1.3110 |
| 20M | 1.3002 | 1.2768 (4.3×) | 1.2625 (8.4×, 8.1–8.5) | 1.3088 |
| 50M | 1.2946 | 1.2612 (3.6×) | 1.2463 (8.9×, 8.5–9.2) | 1.3069 |
| 100M | 1.2723 | 1.2532 (2.9×) | 1.2376 (7.9×, 7.9–8.0) | 1.3040 |
| 200M | 1.2590 | 1.2420 (3.0×) | 1.2282 (8.8×, 7.7–9.5) | 1.2836 |
| $U$ | Tuned Single Pass | Tuned Repetition | Our Recipe | CrystaLLM |
|---|---|---|---|---|
| 5M | 0.4692 | 0.4333 (3.5×) | 0.4162 (7.4×) | 0.5687 |
| 10M | 0.4484 | 0.4181 (3.3×) | 0.3957 (9.3×) | 0.4951 |
| 25M | 0.4241 | 0.3977 (3.4×) | 0.3636 (13.3×) | 0.4562 |
| 50M | 0.4100 | 0.3761 (4.0×) | 0.3456 (11.5×) | 0.4318 |
| 100M | 0.3940 | 0.3553 (4.4×) | 0.3326 (8.2׆) | 0.4176 |
| 200M | 0.3761 | 0.3394 (3.4×) | 0.3201 (5.8׆) | 0.4008 |
BWhat More Compute Can Buy
The main results use one run per budget and average its 64 snapshots. This appendix asks how best to spend more compute: how many snapshots the ensemble needs (B.1), whether separate runs beat snapshots of one run (B.2), and what mixing runs from different configurations adds (B.3). The common theme is diversity. Snapshots of one run are correlated, whereas runs that differ in seed, weight decay, model shape, number of passes or recipe components make more varied errors.
B.1How Many Snapshots
Each snapshot adds a forward pass at inference, and Section 6 measured that cost for DNA. Here we ask, for molecules, how many snapshots the ensemble really needs. We score ensembles of $k$ equally spaced snapshots, ending at the last one, from the same 128-pass runs of the full recipe (Table B1). Most of the gain comes from the first few. At 100M unique tokens, the final snapshot alone is worth 31× the data of the released MLM-Scaling models, four snapshots 50׆, sixteen 56׆ and all 64 also 56׆. Indeed, beyond 16 snapshots the loss changes by at most 0.0003 nats at any budget. Note that the single snapshot here is the last one, whereas the “− ensembling” ablation of Section 5 uses the best one, which is slightly better.
| $U$ | $k=1$ | $k=2$ | $k=4$ | $k=8$ | $k=16$ | $k=64$ |
|---|---|---|---|---|---|---|
| 5M | 0.6427 | 0.6344 | 0.6314 | 0.6304 | 0.6297 | 0.6295 |
| 10M | 0.6179 (15.5×) | 0.6125 (19.6×) | 0.6101 (21.7×) | 0.6092 (22.6×) | 0.6087 (23.1×) | 0.6086 (23.2×) |
| 20M | 0.6007 (16.4×) | 0.5963 (19.9×) | 0.5946 (21.4×) | 0.5940 (22.2×) | 0.5936 (23.1×) | 0.5935 (23.3×) |
| 50M | 0.5849 (22.7×) | 0.5819 (31.5×) | 0.5806 (37.0×) | 0.5801 (39.4×) | 0.5798 (40.6×) | 0.5799 (40.4×) |
| 100M | 0.5764 (30.9×) | 0.5736 (43.3×) | 0.5725 (49.9׆) | 0.5719 (53.8׆) | 0.5715 (55.9׆) | 0.5715 (56.2׆) |
| 150M | 0.5742 (27.0×) | 0.5713 (38.4׆) | 0.5700 (45.0׆) | 0.5694 (48.7׆) | 0.5690 (50.5׆) | 0.5689 (51.5׆) |
| 200M | 0.5730 (23.5׆) | 0.5706 (31.5׆) | 0.5691 (37.7׆) | 0.5682 (41.9׆) | 0.5679 (43.7׆) | 0.5676 (45.0׆) |
Gradient-based selection. Weighting the snapshots and keeping only the most useful ones can cut inference cost further. We fit one mixture weight per snapshot by minimizing the ensemble’s negative log-likelihood on held-out data,
then keep the $K$ snapshots with the largest weights and renormalize. For molecules and crystals, we fit the weights on one half of the validation set, score on the other, and then swap the halves. For DNA, we fit on validation and score on the separate test set (Figure B1).
Sixteen weighted snapshots come within 0.0004 nats of the full 64-snapshot ensemble for molecules and crystals, and match or beat it for DNA: a fourfold saving in inference at almost no cost in loss. Selection helps most where equal weighting struggles. At 5M and 10M for DNA, the equal-weight ensemble scores worse on test than the best single snapshot, and eight weighted snapshots score 0.0086 and 0.0037 nats below it, respectively.
B.2Snapshots or Separate Runs
Instead of one long run, the same training compute could buy several independent runs, whose errors should be less correlated. We split the 128 passes over $U$ into $K$ = 8, 16 or 32 runs of $128/K$ passes each, every run with its own seed, a cosine schedule, and the snapshot run’s learning rate and weight decay, and average their final probabilities. Because these runs use neither distillation nor looping, we compare them with the snapshot run without those components (“cycles only”, 64 snapshots).
Nevertheless, one run’s snapshots beat every split at every budget (Table B2). Separate runs face a trade-off: with few passes each member is undertrained, and with many each member memorizes. The best split is usually 16 runs of 8 passes, which trails the snapshots by 0.008 nats at 5M and 0.001 at 200M. One run also wins at an equal number of members: at 5M, 16 of its snapshots reach 0.6362 nats, against 0.6473 for 16 separate runs.
| $U$ | 1 Run, 64 Snapshots | 8 Runs × 16 Passes | 16 Runs × 8 Passes | 32 Runs × 4 Passes |
|---|---|---|---|---|
| 5M | 0.6357 | 0.6807 | 0.6473 | 0.6439 |
| 10M | 0.6138 (18.5×) | 0.6479 | 0.6228 | 0.6217 |
| 20M | 0.5973 (19.0×) | 0.6191 | 0.6034 (14.6×) | 0.6044 (14.0×) |
| 50M | 0.5827 (28.7×) | 0.5890 (14.9×) | 0.5849 (22.7×) | 0.5881 (16.4×) |
| 100M | 0.5733 (45.0׆) | 0.5817 (16.1×) | 0.5767 (29.7×) | 0.5779 (25.6×) |
| 150M | 0.5702 (43.6׆) | 0.5761 (21.5×) | 0.5723 (33.8׆) | 0.5735 (29.3×) |
| 200M | 0.5690 (38.0׆) | 0.5735 (22.0×) | 0.5698 (34.4׆) | 0.5709 (30.1׆) |
B.3Mixing Runs
Runs that differ in more than the seed, in weight decay, model shape or recipe components, add value on top of snapshots. To combine them, we pool the snapshots of every run at a budget (every second snapshot for DNA) into one mixture. Each snapshot is a member $j$ with next-token prediction $p_j$, and the mixture predicts $p(x_t) = \sum_j w_{t,j}\, p_j(x_t \mid x_{<t})$. Rather than fixing the weights, we start them at a prior $\pi$, fitted with expectation–maximization (EM), and update them after each token of a document:
A member gains weight when it predicts the current document well, so the mixture adapts to each document. $\beta$ sets how strongly each token moves the weights, and $\beta = 0$ recovers a fixed-weight ensemble. $\alpha$ keeps the weights from drifting too far from the prior. We fit $\pi$, $\beta$ and $\alpha$ on data held out from scoring. For crystals, we cross-fit on halves of the held-out CIFs.
| $U$ | Runs | One Run | Mixture, Fixed Weights | Mixture, Adaptive | Adaptive, on CrystaLLM’s Curve |
|---|---|---|---|---|---|
| 5M | 8 | 7.4× | 9.3× | 9.4× | 26.0× |
| 10M | 8 | 9.3× | 10.5× | 10.7× | 27.2× |
| 25M | 8 | 13.3× | 15.1× | 15.3× | 26.7× |
| 50M | 8 | 11.6× | 12.1× | 12.2× | 19.5׆ |
| 100M | 8 | 8.3׆ | 8.8׆ | 9.0׆ | 13.5׆ |
| 200M | 8 | 5.9׆ | 6.4׆ | 6.5׆ | 9.3׆ |
DNA. Mixing 8–11 runs raises the multiplier over our tuned single pass from 3.0–8.9× for one run to 10.0–14.0×. The gain is largest at 5M, where one run’s last snapshots overfit (3.0× to 11.5×). Measured against Evo 2 (scaled down), the mixture is worth 24–32×. Adapting the weights within each document matters: with fixed weights, the mixture reaches only 8.7–10.4×. Even three seeds of a single recipe variant reach 8.4–10.4× at 5M–20M.
Crystals. The mixture adds less here, 0.6–2.0× at 5M–25M (Table B3). Most crystal runs share the same tuned model shape and so make similar errors, giving the 8 runs per budget less diversity than the DNA runs. Measured against CrystaLLM, the mixture is worth 19.5–27.2× at 5M–50M.
CTraining Setup
Every domain uses the same recipe and training setup. We train decoder-only transformers from scratch, packing whole documents into rows with a document mask so that the count of unique training tokens $U$ is exact. Muon updates the matrix parameters and AdamW the embeddings and output head, without warmup or dropout. Each run makes many passes over its $U$ unique tokens. After an optional flat phase at the peak learning rate, training is divided into 64 cycles. Within each cycle, the learning rate falls linearly from its peak to $0.08\times$ while weight decay rises from $0.7\times$ to $1\times$, and the weights at the end of the cycle are saved as a snapshot. Predictions average the snapshots’ next-token probabilities. The model is looped: a prelude block, then a core of blocks applied two or three times with shared weights, with the boundary operator of Chen et al.Zixi Chen, Akshay Vegesna, Samip Dahal, and Andrew Gordon Wilson. How model growth, recursion, and boundary operators influence scaling exponents. arXiv preprint arXiv:2609.19107, 2026. URL https://arxiv.org/abs/2609.19107. at each loop boundary, and finally a coda block.
Per domain and budget, we tune only the model size, number of passes, peak learning rate and weight decay. The cycle shape, the 64 snapshots and the loop count stay fixed. Tuned repetition (Appendix A) tunes its number of passes the same way, so both steps give repetition the same freedom. For molecules, the tuned recipe makes 128 passes at every budget. For DNA it makes 32 at 5M, 48 at 10M and 64 from 20M, and for crystals 128 up to 50M, 96 at 100M and 48 at 200M. Of these hyperparameters, the product of learning rate and weight decay matters most. If it is too small, the snapshots memorize the training data. Its best value falls as the budget grows.
Snapshot ensembling and looped depth are used in every domain, whereas the two cross-cycle regularizers vary: chain distillation is used for molecules and DNA, and weight perturbation for molecules and crystals. Section 5 measures each component’s contribution for molecules.
Fine-tuning. All downstream results use LoRA on a frozen pretrained network, so scores reflect what pretraining learned. Our adapters act on the attention query and value projections (and on Evo 2’s Hyena and Mamba input projections), with rank 8 for molecules and crystals and 64 for DNA. A task-specific head is trained alongside them. Where our recipe averages snapshots, each snapshot is fine-tuned separately and their predictions are averaged.
DDownstream Results in Detail
Tables D1–D2 cover zero-shot evaluation, D3–D4 fine-tuning, and D5–D6 comparisons with released external models fine-tuned with our protocol. Each table lists unique pretraining tokens in the first column and the baseline’s and our recipe’s scores beside them, with data multipliers on the right where they apply.
| $U$ | Evo 2 (Scaled Down) | Our Recipe | Multiplier |
|---|---|---|---|
| 20M | 0.479 | 0.552 | 45.7× |
| 50M | 0.479 | 0.652 | 44.9׆ |
| 100M | 0.486 | 0.700 | 33.7׆ |
| 200M | 0.494 | 0.733 | 22.3׆ |
| 500M | 0.520 | ||
| 1B | 0.557 | ||
| 2B | 0.638 |
| $U$ | CrystaLLM | Our Recipe | Multiplier |
|---|---|---|---|
| 5M | 0.027 | 0.313 | 15.0× |
| 10M | 0.169 | 0.343 | 20.7× |
| 25M | 0.244 | 0.396 | 24.1× |
| 50M | 0.301 | 0.421 | 25.4׆ |
| 100M | 0.321 | 0.433 | 18.4׆ |
| 200M | 0.341 | 0.443 | 12.5׆ |
| 400M | 0.382 | ||
| 694M | 0.401 |
| $U$ | Our Recipe | One Snapshot | Evo 2 (Scaled Down) | Multiplier | Ahead / Behind |
|---|---|---|---|---|---|
| 5M | 0.666 | 0.625 | 0.646 | 4.4× | 14 / 5 |
| 10M | 0.682 | 0.654 | 0.654 | 20.9× | 19 / 3 |
| 20M | 0.706 | 0.685 | 0.665 | 24.0× | 22 / 1 |
| 50M | 0.712 | 0.694 | 0.668 | 13.3× | 26 / 0 |
| 100M | 0.707 | 0.688 | 0.670 | 5.0× | 23 / 1 |
| 200M | 0.715 | 0.699 | 0.681 | 3.8× | 23 / 3 |
| Task | $U$ | CrystaLLM | Tuned Single Pass | Our Recipe |
|---|---|---|---|---|
| Formation energy (eV) | 5M | 0.1039 | 0.1033 | 0.0573 |
| 10M | 0.0964 | 0.0918 | 0.0548 | |
| 25M | 0.1018 | 0.0828 | 0.0504 | |
| 50M | 0.0897 | 0.0811 | 0.0487 | |
| 100M | 0.0867 | 0.0753 | 0.0467 | |
| 200M | 0.0787 | 0.0738 | 0.0474 | |
| Log shear modulus | 5M | 0.1207 | 0.1138 | 0.0865 |
| 10M | 0.1199 | 0.1110 | 0.0842 | |
| 25M | 0.1183 | 0.1050 | 0.0797 | |
| 50M | 0.1097 | 0.1018 | 0.0759 | |
| 100M | 0.1086 | 0.0966 | 0.0730 | |
| 200M | 0.1023 | 0.0951 | 0.0730 | |
| Log bulk modulus | 5M | 0.0995 | 0.0908 | 0.0669 |
| 10M | 0.0967 | 0.0877 | 0.0661 | |
| 25M | 0.0924 | 0.0829 | 0.0636 | |
| 50M | 0.0864 | 0.0790 | 0.0619 | |
| 100M | 0.0848 | 0.0770 | 0.0599 | |
| 200M | 0.0814 | 0.0760 | 0.0605 | |
| Phonon peak (cm⁻¹) | 5M | 74.13 | 66.74 | 63.86 |
| 10M | 74.54 | 65.08 | 57.40 | |
| 25M | 70.25 | 60.60 | 51.28 | |
| 50M | 70.55 | 65.48 | 46.30 | |
| 100M | 65.69 | 59.19 | 43.21 | |
| 200M | 66.05 | 61.26 | 44.02 |