Q Labs Research

Appendix

APretraining Loss in Detail

Tables A1–A3 give the pretraining loss at every budget for each of the three steps behind Figure 3: our tuned single pass, tuned repetition and our full recipe. Where a domain’s published baseline was trained at the same budgets, we list it too, and the molecular table also includes the ablations of Section 5. Unlike Figure 3, these tables measure multipliers against our tuned single pass, so they show what repetition and the recipe add on top of tuning. Against the published baselines, the multipliers are larger.

For molecules (Table A1), our recipe at 200M unique tokens (0.5676 nats) beats our tuned single pass at 800M (0.5694) and nearly matches it at 1.5B (0.5665). For DNA (Table A2), the recipe is worth 7.5–8.9× the single pass’s data from 10M onward, but only 3.0× at 5M, where its last snapshots start to overfit. For crystals (Table A3), our recipe at 100M (0.3326) beats our tuned single pass trained on the entire 694M-token corpus (0.3387).

Table A1: Molecules: validation loss (nats per token) and data multiplier, seed 0. Our tuned single pass extends to 0.5751 at 400M, 0.5694 at 800M and 0.5665 at 1.5B.
$U$ Tuned Single Pass Tuned Repetition Our Recipe − Ensembling − Distillation − Looping
5M 0.7197 3.00× 0.6295 (5.68×) 0.6414 (3.98×) 0.6330 (5.12×) 0.6322 (5.23×)
10M 0.6745 2.85× 0.6086 (5.45×) 0.6175 (4.06×) 0.6115 (4.87×) 0.6109 (4.96×)
20M 0.6412 2.63× 0.5935 (5.44×) 0.6001 (3.98×) 0.5955 (4.87×) 0.5950 (4.97×)
50M 0.6106 2.67× 0.5799 (5.40×) 0.5842 (3.84×) 0.5808 (5.00×) 0.5813 (4.81×)
100M 0.5949 2.48× 0.5715 (6.20×) 0.5757 (3.82×) 0.5717 (6.08×) 0.5729 (5.25×)
150M — 2.30× 0.5689 (5.95×) 0.5731 (3.42×) 0.5689 (5.91×) 0.5700 (4.98×)
200M 0.5835 2.08× 0.5676 (5.85×) 0.5713 (3.19×) 0.5678 (5.68×) 0.5688 (4.52×)
Table A2: DNA: validation loss on OpenGenome2 held-out genera (nats per base) and data multiplier. Each entry is the mean of three seeds (multiplier range in parentheses). Our tuned single pass extends to 1.2445 at 500M, 1.2342 at 1B and 1.2268 at 2B.
$U$ Tuned Single Pass Tuned Repetition Our Recipe Evo 2 (Scaled Down)
5M 1.3172 1.3039 (3.2×) 1.3047 (3.0×, 2.5–3.4) 1.3129
10M 1.3116 1.2906 (5.7×) 1.2814 (7.5×, 7.1–8.1) 1.3110
20M 1.3002 1.2768 (4.3×) 1.2625 (8.4×, 8.1–8.5) 1.3088
50M 1.2946 1.2612 (3.6×) 1.2463 (8.9×, 8.5–9.2) 1.3069
100M 1.2723 1.2532 (2.9×) 1.2376 (7.9×, 7.9–8.0) 1.3040
200M 1.2590 1.2420 (3.0×) 1.2282 (8.8×, 7.7–9.5) 1.2836
Table A3: Crystals: validation loss on 17,349 deduplicated held-out CIFs (nats per structure token) and data multiplier. Our tuned single pass extends to 0.3590 at 400M and 0.3387 at 694M (the whole corpus). † marks multipliers extrapolated past the final segment.
$U$ Tuned Single Pass Tuned Repetition Our Recipe CrystaLLM
5M 0.4692 0.4333 (3.5×) 0.4162 (7.4×) 0.5687
10M 0.4484 0.4181 (3.3×) 0.3957 (9.3×) 0.4951
25M 0.4241 0.3977 (3.4×) 0.3636 (13.3×) 0.4562
50M 0.4100 0.3761 (4.0×) 0.3456 (11.5×) 0.4318
100M 0.3940 0.3553 (4.4×) 0.3326 (8.2׆) 0.4176
200M 0.3761 0.3394 (3.4×) 0.3201 (5.8׆) 0.4008

BWhat More Compute Can Buy

The main results use one run per budget and average its 64 snapshots. This appendix asks how best to spend more compute: how many snapshots the ensemble needs (B.1), whether separate runs beat snapshots of one run (B.2), and what mixing runs from different configurations adds (B.3). The common theme is diversity. Snapshots of one run are correlated, whereas runs that differ in seed, weight decay, model shape, number of passes or recipe components make more varied errors.

B.1How Many Snapshots

Each snapshot adds a forward pass at inference, and Section 6 measured that cost for DNA. Here we ask, for molecules, how many snapshots the ensemble really needs. We score ensembles of $k$ equally spaced snapshots, ending at the last one, from the same 128-pass runs of the full recipe (Table B1). Most of the gain comes from the first few. At 100M unique tokens, the final snapshot alone is worth 31× the data of the released MLM-Scaling models, four snapshots 50׆, sixteen 56׆ and all 64 also 56׆. Indeed, beyond 16 snapshots the loss changes by at most 0.0003 nats at any budget. Note that the single snapshot here is the last one, whereas the “− ensembling” ablation of Section 5 uses the best one, which is slightly better.

Table B1: Molecules: validation loss and data multiplier for ensembles of $k$ equally spaced snapshots from one full-recipe run (85M model, 128 passes, seed 0). †: extrapolated past Xu et al.’s 4.4B model. At 5M, no ensemble reaches their smallest model (146M tokens), so no multiplier is shown.
$U$ $k=1$ $k=2$ $k=4$ $k=8$ $k=16$ $k=64$
5M 0.6427 0.6344 0.6314 0.6304 0.6297 0.6295
10M 0.6179 (15.5×) 0.6125 (19.6×) 0.6101 (21.7×) 0.6092 (22.6×) 0.6087 (23.1×) 0.6086 (23.2×)
20M 0.6007 (16.4×) 0.5963 (19.9×) 0.5946 (21.4×) 0.5940 (22.2×) 0.5936 (23.1×) 0.5935 (23.3×)
50M 0.5849 (22.7×) 0.5819 (31.5×) 0.5806 (37.0×) 0.5801 (39.4×) 0.5798 (40.6×) 0.5799 (40.4×)
100M 0.5764 (30.9×) 0.5736 (43.3×) 0.5725 (49.9׆) 0.5719 (53.8׆) 0.5715 (55.9׆) 0.5715 (56.2׆)
150M 0.5742 (27.0×) 0.5713 (38.4׆) 0.5700 (45.0׆) 0.5694 (48.7׆) 0.5690 (50.5׆) 0.5689 (51.5׆)
200M 0.5730 (23.5׆) 0.5706 (31.5׆) 0.5691 (37.7׆) 0.5682 (41.9׆) 0.5679 (43.7׆) 0.5676 (45.0׆)

Gradient-based selection. Weighting the snapshots and keeping only the most useful ones can cut inference cost further. We fit one mixture weight per snapshot by minimizing the ensemble’s negative log-likelihood on held-out data,

$$\min_{w \ge 0,\;\sum w_k=1}\;-\!\sum_t \log \sum_k w_k\, p_k(x_t \mid x_{<t}),$$
(B1)

then keep the $K$ snapshots with the largest weights and renormalize. For molecules and crystals, we fit the weights on one half of the validation set, score on the other, and then swap the halves. For DNA, we fit on validation and score on the separate test set (Figure B1).

Sixteen weighted snapshots come within 0.0004 nats of the full 64-snapshot ensemble for molecules and crystals, and match or beat it for DNA: a fourfold saving in inference at almost no cost in loss. Selection helps most where equal weighting struggles. At 5M and 10M for DNA, the equal-weight ensemble scores worse on test than the best single snapshot, and eight weighted snapshots score 0.0086 and 0.0037 nats below it, respectively.

Figure B1: Gradient-based snapshot selection (Equation B1): loss of the K selected and weighted snapshots minus the equal-weight 64-snapshot ensemble, one line per budget. Molecules and crystals: weights fit on half the validation set and scored on the other half. DNA: fit on validation, scored on test.

B.2Snapshots or Separate Runs

Instead of one long run, the same training compute could buy several independent runs, whose errors should be less correlated. We split the 128 passes over $U$ into $K$ = 8, 16 or 32 runs of $128/K$ passes each, every run with its own seed, a cosine schedule, and the snapshot run’s learning rate and weight decay, and average their final probabilities. Because these runs use neither distillation nor looping, we compare them with the snapshot run without those components (“cycles only”, 64 snapshots).

Nevertheless, one run’s snapshots beat every split at every budget (Table B2). Separate runs face a trade-off: with few passes each member is undertrained, and with many each member memorizes. The best split is usually 16 runs of 8 passes, which trails the snapshots by 0.008 nats at 5M and 0.001 at 200M. One run also wins at an equal number of members: at 5M, 16 of its snapshots reach 0.6362 nats, against 0.6473 for 16 separate runs.

Table B2: Molecules: validation loss and data multiplier at a fixed training budget of 128 passes over $U$ (85M model). Left: 64 snapshots from one cycles-only run. Right: probability-averaged ensembles of $K$ independent runs of $128/K$ passes each. Bold: best at each budget. –: worse than MLM-Scaling’s smallest model. †: extrapolated past their 4.4B model.
$U$ 1 Run, 64 Snapshots 8 Runs × 16 Passes 16 Runs × 8 Passes 32 Runs × 4 Passes
5M 0.6357 0.6807 0.6473 0.6439
10M 0.6138 (18.5×) 0.6479 0.6228 0.6217
20M 0.5973 (19.0×) 0.6191 0.6034 (14.6×) 0.6044 (14.0×)
50M 0.5827 (28.7×) 0.5890 (14.9×) 0.5849 (22.7×) 0.5881 (16.4×)
100M 0.5733 (45.0׆) 0.5817 (16.1×) 0.5767 (29.7×) 0.5779 (25.6×)
150M 0.5702 (43.6׆) 0.5761 (21.5×) 0.5723 (33.8׆) 0.5735 (29.3×)
200M 0.5690 (38.0׆) 0.5735 (22.0×) 0.5698 (34.4׆) 0.5709 (30.1׆)

B.3Mixing Runs

Runs that differ in more than the seed, in weight decay, model shape or recipe components, add value on top of snapshots. To combine them, we pool the snapshots of every run at a budget (every second snapshot for DNA) into one mixture. Each snapshot is a member $j$ with next-token prediction $p_j$, and the mixture predicts $p(x_t) = \sum_j w_{t,j}\, p_j(x_t \mid x_{<t})$. Rather than fixing the weights, we start them at a prior $\pi$, fitted with expectation–maximization (EM), and update them after each token of a document:

$$w_{t+1,j} = (1-\alpha)\,\frac{w_{t,j}\,p_j(x_t \mid x_{<t})^{\beta}}{\sum_i w_{t,i}\,p_i(x_t \mid x_{<t})^{\beta}} + \alpha\,\pi_j,$$
(B2)

A member gains weight when it predicts the current document well, so the mixture adapts to each document. $\beta$ sets how strongly each token moves the weights, and $\beta = 0$ recovers a fixed-weight ensemble. $\alpha$ keeps the weights from drifting too far from the prior. We fit $\pi$, $\beta$ and $\alpha$ on data held out from scoring. For crystals, we cross-fit on halves of the held-out CIFs.

Table B3: Crystals: data multipliers on the 17,349 deduplicated held-out CIFs when runs at a budget are mixed (cross-fitted). † marks estimates extrapolated past the relevant single-pass curve.
$U$ Runs One Run Mixture, Fixed Weights Mixture, Adaptive Adaptive, on CrystaLLM’s Curve
5M 8 7.4× 9.3× 9.4× 26.0×
10M 8 9.3× 10.5× 10.7× 27.2×
25M 8 13.3× 15.1× 15.3× 26.7×
50M 8 11.6× 12.1× 12.2× 19.5׆
100M 8 8.3׆ 8.8׆ 9.0׆ 13.5׆
200M 8 5.9׆ 6.4׆ 6.5׆ 9.3׆

DNA. Mixing 8–11 runs raises the multiplier over our tuned single pass from 3.0–8.9× for one run to 10.0–14.0×. The gain is largest at 5M, where one run’s last snapshots overfit (3.0× to 11.5×). Measured against Evo 2 (scaled down), the mixture is worth 24–32×. Adapting the weights within each document matters: with fixed weights, the mixture reaches only 8.7–10.4×. Even three seeds of a single recipe variant reach 8.4–10.4× at 5M–20M.

Crystals. The mixture adds less here, 0.6–2.0× at 5M–25M (Table B3). Most crystal runs share the same tuned model shape and so make similar errors, giving the 8 runs per budget less diversity than the DNA runs. Measured against CrystaLLM, the mixture is worth 19.5–27.2× at 5M–50M.

CTraining Setup

Every domain uses the same recipe and training setup. We train decoder-only transformers from scratch, packing whole documents into rows with a document mask so that the count of unique training tokens $U$ is exact. Muon updates the matrix parameters and AdamW the embeddings and output head, without warmup or dropout. Each run makes many passes over its $U$ unique tokens. After an optional flat phase at the peak learning rate, training is divided into 64 cycles. Within each cycle, the learning rate falls linearly from its peak to $0.08\times$ while weight decay rises from $0.7\times$ to $1\times$, and the weights at the end of the cycle are saved as a snapshot. Predictions average the snapshots’ next-token probabilities. The model is looped: a prelude block, then a core of blocks applied two or three times with shared weights, with the boundary operator of Chen et al.Zixi Chen, Akshay Vegesna, Samip Dahal, and Andrew Gordon Wilson. How model growth, recursion, and boundary operators influence scaling exponents. arXiv preprint arXiv:2609.19107, 2026. URL https://arxiv.org/abs/2609.19107. at each loop boundary, and finally a coda block.

Per domain and budget, we tune only the model size, number of passes, peak learning rate and weight decay. The cycle shape, the 64 snapshots and the loop count stay fixed. Tuned repetition (Appendix A) tunes its number of passes the same way, so both steps give repetition the same freedom. For molecules, the tuned recipe makes 128 passes at every budget. For DNA it makes 32 at 5M, 48 at 10M and 64 from 20M, and for crystals 128 up to 50M, 96 at 100M and 48 at 200M. Of these hyperparameters, the product of learning rate and weight decay matters most. If it is too small, the snapshots memorize the training data. Its best value falls as the budget grows.

Snapshot ensembling and looped depth are used in every domain, whereas the two cross-cycle regularizers vary: chain distillation is used for molecules and DNA, and weight perturbation for molecules and crystals. Section 5 measures each component’s contribution for molecules.

Fine-tuning. All downstream results use LoRA on a frozen pretrained network, so scores reflect what pretraining learned. Our adapters act on the attention query and value projections (and on Evo 2’s Hyena and Mamba input projections), with rank 8 for molecules and crystals and 64 for DNA. A task-specific head is trained alongside them. Where our recipe averages snapshots, each snapshot is fine-tuned separately and their predictions are averaged.

DDownstream Results in Detail

Tables D1–D2 cover zero-shot evaluation, D3–D4 fine-tuning, and D5–D6 comparisons with released external models fine-tuned with our protocol. Each table lists unique pretraining tokens in the first column and the baseline’s and our recipe’s scores beside them, with data multipliers on the right where they apply.

Table D1: DNA zero-shot: ClinVar variant-effect prediction (AUROC). Multiplier: unique data Evo 2 (scaled down) would need for the same score. †: extrapolated past its 2B point. Both models are at chance at 5M and 10M and are omitted.
$U$ Evo 2 (Scaled Down) Our Recipe Multiplier
20M 0.479 0.552 45.7×
50M 0.479 0.652 44.9׆
100M 0.486 0.700 33.7׆
200M 0.494 0.733 22.3׆
500M 0.520
1B 0.557
2B 0.638
Table D2: Crystals zero-shot: structure prediction (match rate of the first of 20 samples per composition). Multiplier: unique data CrystaLLM would need for the same score. †: extrapolated past its 694M point (the whole corpus).
$U$ CrystaLLM Our Recipe Multiplier
5M 0.027 0.313 15.0×
10M 0.169 0.343 20.7×
25M 0.244 0.396 24.1×
50M 0.301 0.421 25.4׆
100M 0.321 0.433 18.4׆
200M 0.341 0.443 12.5׆
400M 0.382
694M 0.401
Table D3: DNA fine-tuning: GUE benchmark (mean test MCC). Our recipe averages its fine-tuned snapshots. “One snapshot” is the best single snapshot. Multiplier: unique data Evo 2 (scaled down) would need for the same mean MCC. Ahead/behind: tasks where our recipe leads or trails Evo 2 (scaled down) by more than 0.01 MCC.
$U$ Our Recipe One Snapshot Evo 2 (Scaled Down) Multiplier Ahead / Behind
5M 0.666 0.625 0.646 4.4× 14 / 5
10M 0.682 0.654 0.654 20.9× 19 / 3
20M 0.706 0.685 0.665 24.0× 22 / 1
50M 0.712 0.694 0.668 13.3× 26 / 0
100M 0.707 0.688 0.670 5.0× 23 / 1
200M 0.715 0.699 0.681 3.8× 23 / 3
Table D4: Crystals fine-tuning: Matbench property prediction (MAE, lower is better). Each cell is the mean MAE across five official folds. CrystaLLM uses its own recipe retrained at each budget. Our recipe averages predictions across fine-tuned snapshots. Bold: best per task and budget.
Task $U$ CrystaLLM Tuned Single Pass Our Recipe
Formation energy (eV) 5M 0.1039 0.1033 0.0573
10M 0.0964 0.0918 0.0548
25M 0.1018 0.0828 0.0504
50M 0.0897 0.0811 0.0487
100M 0.0867 0.0753 0.0467
200M 0.0787 0.0738 0.0474
Log shear modulus 5M 0.1207 0.1138 0.0865
10M 0.1199 0.1110 0.0842
25M 0.1183 0.1050 0.0797
50M 0.1097 0.1018 0.0759
100M 0.1086 0.0966 0.0730
200M 0.1023 0.0951 0.0730
Log bulk modulus 5M 0.0995 0.0908 0.0669
10M 0.0967 0.0877 0.0661
25M 0.0924 0.0829 0.0636
50M 0.0864 0.0790 0.0619
100M 0.0848 0.0770 0.0599
200M 0.0814 0.0760 0.0605
Phonon peak (cm⁻¹) 5M 74.13 66.74 63.86
10M 74.54 65.08 57.40
25M 70.25 60.60 51.28
50M 70.55 65.48 46.30
100M 65.69 59.19 43.21
200M 66.05 61.26 44.02
Table D5: Molecules: comparison with released models, all fine-tuned with our protocol (LoRA, scaffold splits, learning rate chosen on validation). ROC-AUC for classification (higher is better). RMSE for regression (lower is better). Bold: best per row. Our recipe: 200M unique tokens. MLM-Scaling: best released model (4.4B tokens). NovoMolGen: 1.5B ZINC-22 molecules (~70B tokens).
Table D6: DNA: comparison with HybriDNA on GUE per-task test MCC, all fine-tuned with the same protocol (LoRA rank 64, learning rate chosen on validation). Our recipe: 200M unique bases, snapshots averaged. HybriDNA: released checkpoint (161B nucleotides).

← Back to the post