Towards Universal Data-Efficient Pretraining
@misc{satheesh2026dataefficient,
title = {Towards Universal Data-Efficient Pretraining},
author = {Satheesh, Anirudh and Mandal, Bishwas and Singhal, Madhav and Vegesna, Akshay and Dahal, Samip},
year = {2026},
url = {https://qlabs.sh/research/applications}
}
TL;DR
- We take a step toward a general recipe for data-efficient pretraining. The same method works across molecules, eukaryotic DNA, and crystal structures without being redesigned for each domain.
- Against each domain’s baseline, one training run matches models trained on up to 56× more unique data for molecules and 24× more for both DNA and crystals. This suggests that learning effectively from repeated data is not specific to text.
- These gains persist even against much larger models trained on far more data. After fine-tuning, we achieve state-of-the-art results among autoregressive foundation models for molecules and crystals. We outperform NovoMolGen, trained on about 350× more data, on all eight MoleculeNet tasks, and improve on MatBench tasks compared to CrystalLLM with only 10M unique tokens. For DNA, we match HybriDNA on Genome Understanding Evaluation tasks using about 1/800 as much pretraining data.
1Introduction
Scientific foundation models are increasingly limited by data rather than compute. Compute keeps getting cheaper, while scientific data often remains costly to produce. In drug discovery, candidate compounds must be synthesized and tested to measure their properties. MoleculeNet (Wu et al.), one of the largest public benchmarks for molecular machine learning, contains only 1,500 to 450,000 labeled molecules per task despite decades of experimental work. Crystal data is similarly scarce: just over two million structures are publicly available (Antunes et al.), and simulating one can take hours on a supercomputer. DNA faces a different version of the same constraint. Public sequence repositories such as GenBank contain roughly 60 trillion bases (NCBI), yet only about 2,400 eukaryotic species are represented in RefSeq, the curated, nonredundant reference genome collection (Thibaud-Nissen et al.). The challenge is therefore not acquiring more compute, but making better use of the data that already exists.
When new data is this costly, the usual answer is to reuse what already exists by repeating it. For web text, Muennighoff et al. found that the first few repeated passes are almost as good as new data, but that later ones add little. Subsequent work showed that repetition can go much further. Kim et al. found that with data fixed and compute unconstrained, the best weight decay is about 30× larger than standard practice, and that averaging the outputs of independently trained models (ensembling) improves further still. Further, Baron et al. found that data-constrained pretraining can be improved by tuning weight decay with the number of epochs, and by averaging the outputs of multiple model checkpoints. Mandal et al. introduced a set of techniques designed specifically for many passes over the same data. All of these advances, however, were developed for text, and whether they transfer to scientific domains, where the data looks nothing like English, has not been tested.
We test this question in three domains with almost no shared surface structure: molecules, eukaryotic DNA, and crystal structures. Molecules are represented as SMILES strings, which encode atoms and bonds as characters; DNA as four-letter nucleotide sequences; and crystals as CIF files, which describe atoms and their positions in a repeating lattice. For each domain, we start from an established model and training recipe: MLM-Scaling for molecules, a scaled-down Evo 2 for DNA, and CrystaLLM for crystals. We train on the same data as these baselines and measure gains as a data multiplier: how much unique data the baseline would need to match our performance. This setup lets us test whether a single repeated-data recipe can improve data efficiency across domains without being redesigned for each one.
We make three contributions.
- We show that one repeated-data recipe improves data efficiency across molecules, DNA, and crystals without being redesigned for each domain. It combines snapshot ensembling (Huang et al.; Mandal et al.) with looped depth (Chen et al.). To match our performance, the baseline needs up to 56× more unique data for molecules, 24× more for DNA, and 24× more for crystals.
- These gains transfer downstream. Zero-shot, our models beat scaled-down Evo 2 trained on 40× more DNA data and match CrystaLLM trained on 15–24× more crystal data, eventually surpassing CrystaLLM trained on its full corpus. After fine-tuning, they beat the corresponding baselines across MoleculeNet, GUE, and Matbench. They also beat NovoMolGen on molecules using about 1/350 as much pretraining data and match HybriDNA on GUE using about 1/800 as much.
- Leave-one-out ablations on molecules show that snapshot ensembling and looping drive most of the gain. Looping matters more as the dataset grows, while chain distillation helps most when data is scarcest.
2Method
When the training set is small and must be reused for many passes, a standard model simply memorizes it. Our recipe counters this with two ideas. First, training is split into cycles that each produce a snapshot. At prediction time the snapshots’ next-token probabilities are averaged, so a single run yields an ensemble. Second, the model is looped (Chen et al.): a core of transformer blocks is applied several times with shared weights, adding computation per token without adding memorizable parameters. Chain distillation and weight perturbation further regularize across cycles. The entire recipe is domain-agnostic: the same design applies to molecules, DNA and crystals.
2.1Snapshot Ensembling
A model trained for many passes on a small dataset eventually memorizes it. Snapshot ensembling avoids this by extracting multiple models from one run (Huang et al.; Mandal et al.): we divide training into 64 cycles and save the weights at the end of each (Figure 1). Within a cycle, the learning rate decays linearly from its peak to $0.08\times$ while weight decay rises from $0.7\times$ to $1\times$ in anti-phase. The high-learning-rate, low-weight-decay start of each cycle favors exploration, while the low-learning-rate, high-weight-decay end draws the parameters into a low-norm basin just before the snapshot is saved. The decay drives the model toward a local minimum while the reset at the next cycle pushes it away, so successive snapshots land in different minima. At prediction time we average the $K$ snapshots’ next-token probabilities:
Nothing in this procedure depends on what the tokens represent: every domain uses the same 64 cycles and the same probability averaging. The only variation is when cycling starts: molecular runs cycle from the first step, while DNA and crystal runs hold the learning rate at its peak for the first half and cycle in the second. In the molecular run (Figure 1), the ensemble already improves over the no-snapshot baseline after just 16 epochs, and continues to widen the gap as more snapshots accumulate.
2.2Looped Depth
Looping adds depth without adding parameters. The model has three parts (Chen et al.): a prelude block, a core of blocks applied several times with shared weights, and a coda block. At each loop boundary, the hidden state is normalized and the prelude output $e$ is added back with a learned scalar: $h \leftarrow \mathrm{RMSNorm}(h) + \alpha e$ (Figure 2). This lets a small model act like a much larger one: the 113M-parameter crystal model, for example, executes roughly 300M parameters’ worth of computation per token.
2.3Regularization Across Cycles
Two further components regularize across cycles. Chain distillation makes each cycle learn from the previous snapshot as well as from the data. During cycle $k$, the model minimizes the next-token loss and also matches the temperature-softened predictions of $\theta_{k-1}$, the snapshot from the previous cycle:
where CE is the next-token cross-entropy, $p^{(T)}$ are next-token probabilities at temperature $T$, and $\lambda$ weights the distillation term. Weight perturbation adds Gaussian noise to the weights after each snapshot, so the next cycle starts from a different basin:
where $\sigma$ scales the noise relative to the spread of the weights.
3Pretraining With Repeated Scientific Data
3.1Setup
To test whether one recipe serves any data-constrained domain, we hold everything but the data fixed: the same model family, the same optimizer, the same evaluation. We train decoder-only transformers from scratch in every domain (Table 1). Documents are packed whole with a document mask (FlexAttention; Dong et al.), so the count of unique training tokens $U$ is exact. All models use Muon for matrix parameters and AdamW for embeddings and the output head, without warmup or dropout.
| molecules | eukaryotic DNA | crystals | |
|---|---|---|---|
| corpus | SMILES corpora of Xu et al. | OpenGenome2 eukaryotic genic windows (451 assemblies, 367 genera) | CrystaLLM’s CIFs from MP, OQMD, NOMAD (2.05M structures) |
| vocabulary | 591 tokens | 8 (ACGTN + 3) | 371 tokens |
| unique budgets | 5M–200M | 5M–200M | 5M–200M |
| tuned single pass | 5M–1.5B | 5M–2B | 5M–694M (whole corpus) |
| model size | 85M | 25M–85M | 113M |
| downstream | 8 MoleculeNet tasks, scaffold splits | gene annotation in 6 held-out genera and GUE benchmarks; ClinVar zero-shot | 4 Matbench tasks; zero-shot structure prediction |
| baseline | MLM-Scaling (Xu et al.’s released checkpoints) | Evo 2 (scaled down) | CrystaLLM (its recipe) |
Each domain has one baseline, its established recipe: MLM-Scaling for molecules (the released checkpoints of Xu et al.), Evo 2 scaled down to our model sizes for DNA, and CrystaLLM for crystals. We measure every method in data multipliers: how much unique data the baseline would need to reach the same loss. Unlike raw loss, this unit compares across domains. Let $L_p(U)$ be the validation loss the baseline reaches after one pass over $U$ unique tokens. If a method reaches loss $L(U)$ with the same $U$ tokens, its data multiplier is
A multiplier of 10× at 25M means that 25M tokens trained with our recipe match what the baseline needs 250M tokens to reach. We invert the baseline’s loss curve by linear interpolation in log $U$. When a result surpasses the baseline’s longest run, we extrapolate the curve’s final segment. These multipliers depend on its slope and are marked † throughout.
All scores use held-out data deduplicated against the training set. Per domain and budget, we tune model size, peak learning rate and weight decay for every baseline. The cycle shape, 64 snapshots and loop count are fixed across all recipe runs (Appendix C).
3.2Results
The same recipe gives order-of-magnitude gains in every domain (Figure 3), reaching in each case the loss that the baseline needs 10–50× as much unique data to match. Even a tuned single pass alone is worth up to 7× the baseline’s data for molecules, 6× for DNA and 4× for crystals. The full recipe raises this to 56×, 24× and 24×.
For molecules, our recipe matches the best released MLM-Scaling checkpoint (trained on 4.4B tokens) using about 85M unique tokens, roughly 50× less data. At smaller budgets, MLM-Scaling needs 23–40× as much data to reach the same loss.
For DNA, we train Evo 2 scaled down to our model sizes with its original recipe (AdamW, warmup, cosine decay). Its loss changes little until about 300M tokens. At 100M unique tokens, our recipe already matches the loss Evo 2 reaches at 2B.
For crystals, we retrain CrystaLLM’s recipe ourselves at every budget, because its released checkpoints were trained on structures from our held-out split. From just 50M unique tokens, our recipe reaches a lower loss than CrystaLLM trained on its entire corpus.
4Downstream Transfer
Pretraining gains transfer to real tasks in all three domains. We fine-tune following MLM-Scaling (Xu et al.): the pretrained weights stay frozen while LoRA adapters (Hu et al.) and a task-specific head are trained on top, so scores reflect what pretraining learned. Every model, ours and every baseline, uses the same protocol, splits and hyperparameters. Because our recipe produces 64 snapshots, each is fine-tuned independently and their predictions averaged at test time (Appendix C).
4.1Zero-shot Evaluation
The most direct test of pretraining uses the model with no task-specific training at all. We run such zero-shot tests for DNA and crystals (Figure 4; Tables D1–D2 in Appendix D).
DNA (Figure 4, left). We test whether the model can tell disease-causing mutations in human genes from harmless ones. For each variant in the ClinVar database, we score the DNA sequence with and without the mutation: if the mutation lowers the model's likelihood, the variant is predicted harmful. Evo 2 (scaled down) stays near chance until 500M tokens and reaches 0.638 AUROC at 2B. Our recipe passes that from just 50M unique tokens, more than 40× less data.
Crystals (Figure 4, right). We give the model only a chemical formula (such as BaNiF3) and ask it to generate the full crystal structure, including every atom's position. A structure counts as correct if it matches the ground truth within standard tolerances. At 5–25M unique tokens, our recipe matches CrystaLLM trained on 15–24× more data, and from 50M it beats CrystaLLM trained on its full corpus.
Does a one-letter change in a human gene cause disease?
A cell reads a gene three letters at a time, and each triplet adds one building block (amino acid) to a protein. Here, in the haemoglobin gene HBB, one C has become a T.
Given only a chemical formula, can the model build the crystal?
A crystal repeats one small lattice of atoms in every direction. The model gets the formula BaNiF3, which never appears in training, and must write where every atom in that lattice sits.
4.2Fine-tuning: Molecules
In practice, molecular models are used to predict properties such as toxicity, solubility or activity against a drug target. We fine-tune on eight such tasks from MoleculeNet (Figure 5), five classification and three regression, all on scaffold splits so test molecules are structurally different from those seen during fine-tuning.
Pooled over eight tasks, our recipe at 5M unique tokens already fine-tunes better than the released MLM-Scaling
checkpoint at 146M. At 50M it beats the 1.5B checkpoint, and from 100M it beats their best at 4.4B. Our recipe
at 200M also beats all four released NovoMolGen checkpoints (Chitsaz et
al.), trained on 1.5B ZINC-22 molecules (~70B tokens), on all eight tasks (Table D5).
Does it inhibit BACE-1, an Alzheimer’s drug target? Does it cross the blood–brain barrier into the brain? Did it fail clinical trials because of toxicity? Which of 27 kinds of side effect does it cause? Is it toxic in any of 12 lab screens? How well does it dissolve in water? How favorable is moving it from gas into water (hydration free energy)? How strongly does it prefer oil (octanol) to water at pH 7.4?
4.3Fine-tuning: DNA
For DNA, we ask whether a model pretrained on little data helps with species it never saw. Our first task, gene annotation, labels every base in a window by its role: coding exon, intron, untranslated region or intergenic (Figure 6, top). We test on six genera held out from pretraining, as when annotating a newly sequenced genome. Our recipe is worth up to 40× the baseline’s data, and from 50M it beats Evo 2 (scaled down) trained on 2B tokens. The released HybriDNA (161B nucleotides), fine-tuned with the same protocol, scores 0.639 macro-F1. Our recipe passes it at about 120M unique bases and reaches 0.652 at 200M, roughly 1/800 of HybriDNA’s data.
On the Genome Understanding Evaluation (GUE) benchmark (Zhou et al.), a suite of classification tasks on short sequences, our recipe leads at every budget and is worth 4–24× the baseline’s data (Table D3). At 200M it scores 0.715 mean MCC, above Evo 2 (scaled down) trained on 500M tokens (0.707). The released HybriDNA, fine-tuned identically, also scores 0.715, so our recipe matches it here too (Table D6).
Which part of a gene is each letter of DNA in?
4.4Fine-tuning: Crystals
For crystals, we fine-tune on four Matbench tasks (Dunn et al.). Each gives the model a crystal’s CIF (the text listing its atoms and lattice) and asks for one physical property: formation energy, shear modulus, bulk modulus or phonon frequency (Figure 7, top). With only 1,265–18,928 labeled structures per task, fine-tuning relies heavily on what pretraining learned. We score by mean absolute error. Our recipe beats CrystaLLM at matched data in all 24 task–budget pairs, and from just 10M unique tokens it surpasses CrystaLLM trained on the entire 694M-token corpus on all four tasks.
Is this crystal stable?
How much does it resist shearing?
How hard is it to compress?
How fast can its atoms vibrate?
5Which Components Matter?
Which parts of the recipe produce the gain? To find out, we remove one component at a time from the molecular recipe and retrain the best configuration at each budget (Figure 8). To measure the recipe without ensembling, we take the best single snapshot of the full run instead of the average. Snapshot ensembling and looping carry most of the gain, while chain distillation helps only where data is scarcest.
5.1Snapshot Ensembling Carries the Largest Share
Averaging snapshots gives the largest single gain. From 10M to 200M unique tokens, the best single snapshot is worth 16–34× the data of the released MLM-Scaling models, while the ensemble is worth 23–56×, a further factor of 1.4–1.7. Why does averaging help so much? With 128 passes over their data, the molecular runs have every chance to memorize it. Each cycle ends at a low learning rate, where the weights settle into a solution that fits the training documents closely, including details that do not generalize. The next cycle’s high learning rate then moves the weights away before they settle again. Snapshots therefore memorize different details of the same data but agree on what generalizes, so averaging their probabilities cancels much of the memorized part and keeps the shared signal.
This matches Baron et al., who find on web text that ensembling is a separate axis for scaling when data is limited: adding members as compute grows keeps loss falling, whereas spending more compute on a single model can make loss worse. The same reasoning predicts how the gain should change with scale. Larger models memorize a fixed dataset in fewer passes, so averaging should matter more for them, not less. Conversely, with more unique data each snapshot memorizes less, so the gain should shrink unless the number of passes grows too.
5.2Looping Matters More as Data Grows
Looping helps more as the dataset grows: removing it costs about 9% of the multiplier at 10M–20M and 12–16% from 50M. Recall that a looped core adds computation per token without adding parameters (Section 2.2). When data is limited, extra parameters are a liability, because they let the model memorize the training set sooner. This is why, in our tuning sweeps, the best model size stops growing long before compute runs out. Looping instead spends compute on depth, so the model extracts more from each token without gaining room to memorize. With very little data, a shallow model already learns most of what the data supports, and extra depth changes little. As unique data grows, however, there is more structure to learn, and the depth pays off. Chen et al. likewise find that looping regularizes multi-epoch training.
Because this cost holds steady from 50M upward, we expect looping to keep mattering beyond 200M unique tokens. The loop count, which we fixed per domain, should probably grow with the budget. Moreover, Chen et al. find that looped models become compute-optimal as they grow, so for larger models looping offers a way to scale compute per token without scaling the parameters that memorize.
5.3Chain Distillation Helps Where Data Is Scarcest
Chain distillation adds 11–12% to the multiplier at 10M–50M but 2% or less above. Recall that it pulls each cycle toward the previous snapshot’s predictions (Section 2.3). These soft targets give a full distribution over the next token rather than the single token observed, so they carry more information per token than the data alone, which matters most when data is scarce. However, they also make consecutive snapshots more alike. Once there is enough data for snapshots to differ on their own, this lost diversity offsets the benefit, so distillation stops helping above 50M.
6Inference Costs
Snapshot ensembling has a price: using $K$ snapshots takes $K$ forward passes, so inference costs $K$ times as much. We measure this price on DNA and find it small, because a few snapshots carry most of the gain (Figure 9).
Figure 9 plots held-out test loss against the cost of scoring one million bases. Each line is our recipe at one data budget, with one snapshot on the left and 64 on the right. Snapshots are chosen and weighted on held-out data. Even one snapshot of our 200M run reaches 1.2289 nats for 1.1 GPU-seconds, lower than Evo 2 (scaled down) trained on 2B bases (1.2380 for 2.2): a better model at half the cost, from a tenth of the data. Four snapshots reach 1.2269, matching our 1B-parameter single pass on 2B bases (1.2268) at the same cost of 4.5 GPU-seconds. Beyond four snapshots the lines are nearly flat, so most of the ensemble’s gain costs four forward passes, not 64. Weighting also helps: four weighted snapshots are worth 19.3× Evo 2 (scaled down)’s data, more than an equal average of all 64 (18.8×). Fine-tuning shows the same pattern: on gene annotation, a single snapshot of the 200M run scores 0.637 macro-F1, above the best Evo 2 (scaled down) model at any budget (0.588).
Generation, which produces one base at a time, favors the Evo 2 (scaled down) architecture. Its Hyena filters, the long convolutions Evo 2 uses in place of most attention layers, run as recurrences during generation and so avoid the per-step cost of attention (Table 2). Generating 100k bases takes 1.5 minutes with Evo 2 (scaled down), 2.4 minutes with one snapshot of our 200M run and 3.2 minutes with our 1B single pass. Four snapshots, which match the 1B model’s loss, take 3.1 minutes. Evo 2 (scaled down) is fastest, but its loss is 0.009–0.011 nats worse.
Three further points keep the cost of ensembling down. Snapshots come from a single run, so extra members cost no extra training, and at equal training compute they beat ensembles of independently trained runs (Appendix B). The remaining inference cost could also be distilled away: Kim et al. distill an ensemble of eight independently trained models into a single model 8× smaller that retains 83% of the ensemble’s improvement, and the same technique should apply to snapshots. Finally, the members of an ensemble are independent, so they can run in parallel. Our generation benchmark already batches them into one model (Table 2). We believe custom inference engines built for serving ensembles of small models could speed up inference further while keeping their advantage over a single larger model.
7Conclusion
We show that a single recipe combining snapshot ensembling with looped depth substantially improves data efficiency across molecules, eukaryotic DNA, and crystal structures. One training run is worth up to 56× as much unique data for molecules and 24× for DNA, and crystals. These gains also transfer downstream: after fine-tuning, our models outperform MLM-Scaling (Xu et al.) and NovoMolGen (Chitsaz et al.) on all eight MoleculeNet tasks, despite NovoMolGen using 70B tokens. They match HybriDNA (Ma et al.) on the Genome Understanding Evaluation (GUE) benchmark (Zhou et al.) and surpass it on gene annotation using roughly 1/800 of its pretraining data. They also outperform CrystaLLM (Antunes et al.) on all four Matbench (Dunn et al.) tasks at every data budget.
That the same recipe works across three domains with almost no shared surface structure suggests that data-efficient pretraining is not specific to text. What these domains share is a common learning problem: making better use of sequences that are seen repeatedly during training. This points to repetition itself as a useful axis for developing general pretraining methods. Improvements in ensembling, architecture, or optimization that help models learn from repeated data may therefore transfer across domains.
If these methods can transfer across domains, then pretraining need not be developed in isolation. Data scarcity is often treated as a local problem, with each field or application building its own recipe from scratch. Our results suggest that progress can instead compound across domains. Advances developed in text, where pretraining research is most mature and moves fastest, may provide broadly useful methods for any domain where models must learn effectively from limited data.
8Related Work
Muennighoff et al. describe how the value of training data changes as it is repeated. Baron et al. fit power laws that transfer hyperparameters, in particular weight decay, across model size and number of epochs, and scaling laws for adding ensemble members. Kim et al. study pre-training with fixed data and unlimited compute: tuning regularization for many epochs, scaling parameters and ensemble members, and distilling the ensemble into a smaller model. Together they explain why our weight decay changes with the budget and why snapshot ensembling keeps helping.
Snapshot ensembles (Huang et al.) collect several models from one run using a cyclic schedule (Loshchilov and Hutter). Training a model on an earlier model’s predictions builds on Hinton et al. and Furlanello et al.. Mandal et al. bring these ideas together for repeated-data pretraining. Reusing blocks across depth dates to the Universal Transformer (Dehghani et al.), and Chen et al. introduce the loop-boundary operator used here.
Xu et al. study SMILES language models. Evo 2 (Brixi et al.) is trained on OpenGenome2, the source of our DNA data. CrystaLLM (Antunes et al.) generates crystal structures as CIF text. Our downstream evaluations draw on MoleculeNet (Wu et al.), Matbench (Dunn et al.) and ClinVar (Landrum et al.).
References
Luis M. Antunes, Keith T. Butler, and Ricardo Grau-Crespo. Crystal structure generation with autoregressive large language modeling. Nature Communications, 15 (1): 10570, 2024. doi: 10.1038/s41467-024-54639-7.
Ethan Baron, Shmuel Berman, Anirudh Satheesh, Akshay Vegesna, Samip Dahal, and Andrew Gordon Wilson. Improving data-constrained scaling with hyperparameter transfer and ensembling. arXiv preprint arXiv:2610.37842, 2026. URL https://arxiv.org/abs/2610.37842.
Garyk Brixi, Matthew G. Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel A. Gonzalez, Samuel H. King, David B. Li, Aditi T. Merchant, Mohsen Naghipourfar, Eric Nguyen, Chiara Ricci-Tam, David W. Romero, Gwanggyu Sun, Ali Taghibakshi, Anton Vorontsov, Brandon Yang, Myra Deng, Liv Gorton, Nam Nguyen, Nicholas K. Wang, Etowah Adams, Stephen A. Baccus, Steven Dillmann, Stefano Ermon, Daniel Guo, Rajesh Ilango, Ken Janik, Amy X. Lu, Reshma Mehta, Mohammad R.K. Mofrad, Madelena Y. Ng, Jaspreet Pannu, Christopher Ré, Jonathan C. Schmok, John St. John, Jeremy Sullivan, Kevin Zhu, Greg Zynda, Daniel Balsam, Patrick Collison, Anthony B. Costa, Tina Hernandez-Boussard, Eric Ho, Ming-Yu Liu, Thomas McGrath, Kimberly Powell, Dave P. Burke, Hani Goodarzi, Patrick D. Hsu, and Brian L. Hie. Genome modeling and design across all domains of life with Evo 2. bioRxiv, 2025. doi: 10.1101/2025.02.18.638918.
Zixi Chen, Akshay Vegesna, Samip Dahal, and Andrew Gordon Wilson. How model growth, recursion, and boundary operators influence scaling exponents. arXiv preprint arXiv:2609.19107, 2026. URL https://arxiv.org/abs/2609.19107.
Kamran Chitsaz, Roshan Balaji, Quentin Fournier, Nirav Pravinbhai Bhatt, and Sarath Chandar. NovoMolGen: Rethinking molecular language model pretraining. arXiv preprint arXiv:2508.13408, 2025. URL https://arxiv.org/abs/2508.13408.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019.
Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex Attention: A programming model for generating optimized attention kernels. arXiv preprint arXiv:2412.05496, 2024. URL https://arxiv.org/abs/2412.05496.
Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain. Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm. npj Computational Materials, 6 (1): 138, 2020. doi: 10.1038/s41524-020-00406-3.
Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pages 1607–1616, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. URL https://arxiv.org/abs/1503.02531.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. Snapshot ensembles: Train 1, get M for free. In International Conference on Learning Representations, 2017.
Konwoo Kim, Suhas Kotha, Percy Liang, and Tatsunori Hashimoto. Pre-training under infinite compute. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2509.14786.
Melissa J. Landrum, Jennifer M. Lee, Mark Benson, Garth R. Brown, Chen Chao, Shanmuga Chitipiralla, Baoshan Gu, Jennifer Hart, Douglas Hoffman, Wonhee Jang, Karen Karapetyan, Kenneth Katz, Chunlei Liu, Zenith Maddipatla, Adriana Malheiro, Kurt McDaniel, Michael Ovetsky, George Riley, George Zhou, J. Bradley Holmes, Brandi L. Kattman, and Donna R. Maglott. ClinVar: improving access to variant interpretations and supporting evidence. Nucleic Acids Research, 46 (D1): D1062–D1067, 2018. doi: 10.1093/nar/gkx1153.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Mingqian Ma, Guoqing Liu, Chuan Cao, Pan Deng, Tri Dao, Albert Gu, Peiran Jin, Zhao Yang, Yingce Xia, Renqian Luo, Pipi Hu, Zun Wang, Yuan-Jyue Chen, Haiguang Liu, and Tao Qin. HybriDNA: A hybrid Transformer-Mamba2 long-range DNA language model. arXiv preprint arXiv:2502.10807, 2025. URL https://arxiv.org/abs/2502.10807.
Bishwas Mandal, Shmuel Berman, Akshay Vegesna, and Samip Dahal. q0: Primitives for hyper-epoch pretraining. arXiv preprint arXiv:2606.03938, 2026. URL https://arxiv.org/abs/2606.03938.
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Advances in Neural Information Processing Systems, volume 36, pages 50358–50376, 2023.
National Center for Biotechnology Information. GenBank release 274.0, October 15, 2026. URL https://ncbiinsights.ncbi.nlm.nih.gov/2026/10/19/genbank-release-274/.
F. Thibaud-Nissen, E. W. Sayers, P. Akuiyibo, J. R. Brister, G. M. Boratyn, C. Camacho, L. Frisse, J. Hoffman, W. Klimke, V. K. Kodali, A. Marchler-Bauer, T. D. Murphy, N. A. O’Leary, C. O’Sullivan, S. Pujar, S. H. Rangwala, K. Sauls, C. L. Schoch, B. A. Underwood, J. Wang, L. Yankie, J. Ye, I. Karsch-Mizrachi, V. A. Schneider, and K. D. Pruitt. Sequence resources of the National Center for Biotechnology Information in 2027. Nucleic Acids Research, 2026. doi: 10.1093/nar/gkag1159.
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: a benchmark for molecular machine learning. Chemical Science, 9 (2): 513–530, 2018. doi: 10.1039/C7SC02664A.
Dong Xu, Qihua Pan, Sisi Yuan, Jianqiang Li, Zexuan Zhu, and Junkai Ji. Unveiling scaling behaviors in molecular language models: Effects of model size, data, and representation. arXiv preprint arXiv:2601.22757, 2026. URL https://arxiv.org/abs/2601.22757.
Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. DNABERT-2: Efficient foundation model and benchmark for multi-species genome. In International Conference on Learning Representations, 2024.