The Generalization Problem

Current AI research has mostly converged on a single approach to AGI: gradient-based optimization of large neural networks on massive datasets. Deeper questions about intelligence have taken a backseat to brute empiricism, as theoretical results that aim at the fundamentals, Solomonoff induction [1]Ray J. Solomonoff. A formal theory of inductive inference. Part I. Information and Control, 7(1):1–22, 1964., PAC learning [2]Leslie Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134–1142, 1984., remain uncomputable or utterly impractical.

Can we formulate principles of intelligence that are both universal and practical? How can these principles lead to learning algorithms where generalization scales optimally with compute, the one resource we can control? If we have infinite compute, what are the limits to how intelligent systems can be? We argue that solving these fundamental questions is a promising path to AGI.

The Current State of AI: The Scaling Bottleneck

The bitter lesson says general methods that scale with compute always win [3]Richard S. Sutton. The Bitter Lesson, 2019. URL http://www.incompleteideas.net/IncIdeas/BitterLesson.html.. Yet our current scaling is fundamentally constrained by data availability. Compute grows exponentially and is fully programmable; data grows much slower and remains outside of our control. But neural nets trained with gradient descent scale primarily with data and only indirectly with compute. Recent RL approaches also depend on data generated through pretrained models and verifiers. For most problems where intelligence matters, we lack both infinite data and perfect verification.

It's also worth noting that big labs are likely pursuing RL on the problem of intelligence itself largely using existing approaches. In theory, this is sound: use strong pretrained models to generate code for learning algorithms, test their generalization capabilities at runtime, and bootstrap to better systems. But current models aren't yet strong enough to navigate the vast space of possible algorithms. Going forward, this route requires scaling data through RL environments, then hoping the resulting system discovers the fundamentals of intelligence well enough to build more intelligent systems.

The alternative is to solve the fundamentals of intelligence first.

Towards Optimal Generalization

Intelligence is about generalization. We want generalization to scale with compute independent of data availability. Since data is almost always the limiting factor, the priority is achieving strong performance with minimal training data, though the approach should be able to leverage larger datasets when they exist.

This requirement raises a natural question: what would the optimal learning algorithm look like? We hypothesize that with infinite compute, it would:

• Search over all models consistent with the data
• Use universal generalization priors to select which model generalizes best

The no-free-lunch theorem [4]David H. Wolpert. The lack of a priori distinctions between learning algorithms. Neural Computation, 8(7):1341–1390, 1996. establishes that search alone isn't sufficient for intelligence: we also need priors. For any finite dataset, infinitely many functions fit the data perfectly, each behaving differently on unseen data. Search finds the functions consistent with the data; priors determine which ones generalize. This would inherently scale with compute because search scales with compute.

Search over models

We need ways to search broadly across the model solution space, the parameter space of a neural net with a given architecture. Pure gradient descent follows training loss gradients and converges to one solution without any mechanism to discover alternatives. The farthest we've gotten is training different models with slightly different settings (different seeds, optimizers), but that's a far cry from the exploratory search our hypothesis requires.

Our recent paper [5]Akshay Vegesna and Samip Dahal. Decoupling Search and Learning in Neural Net Training. arXiv preprint arXiv:2509.10973, 2025. URL https://arxiv.org/pdf/2509.10973. takes the first step by decoupling search from learning during training. We perform evolutionary search in representation space, a lower-dimensional manifold where search is feasible, while using gradient descent in parameter space to implement the discovered representations. This separation allows us to discover diverse solutions by scaling compute.

Universal Generalization Priors

Current neural networks embed priors implicitly in their loss landscapes, optimization dynamics, and architectural choices. These implicit priors enable some generalization, but they cannot be optimized directly and the massive data requirements of modern systems suggest these priors are far from optimal. We need to make generalization priors both explicit and universal: they should be directly optimizable and work across all phenomena we want to model.

The classical example is simplicity, formalized in Solomonoff's prior through Kolmogorov complexity [6]Ming Li and Paul Vitányi. An Introduction to Kolmogorov Complexity and Its Applications. Springer, 1997.: simpler programs are more likely to be correct. Solomonoff's prior is uncomputable, and simplicity has proven both difficult to implement and even misleading. Consider the overparameterization phenomenon: recent work suggests overparameterized networks achieve better generalization because they converge to flatter minima with lower effective dimensionality [7]Andrew Gordon Wilson. Deep Learning is Not So Mysterious or Different. arXiv preprint arXiv:2503.02113, 2025. URL https://arxiv.org/pdf/2503.02113.. Lotfi et al. (2024) [8]Sanae Lotfi et al. Non-Vacuous Generalization Bounds for Large Language Models. arXiv preprint arXiv:2312.17173, 2024. URL https://arxiv.org/pdf/2312.17173. further supports that larger language models achieve tighter generalization bounds and are more compressible than smaller ones. But if we used naive parameter count as the measure of simplicity, we would be misled to build smaller models rather than more compressible ones.

We're still early in developing a principled theory of generalization in neural networks that could drive our model search.

Moving Forward

Search and universal generalization priors form the foundation of generalization, and thus intelligence, that scales with compute. Rather than rely on brute empiricism, our research starts from these fundamentals to build learning algorithms that approach the theoretical limits of intelligence.

← Back to Q