Computational depth is all you need: Towards 10⁷-layer neural nets

TL;DR

Depth stopped scaling: frontier models sit near 100 layers, where GPT-3 was six years ago.

Deeper keeps winning: contrastive RL agents gain new capabilities all the way to 1,024 layers (Wang et al., 2025).

Our language models keep improving to 128 layers at fixed width, past frontier depths, with just 1B tokens.

Increasing computational depth yields compute-efficiency gains: 1.6× at 10²⁰ FLOPs, projected 3.1× at 10²⁶.

1.Introduction

Since GPT-3's release six years ago, frontier labs have scaled test-time reasoning by 10,000x. Likewise, parameter counts, dataset sizes, and sparsity through MoE have each been scaled by orders of magnitude. Depth, meanwhile, hasn't scaled at all: frontier models are all around 100 layers, the same as six years ago. We have stopped doing deep learning.

A model's expressive power depends on its computational depth: the number of layers influencing the predicted distribution, distinguished from the raw layer count since many transformer layers typically contribute very little (Sun et al., 2025). We should aim to scale this quantity directly. Current architectures are not designed for very large computational depths, and we believe this caps their expressive power. When transformers replaced LSTMs, they unlocked long context. For the first time, per-token loss improves well beyond a thousand tokens, whereas LSTMs' per-token loss plateaus after a hundred tokens (Kaplan et al., 2020). There may be a similar opportunity in architectures that scale computational depth by orders of magnitude beyond today’s models without plateauing.

Quantifying precisely how much a layer influences the predicted distribution is a hard problem, so computational depth can only be measured via imperfect proxies. We find that a proxy built on the logit lens (nostalgebraist, 2020) works well empirically: it takes the first block after the KL peak at which the decoded prediction is within a KL threshold of the model’s final output.

Concretely, let $h_\ell$ be the residual stream after block $\ell$ of an $L$-block transformer. The logit lens decodes any intermediate state with the model’s own unembedding $W_U$, giving a prediction at every block:

$$q_\ell = \operatorname{softmax}\!\big(\operatorname{LayerNorm}[h_\ell]\,W_U\big), \qquad p = q_L .$$

The proxy $d_{\mathrm{lens}}$ is then the first block after the KL peak $\ell^\star = \arg\max_\ell D_{\mathrm{KL}}(p \,\|\, q_\ell)$ whose prediction is within a threshold $\varepsilon$ of the final output:

$$d_{\mathrm{lens}} = \min\big\{\, \ell > \ell^\star \;:\; D_{\mathrm{KL}}(p \,\|\, q_\ell) \le \varepsilon \,\big\}.$$

We find that increasing even proxies for the computational depth improves compute efficiency.

2.End-to-end learning

The field today prefers chain of thought in order to improve the expressive power rather than depth. Intermediate reasoning is externalized as text and fed back in, so the model gets more serial steps of computation not by being deeper but by running longer. This is sub-optimal for two reasons.

First, in decoding to language on every token during the chain of thought, we force the ~100,000-bit hidden state of the final layer into a ~17-bit token. For ideal expressive power, we should prefer depth to compose many steps of computation without this wasteful decoding step.

Second, the language prior enforced in the chain of thought is artificial and restrictive for the network’s search space. In continuous latent space, there are no such restrictions and so the model can learn optimal representations. This improved expressiveness may lead to the emergence of better reasoning structures. Recent interpretability work shows that transformers develop an internal global workspace (the J-space) that chains reasoning across depth without decoding to tokens (Gurnee et al., 2026). This workspace emerged from end-to-end gradient-based learning, not from a chain of thought in language.

We’ve seen this pattern many times in the history of deep learning. Whenever there is a handcrafted human prior, moving to an end-to-end learned approach works much better. Chain of thought fixes the reasoning representation as language; depth lets it be learned. Depth is the obvious bitter-lesson-pilled bet to make.

However, as we describe in Section 4, it’s important to disentangle reasoning in chain of thought and depth for monitorability purposes. We should scale both simultaneously, at least until we are good at verbalizing latent reasoning (Gurnee et al., 2026).

3.Depth is the missing scaling axis

Compute-Optimal Frontier

With the right architecture, depth is a useful scaling axis. We identify two techniques that raise the compute-optimal computational depth: model growth and a boundary operator. Models trained with them have compute efficiency gains over standard models that grow exponentially with each order of magnitude of compute, from 1.6× at 10²⁰ FLOPs to a projected 3.1× at 10²⁶ FLOPs (Chen et al., 2026). We can see from the graph that the depth frontier is far beyond the 100s of layers with which current models are trained.

Model growth increases the size of a model partway through training. We find that it is useful because neural networks fit simple structure early in training, and more complex structure later in training, so increasing depth only late in training makes better use of the compute budget and makes the final models significantly deeper than standard models.

A boundary operator, which simply normalizes the residual stream and re-injects an earlier block, keeps later layers anchored to the input and similarly improves the computational depth. Standard transformers are prone to the curse of depth (Sun et al., 2025), a phenomenon where deeper blocks drift to doing nothing, and the boundary operator mitigates this effect. We find that very deep networks fail to train properly without the boundary operator; current architectures are not built for large depths.

It is the conventional wisdom that changes to the model architecture only affect the scaling constants, not the exponent (Hestness et al., 2017; Bansal et al., 2022). We show that our architecture improvements increasing computational depth can actually improve the scaling exponent. More precisely, the compute-optimal frontier of an architecture is a power law in training compute $C$ (Hoffmann et al., 2022),

where $E$ is the irreducible loss of the standard transformer, shared across architectures, $C_0$ is the scaling constant, and $\gamma$ is the scaling exponent. On FineWeb, we fit an exponent $\gamma_{\mathrm{vanilla}} = 0.1112$ on the scaling ladder from the standard transformer, and an exponent $\gamma_{\mathrm{growth}} = 0.1168$ on the scaling ladder from the model growth and boundary operator architecture (Chen et al., 2026). The compute multiplier, the factor by which the standard transformer's compute budget $C$ exceeds the compute the variant needs to achieve the same loss, scales as

so the multiplier grows by $10^{0.048} \approx 1.12\times$ with every order of magnitude of compute. Our better scaling exponent thus makes the multiplier grow exponentially with each order of magnitude of compute, which is how 1.6× at $10^{20}$ FLOPs becomes 3.1× at $10^{26}$.

We find that model growth and the boundary operator are both important to achieve these gains.

Infinite Compute Frontier

As we scale depth, we often see useful new emergent capabilities. Wang et al. (2025) scale contrastive RL agents from 4 to 1024 layers on humanoid tasks and see qualitatively new behavior at each jump: the depth-4 agent falls over or throws itself at the goal, the depth-16 agent walks upright, the depth-64 agent reaches the maze wall and collapses against it, and the depth-256 agent folds itself over the wall. We build on their codebase and observe the same in Figure 3. While the conventional wisdom is that shallow architectures (2-5 layers) are best for RL, they observe the performance is still rising at 1024 layers, and they stop only for compute reasons.

We also observe the same phenomenon in LLM training runs. We train a standard transformer with a boundary operator every 8 layers that re-injects an earlier block into the residual stream, keeping later layers anchored to the input. The boundary operator is necessary to get continued returns from depth scaling due to the curse of depth. When we hold the width fixed and scale from 4 to 128 layers, the loss keeps dropping beyond the depths many frontier models use today.

4.Chain-of-thought monitorability

We view chain of thought and computational depth as independent axes of scaling. We aim to have a large computational depth per token in our experiments, and maintain chain of thought as a separate axis of scaling test-time compute across tokens. This is critical for chain-of-thought monitorability. We also expect to see better alignment using a larger computational depth because it allows models to generalize more flexibly from their training data, including alignment data. Models that generalize better have been more aligned and hallucinate less (OpenAI, 2026).

5.Towards 10⁷-layer networks

10⁷-layer networks could have transformative emergent capabilities, but no one is trying to build them. The entire stack (model architecture, learning algorithm, inference) needs to be rebuilt around the requirement to have a large computational depth. For example, we will want to revisit test-time scaling with looped models, since they offer a way to scale depth indefinitely without increasing the number of stored weights; more on this soon. We think that scaling computational depth is one of the most exciting projects over the next few years.