3.5B
Parameters
Stays the same no matter how many times the core runs.
Recurrent Depth
Same layers, run again. Why a model might loop instead of stack.
Loop instead of stack
Every transformer you've used so far works the same way. A token goes in at the bottom, passes through layer 1, layer 2, and so on up to layer N, once each, and a prediction comes out. Every layer has its own weights. Every token gets the same amount of work, whether the question is "2+2" or a competition maths problem.
A recurrent-depth transformer changes one thing. Instead of N different layers, it has a small block of layers and runs that same block several times. Each pass takes the previous state and updates it. When it's done looping, a few final layers turn the state into a token.
tokens → prelude → [ shared core ↻ r passes ] → coda → next token
si = R(e, si−1) where e = P(x)
The number of passes, r, is a dial. In Huginn, the model I use throughout this page, r was drawn at random during training (mean 32, with a long tail), so the trained model works at whatever r you pick at inference time.[1] That is the whole pitch: more compute per token, with no extra weights and no extra tokens.
The authors call this latent reasoning. I'd put it more cautiously: the model does more work in its hidden state. The benchmark numbers show the extra passes help. They don't show the model is silently working through steps you could write down.
It is not an RNN. An RNN loops along the sequence, one token after another. This loops in depth, while working on a single token. It is not chain of thought. Chain of thought spends compute by writing more tokens, which you can read. This spends compute inside one forward pass, which you can't. Nothing stops a model doing both.
Have a look inside Huginn
Click a component (or use the buttons) to read what it does. Drag r and hit Run flow to watch the state go round the core r times before anything comes out. Keep an eye on the counters: eight stored layers, but far more layer applications.
Data goes through the prelude once, through the same four-layer core r times, and through the coda once. The prelude output e is fed into every pass of the core.
stored layers 8
layer applications 20
unrolled depth 20
Pick a component
Huginn has a 2-layer prelude, a 4-layer core and a 2-layer coda, so eight transformer layers in total are stored. At r = 32 those eight layers get applied 132 times: 2 + 4×32 + 2.[1]
That's real depth in terms of computation. It is not the same as a 132-layer model with 132 sets of weights.
What the paper actually shows
Huginn is a 3.5B-parameter model trained on 795B tokens, and the authors are upfront that it's a proof of concept: one run, public data, learning rate never cooled down.[1]
Here are the standard benchmark numbers at two depths, straight from Table 1. I originally had a chart here. I took it out because I'd interpolated between the published points and drawn a decline past r = 32 that isn't in the data.
795B
Training tokens
One academic run on Frontier. Small by industry standards.
132
Layer applications at r = 32
Eight stored layers; the four in the core run 32 times.
| Benchmark | r = 16 | r = 32 | Change |
|---|---|---|---|
| ARC-Challenge | 37.71 | 38.23 | +0.52 |
| HellaSwag | 64.67 | 65.21 | +0.54 |
| MMLU | 31.25 | 31.38 | +0.13 |
Going from 16 to 32 passes buys very little on these tasks. The paper reports bigger effects elsewhere: on GSM8K, and in a controlled comparison where the recurrent model beat a one-pass twin trained on the same data.[1] What none of it tells you is whether looping beats simply adding more layers at the same compute budget. That question is still open.
The bill
Sharing weights saves memory and bandwidth. It does not save compute. Each pass runs attention and the feed-forward layers again, one after the other, so latency climbs with r. The authors put Huginn's heaviest setting at roughly the FLOPs of a 50B-parameter conventional model. That's a cost figure, not a capability claim.[1]
Training was fragile. Two earlier large runs failed: one collapsed to the same hidden state for every token, the other learned to ignore the recurrence and got the same loss at r = 1 as at r = 32. The run that worked needed a particular normalisation layout, a specific init, and backprop through only the last eight passes.[1]
More passes don't always help. And serving gets harder: tokens that want different depths don't batch neatly, and the KV cache needs special handling.
About Astra
That report is why you're hearing about this now. It isn't evidence for how Astra works or why it scores what it scores. We don't know which parts loop, how many times, or whether it matters.
The safety worry is real, but it's general, not Astra-specific. If useful work moves from visible tokens into hidden activations, there is less to read when you monitor a chain of thought. Two caveats. Chain-of-thought text was never a complete or reliable record to begin with. And when researchers probed Huginn's hidden states for step-by-step arithmetic, they found little.[4] "Hidden work" is the honest description. "Hidden monologue" is not.
Sources
- Geiping et al., "Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach," NeurIPS 2025. The Huginn paper. Everything about architecture, training and results comes from here.
- Dehghani et al., "Universal Transformers," 2018. The same idea, seven years earlier.
- Lan et al., "ALBERT," 2019. Weight sharing across layers at scale, at fixed depth.
- Lu et al., "Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer," 2025. Probes Huginn's hidden states.
- OpenAI, "Safety overview: GPT-6 Astra," 2026. First-party. No architecture details.
- The Information, "OpenAI Technique in 'Astra' Model Sparks Security Concerns," 2026. Paywalled, one anonymous source.
This page explains one architecture and one published proof of concept. It doesn't claim looping is better than stacking, and it doesn't claim to explain Astra.