Engram: Conditional Memory

Two kinds of sparsity

A dense transformer uses every parameter for every token. Mixture of Experts breaks that: each FFN becomes many experts and a router picks a few per token. The model gets big; the per-token compute stays small. The paper calls this conditional computation.

Engram adds a second axis the authors call conditional memory. Instead of choosing which computation to run, you choose which rows of a very large embedding table to read, using the last few tokens as the key. Reading a row costs O(1). Making the table ten times bigger costs nothing per token.^[1]

MoE conditional computation

Key: the current hidden state, via a learned router.

What's sparse: which expert FFNs run.

Good at: dynamic, context-dependent transformation.

Cost of scaling: more experts still need HBM; routing is decided at runtime.

Engram conditional memory

Key: the last 2–3 token IDs, hashed.

What's sparse: which table rows are read.

Good at: static, local, stereotyped patterns (names, idioms, formulae).

Cost of scaling: addresses are known before the layer runs, so the table can live off-GPU.

The paper's framing is that language has both kinds of work in it. "Diana, Princess of Wales" is a fixed string you should be able to look up. Whether she is the subject or the object of the sentence is something you have to compute.^[1]

How the lookup works

Five steps, all from Section 2 of the paper.^[1]

1. Normalise the tokens. Standard tokenizers give Apple, apple and ␣apple different IDs. Engram collapses them (NFKC, lowercasing, whitespace) into a canonical ID before hashing. On DeepSeek's 128k tokenizer this shrinks the effective vocabulary by 23.43% (Appendix C, Table 6).

2. Take suffix N-grams. At each position, form the 2-gram and 3-gram ending at the current token. Engram-27B uses N up to 3; 4-grams were tried and were slightly worse at a fixed budget (Section 6.2).

3. Hash into prime-sized tables. Each N-gram order gets K hash heads (8 in Engram-27B), each mapping the N-gram to a row in its own table via a multiplicative-XOR hash. Collisions happen; using several heads means two N-grams rarely collide in every head. The retrieved rows are concatenated (Eq. 1–2).

e_t = ‖_n=2..N ‖_k=1..K E_n,k[ φ_n,k(g_t,n) ]

4. Gate against context. The retrieved vector knows nothing about the sentence. The current hidden state h_t, which has been through at least one attention layer, acts as a query. A sigmoid of the normalised dot product gives a scalar gate α in (0, 1). If the memory doesn't fit the context, α goes toward zero and the lookup is suppressed (Eq. 4).

α_t = σ( RMSNorm(h_t)^⊤ RMSNorm(W_K e_t) / √d ) ṽ_t = α_t · W_V e_t

5. Short causal conv, then add. A depthwise causal convolution (kernel 4, dilation = max N-gram order, SiLU) widens the receptive field slightly, then the result is added to the residual stream before that layer's attention and MoE (Eq. 5). The conv is zero-initialised so the module starts as an identity. Engram-27B places the module at layers 2 and 15 of 30 (Section 4.1, Table 5).

Edit the sentence. Tokens are split on spaces and lowercased; this stands in for the real tokenizer.

Each column is a token. Rows show its 2-gram and 3-gram keys, the table slot each key hashes to (two heads per order, tables of 13 and 17 slots so collisions are visible), and a gate value.

2-gram head 3-gram head collision (slot shared with another key in this sentence) gate α

This is an illustration of the mechanism, not the model. The hash is a toy multiplicative-XOR over the token strings, the tables are tiny so you can see collisions, and the gate value is a made-up heuristic (higher when the phrase ends a multi-word name or idiom). The paper's tables have millions of prime-sized slots and the gate is learned.

Why it helps reasoning, not only facts

The intuitive prediction is that a memory module helps knowledge benchmarks. It does: Engram-27B beats the iso-parameter, iso-FLOPs MoE-27B by +3.4 on MMLU and +4.0 on CMMLU. The less intuitive result is that the gains on reasoning and code are larger: BBH +5.0, ARC-Challenge +3.7, HumanEval +3.0, MATH +2.4 (Abstract, Table 1).^[1]

The paper's explanation is that a normal LLM spends its early layers reconstructing static things it could have looked up. Table 3 reproduces an example from Ghandeharioun et al.^[4] showing what the hidden state at the token "Wales" means, layer by layer, in a model with no memory module:

Latent-state translation at "Wales" in "Diana, Princess of Wales", reproduced from Table 3 of the Engram paper (original data: Ghandeharioun et al. 2024).
Layer	What the state decodes to	Authors' reading
1–2	Country in the United Kingdom	Wales
3	Country in Europe	Wales
4	Title held by female sovereigns … or by queens consort	Princess of Wales (unspecific)
5	Title given to the wife of the Prince of Wales	Princess of Wales (unspecific)
6	Diana, Princess of Wales (1961–1997), the first wife of Prince Charles …	Diana, Princess of Wales

Six layers to assemble one entity. If a 3-gram lookup on "Princess of Wales" can hand the model that vector at layer 2, those layers are free for something else. The paper supports this with two measurements: LogitLens KL divergence drops faster in early layers with Engram, and CKA alignment shows Engram's layer 5 representations match roughly layer 12 of the MoE baseline (Figure 4).^[1] I'd treat "effectively deeper" as the authors' interpretation of those two plots rather than a settled fact.

The cleanest evidence for a division of labour is the ablation in Figure 6. Take the trained Engram-27B, switch the memory output off at inference, and see what survives:

Performance retained with Engram output suppressed at inference, Figure 6.
Task family	Retained	Example
Factual knowledge	29–44%	TriviaQA 29%
Reading comprehension	81–93%	C3 93%

The authors note this post-hoc ablation creates a train/inference mismatch, so they only report the two extremes where the signal is clear.

The allocation law

Fix total parameters and per-token FLOPs. Take the inactive parameters (unselected experts, unread rows) and split them: fraction ρ to MoE experts, 1−ρ to Engram. ρ = 1 is pure MoE. The paper sweeps ρ at two compute budgets and finds a U-shape in validation loss (Figure 3, left).^[1]

I haven't redrawn the curve. The paper's HTML doesn't expose the plotted values, and I'd rather show you the numbers the text actually states:

Reported points from Section 3.1, 6×10²⁰ FLOPs regime (P_tot ≈ 9.9B, P_act = 993M).
Allocation ρ (to MoE)	Val loss	Note
100% (pure MoE, 99 experts)	1.7248	baseline
≈80%	1.7109	reported optimum, Δ = 0.0139
≈40% (43 experts)	≈ baseline	"comparable" to pure MoE; no figure given

So roughly 20–25% of the sparse budget to Engram is the sweet spot, and that location was stable across the two budgets tested (ρ ≈ 75–80%). Go too far the other way and you lose the computation you need for context-dependent work; memory can't replace it. Engram-27B uses ρ = 74.3%: 72 experts cut to 55, and the freed parameters become a 5.7B-parameter table (Section 4.1).^[1]

Both sweeps hold P_tot / P_act ≈ 10. The paper doesn't claim the optimum holds at other sparsity ratios or at frontier scale.

The systems angle: HBM vs DRAM

This is the part that got the market commentary going, so it's worth being exact. An MoE router needs the hidden state to decide which experts to fetch, so the experts have to be in fast memory when the layer runs. Engram's addresses depend only on token IDs. You know every row you'll need for layer 15 the moment the sequence arrives. So the rows can be pulled from host DRAM over PCIe while layers 1 through 14 are still computing.^[1]

Table 4 tests the worst case: a 100B-parameter Engram table, entirely in host DRAM, inserted at block 2 of a dense model, with every lookup crossing PCIe. Hardware is one H800; workload is 512 sequences of 100–1024 tokens.

End-to-end inference throughput, Table 4.
Backbone	Baseline tok/s	+100B Engram, CPU offload	Penalty
Dense-4B	9,031.62	8,858.28	1.9%
Dense-8B	6,315.52	6,140.02	2.8%

Under 3% for 100B parameters that never touch HBM. The paper also observes that N-gram frequencies are Zipfian, so a small fraction of rows serves most lookups, and proposes a cache hierarchy: hot rows in HBM, warm in DRAM, cold on NVMe (Section 2.5). Table 4 doesn't use that hierarchy; it forces everything through PCIe, so the authors call it a conservative baseline.

Top track: with prefetch, the PCIe transfer starts when the token IDs are known and finishes before the Engram layer runs, so the GPU never waits. Bottom track: if the table were read only when the layer reached it, the GPU would stall for the whole transfer.

Engram at layer 2

fetch time (layer units) 1.5

Schematic only. Durations aren't measured; the point is that placing Engram deeper widens the window for hiding the fetch, while Section 6.2 shows modelling quality prefers it early. Layer 2 is where the paper landed for the first module.

Three questions I had

Isn't it just a big lookup table?: Mostly, yes, and the authors say so: it "modernizes classic N-gram embedding." Hashed N-gram embeddings go back to FastText (2016); product-key memory layers (Lample et al. 2019)^[5] put large sparse tables inside transformers. What's new here: the rows are learned vectors, not readable facts; the context gate handles collisions and polysemy; the module sits mid-network, which gives the gate something to work with and hides the fetch; and there's an allocation law telling you how much budget to give it. "Just a lookup table" is accurate the way "a transformer is just matmuls" is accurate.
Is it like instinct?: No. Instinct is innate. This is learned from data like everything else in the model. A better analogy is chunking: a chess master sees a board position as a handful of units rather than 32 pieces; a fluent reader takes in "by the way" without parsing the words. Engram supplies a representation for a familiar chunk. It doesn't decide anything. The gate and the layers above decide what to do with it.
Why wasn't this done before?: It was tried: PKM, N-Grammer, OverEncoding, SCONE, Meta's Memory Layers, UltraMem are all cited in Section 7. It stayed marginal for a stack of reasons. MoE was the incumbent way to add sparse capacity. Earlier tables lost fair comparisons; the paper says OverEncoding "fails to yield meaningful improvements on sparse MoE backbones." Tables were put at the input layer, which serialises the fetch and forces them into HBM. There was no context gate. And N-grams were unfashionable. Meanwhile recommender systems have run billion-row hashed embedding tables in host memory for a decade; the paper cites that literature directly.^[1]

What this doesn't mean

YOCO^[3] (Microsoft, 2024) is the other half of that argument. It keeps one shared KV cache for the upper half of the decoder instead of one per layer, so KV memory shrinks by roughly the number of layers you share across. If Engram lowers the HBM needed for parameters and YOCO lowers the HBM needed for KV, you can see why someone would connect them. But the papers were written independently, neither tests the combination, and both leave the bandwidth-hungry parts of inference exactly where they were.

Sources

Cheng, Tian et al., "Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models," arXiv 2601.07372, v2 July 2026. Every Engram number on this page, with the section, figure or table given inline.
deepseek-ai/Engram on GitHub. Code release.
Sun et al., "You Only Cache Once: Decoder-Decoder Architectures for Language Models," arXiv 2405.05254, 2024. YOCO.
Ghandeharioun et al., "Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models," ICML 2024. Source of the Diana example in Table 3.
Lample et al., "Large Memory Layers with Product Keys," NeurIPS 2019. Product-key memory.

The two interactives on this page are schematics with made-up values. Every numeric claim in the prose is from source 1 unless another source is cited.