Public paper breakdown
How V4 makes a million-token context actually affordable — compressed attention (CSA + HCA), a much smaller KV cache, mHC, and Muon — explained with diagrams, not walls of text.
Based on “DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence.” · Read the original
Reasoning models, big codebases, long tool-use sessions — they all want to keep more in context. But plain attention gets expensive fast as the history grows, in two ways at once.
Every new token has to figure out which of the earlier tokens matter. More history → more comparisons.
The model stores each past token's key and value so it doesn't recompute them. That store grows with the sequence.
The whole idea in one line
Shrink the history, look at only the parts that matter, keep a rough global view, and make deep layers propagate cleanly — so a million tokens fit in a budget you can ship.
V4 isn't a rewrite. Most of it is V3 carried forward — the new work is concentrated in attention, residuals, and training.
One token's path through V4. The teal stages are the new work; the muted ones come straight from V3.
Simplified for teaching — CSA and HCA sit on different layers, not side by side on one.
Every token the model has seen leaves behind a key and a value it reuses instead of recomputing. That growing pile is the KV cache — and at a million tokens it, not the math, is what pins you down.
8 tokens
a sentence — trivial
8,000 tokens
a long doc — noticeable
1,000,000 tokens
memory & bandwidth dominate
Key point
V4's real trick is making that pile smaller and reading less of it per token — everything below is a version of those two moves.
A 30-second refresher, because CSA and HCA are both edits to this loop. For each new token:
Attention(Q, K, V) = softmax( Q · Kᵀ / √(d_k) ) · V
K and V are exactly what the cache stores. The cost of scoring and blending grows with how many entries are in there — which is the whole problem at a million tokens.
CSA makes two moves: squeeze the history into fewer blocks, then let a cheap indexer pick only the blocks worth reading in full.
24 historical KV entries
6 compressed blocks — indexer keeps top-k
recent window
Counts in the diagram are illustrative — the shapes show the mechanism, not the paper's configured numbers.
Key point
Compression cuts how many entries exist; sparsity avoids reading all of them. CSA uses both.
This is the “pick the blocks worth reading” step. It's deliberately cheap — a representative form of its score:
I(t, s) = Σ_j w_j · ReLU( q_j · k_s )
The current token t gets a few tiny indexer-query heads; every compressed block s carries an indexer key. These are separate from — and far cheaper than — the real attention heads.
Each head dot-products with the block's key and passes it through ReLU, so a head can only add to a block's relevance, never subtract.
A learned weight per head sums them into a single number for each block. That number picks what to look at — it isn't the output itself.
Source note
This mirrors the Lightning Indexer from DeepSeek's sparse-attention line (introduced with V3.2); exact head counts and precision live in the report. It selects candidates — it isn't the final attention, and comparing it to search is just an analogy.
Reading another architecture paper this closely? Upload it and get the same treatment — diagrams, notation, worked steps.
Break down a paperHCA takes the opposite bet: compress much harder, end up with only a handful of blocks, and just read all of them. No picking.
24 historical KV entries
2 compressed blocks — all read densely
recent window
CSA vs HCA in one line
CSA: keep more detail, read a chosen slice. HCA: keep a tiny gist, read all of it. “Dense” here means dense over those few blocks — never over every original token.
Because they fail differently. CSA can miss a relevant block; HCA can blur detail. Alternate them layer by layer and each covers for the other — sharp detail from CSA, broad gist from HCA.
Layers alternate ↓
| Property | CSA | HCA |
|---|---|---|
| Compression | Moderate (~×4) | Heavy (~×128) |
| Sparse selection | Yes — top-k | No |
| Core attention over | Only the selected blocks | All of the (few) blocks |
| Recent window | Yes | Yes |
| Its job | Grab the relevant detail | Hold a broad global gist |
| Where it can slip | Indexer misses something | Compression blurs detail |
Compression
Sparse selection
Core attention over
Recent window
Its job
Where it can slip
Versus DeepSeek-V3.2, the report estimates how much compute and cache V4 needs per generated token. Short bars are the point.
Lower is better · DeepSeek-V3.2 = 100% (full track)
compute to make the next token
stored context state
Reported by DeepSeek in the official technical report; not independently reproduced by Deconstructed. FLOPs are precision-normalized — real latency depends on hardware, kernels, batch size, bandwidth, and caching.
Not from CSA alone — it's architecture, precision, and systems work stacked together.
V3.2 was already sparse. V4 pushes further on several fronts at once.
V3.2 already did sparse attention. V4 compresses the KV history first, so the indexer ranks a smaller, pre-summarized set.
HCA contributes a tiny, densely-read summary that sits next to CSA's selective detail.
Alternating CSA and HCA layers gives the model both sharp detail and broad context.
A smaller top-k, mixed-precision KV, a low-precision indexer, and better shared-prefix storage stack on top.
mHC and Muon changed too, but neither shrinks the KV cache — mHC is about residual propagation, Muon is a training optimizer. Different parts of the system.
V4 also changes how signal moves between layers. A normal residual carries one stream forward and adds each block's output. Hyper-Connections use several streams and learn to mix them — more expressive, but unconstrained mixing can destabilize a very deep stack. mHC keeps the mixing matrices doubly stochastic, so you get the extra streams without the instability.
Not attention
mHC doesn't pick context or shrink the cache — it's a residual-connection change between layers. Full mHC breakdown →
V4 trains most weights with Muon (AdamW still handles embeddings, heads, and norms). Roughly: it keeps momentum, orthogonalizes the matrix update, rescales it, then applies it — which the report links to faster, steadier training on matrix-shaped weights.
Training only
Muon is an optimizer. It doesn't run at inference and doesn't create the context window — it just gets you to better weights.
Two configs, trained long, then assembled from specialists.
| Configuration | V4-Pro | V4-Flash |
|---|---|---|
| Transformer layers | 61 | 43 |
| Hidden dimension | 7168 | 4096 |
| CSA compression rate | 4 | 4 |
| HCA compression rate | 128 | 128 |
| CSA top-k | 1024 | 512 |
| Sliding-window size | 128 | 128 |
| mHC expansion factor | 4 | 4 |
| mHC Sinkhorn iterations | 20 | 20 |
| Total parameters | 1.6T | 284B |
| Activated parameters | 49B | 13B |
Transformer layers
Hidden dimension
CSA compression rate
HCA compression rate
CSA top-k
Sliding-window size
mHC expansion factor
mHC Sinkhorn iterations
Total parameters
Activated parameters
Configuration figures reported by DeepSeek in the official technical report.
Pretraining: Flash on 32T tokens, Pro on 33T. Training starts with short sequences and stretches them stage by stage up to a million tokens — so long context is trained in, not bolted on. (“Native” doesn't mean every task benefits from using all million.)
Post-training: build domain specialists, then distill them into one model.
Common mix-up
These “specialists” are separate teacher models. They're not the routed MoE experts inside a single Transformer layer.
The report also describes reasoning-effort modes — Non-think, Think High, Think Max — which spend more test-time compute for harder problems. Same model, different budgets, not different architectures.
Two separate models with different targets — not the same model at two precisions.
| Property | DeepSeek-V4-Pro | DeepSeek-V4-Flash |
|---|---|---|
| Total parameters | 1.6T | 284B |
| Activated per token | 49B | 13B |
| Context length | 1M tokens | 1M tokens |
| Pretraining tokens | 33T | 32T |
| Built for | Max capability | Cheaper inference |
| Knowledge | Higher | Lower (smaller scale) |
| Reasoning | Stronger overall | Competitive on some tasks with more thinking budget |
Total parameters
Activated per token
Context length
Pretraining tokens
Built for
Knowledge
Reasoning
Flash isn't a quantized Pro, and matching Pro on a few reasoning tests doesn't make them equivalent. Pro is bigger and pricier; Flash is smaller and much cheaper to run long-context, but weaker on knowledge-heavy work.
A fair breakdown says where the claims stop. The report does not show that:
The questions that trip most people up on a first read of the V4 report.
Upload it to Deconstructed for section-by-section explanations, notation, equation walkthroughs, and comparisons.
Every claim above traces to one of these. Figures attributed to the report are DeepSeek's own estimates, not independently reproduced.
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (arXiv).
Official model cards and weights on Hugging Face.
The companion architecture for residual-stream mixing.
The step-by-step companion explainer.
Unofficial derivative papers are left out as primary sources. Engram / “Conditional Memory via Scalable Lookup” may show up as related future research, but it isn't part of the released V4 architecture per the main report.
We’re adding new public breakdowns over time. Each one walks through a foundational paper with the same step-by-step style.
DeepSeek-AI, 2025
Residual streams, doubly stochastic mixing matrices, the Birkhoff polytope, and Sinkhorn normalization for stable depth.
Read the breakdownVaswani et al., 2017
Self-attention, Q/K/V, and the scaled dot-product attention equation explained step by step.
Read the breakdownDevlin et al., 2018
Masked language modeling, bidirectional context, and the pre-train / fine-tune recipe that reset NLP.
Hu et al., 2021
Why low-rank adapters work, what gets frozen vs. trained, and the math behind parameter-efficient fine-tuning.
Radford et al., 2021
Contrastive image–text training, the joint embedding space, and how zero-shot transfer falls out of it.
Ho et al., 2020
Forward noise, reverse denoising, and the variational objective that powers modern image synthesis.