Public paper breakdown · DeepSeek-V4 companion
Understand residual connections, Hyper-Connections, doubly stochastic mixing matrices, the Birkhoff polytope, Sinkhorn normalization, and why mHC improves deep-network stability.
A companion architecture used by DeepSeek-V4 to strengthen signal propagation through deep Transformer layers. · Read the original
mHC sits at the intersection of two ideas: the residual connections that make deep networks trainable, and Hyper-Connections that make residual routing richer. The tension between them is the problem.
Preserve an identity path, add a transformed branch, and help gradients and activations propagate through deep networks.
Expand the residual stream into multiple streams and learn richer connectivity and mixing between them.
Unconstrained mixing can disrupt the stabilizing identity behavior, deep stacks can become numerically unstable, and memory-access overhead can increase.
Can the model retain the expressivity of multiple residual streams while constraining their mixing enough to remain stable?
Core idea, in one sentence
Project the residual mixing matrix onto the set of doubly stochastic matrices so each update behaves like a controlled, non-expansive mixture of residual streams.
The rest of this page builds up to that one sentence: what residual streams are, what doubly stochastic means, how Sinkhorn normalization produces such a matrix, and why constraining the mixing this way keeps very deep networks stable.
Each is defined inline so the core explanation stays self-contained.
Everything starts here. A residual connection adds a layer's input back to its transformed output.
x_{l+1} = x_l + F(x_l)
The input to layer l is carried forward unchanged. This direct path is what lets gradients and activations reach deep layers without vanishing.
The sublayer — an attention block or a feed-forward block — computes an update from the input.
The layer's output is input plus update. Because the identity term is always present, optimization stays well-behaved even as the network gets very deep.
Hyper-Connections generalize the single residual stream into several, with learned mixing at each step.
This adds expressivity — but when the mixing is unconstrained and applied over and over through a deep stack, it can become unstable. That instability is exactly what the manifold constraint targets.
This is the single most important definition on the page. A doubly stochastic matrix satisfies three simple conditions.
P[i,j] ≥ 0, every row sums to 1, every column sums to 1
Every entry is zero or positive, so mixing is a convex combination — a genuine average of streams, never a signed amplification.
Each output stream is a weighted average of the available input streams; its total weight is exactly one.
Each source stream is neither systematically amplified nor discarded across the outputs — its contribution is conserved.
Worked example
Both rows sum to 1, both columns sum to 1, and every entry is non-negative:
Each row says how one output stream combines the available input streams; each column constraint prevents any source stream from being systematically amplified or discarded. “Stochastic” here is a matrix property — it does not mean the layer randomly samples a route during inference.
The set of all doubly stochastic matrices forms a geometric object called the Birkhoff polytope. The terminology matters less than the practical idea: mHC restricts the learned mixing matrix to this well-behaved set rather than allowing arbitrary transformations.
Source note
Restricting the matrix to the Birkhoff polytope makes each mixing step better-behaved. That is a design goal, not a proof that all training instability disappears.
How does the model actually build a doubly stochastic matrix from a freely learned one? It alternately normalizes rows and columns.
Worked example
Start from an unconstrained score matrix:
The exact values are made positive and then repeatedly row- and column-normalized until both rows and columns approximately sum to one.
One row normalization followed by one column normalization is generally not enough; the steps are repeated until the matrix approaches the doubly stochastic constraints.
When a deep network repeatedly applies unconstrained residual-stream mixing, small distortions can compound across layers. Constraining the mixing matrix changes that.
On spectral norm and non-expansiveness
A doubly stochastic mixing matrix is non-expansive in a way that keeps residual updates from blowing up — intuitively, it averages rather than amplifies. Treat the exact operator-norm statement as a mathematical property of such matrices and the “training is more stable” conclusion as an empirical claim supported by the paper, not the same thing.
Putting it together, a conceptual mHC layer processes multiple residual streams like this.
Several parallel streams enter the layer
Doubly stochastic mix selects the input to the sublayer
The usual Transformer sublayer runs on the mixture
Old streams and the new output are recombined with constrained matrices
Passed on to the next layer
Several details are simplified here for teaching. The essential change from a standard residual block is that both the pre-mixing and the post-mixing use constrained, doubly stochastic matrices.
Step by step
Where mHC sits between the classic residual connection and unconstrained Hyper-Connections.
| Property | Standard residual | Hyper-Connections | mHC |
|---|---|---|---|
| Residual streams | One | Multiple | Multiple |
| Mixing | Fixed addition | Learned and unconstrained | Learned and constrained |
| Expressivity | Lower | Higher | Higher |
| Stability mechanism | Identity path | Weakened by arbitrary mixing | Doubly stochastic projection |
| Implementation complexity | Low | Higher | Higher |
| Primary goal | Stable depth | Richer connectivity | Richer connectivity with controlled propagation |
Residual streams
Mixing
Expressivity
Stability mechanism
Implementation complexity
Primary goal
mHC does not dominate standard residual connections in every architecture or setting; it targets stable, expressive propagation in very deep sparse models.
DeepSeek-V4 is a very deep and large sparse model. Its attention and MoE components create complex transformations across many layers.
mHC is intended to improve how signals propagate through that depth by maintaining multiple residual streams, allowing learned information routing between them, constraining the routing to a stable geometric set, and reducing the risk that repeated residual mixing destabilizes training.
Keep the roles straight
Questions that trip up most readers on a first pass through the mHC paper.
Upload it to Deconstructed for section-by-section explanations, notation definitions, equation walkthroughs, and architecture comparisons.
Grounded in the official mHC paper and the DeepSeek-V4 technical report that incorporates it.
We’re adding new public breakdowns over time. Each one walks through a foundational paper with the same step-by-step style.
DeepSeek-AI, 2026
CSA, HCA, the Lightning Indexer, KV-cache compression, Muon, and Pro vs Flash — the V4 architecture, visualized.
Read the breakdownVaswani et al., 2017
Self-attention, Q/K/V, and the scaled dot-product attention equation explained step by step.
Read the breakdownDevlin et al., 2018
Masked language modeling, bidirectional context, and the pre-train / fine-tune recipe that reset NLP.
Hu et al., 2021
Why low-rank adapters work, what gets frozen vs. trained, and the math behind parameter-efficient fine-tuning.
Radford et al., 2021
Contrastive image–text training, the joint embedding space, and how zero-shot transfer falls out of it.
Ho et al., 2020
Forward noise, reverse denoising, and the variational objective that powers modern image synthesis.