In recent years, the development of artificial intelligence (AI) has focused on enormous models, such as those powering systems similar to ChatGPT. But training these models faces serious obstacles, such as signal instability during learning. DeepSeek introduced a method called mHC (manifold-constrained hyper-connections), which addresses the long-standing problem of "exploding" signals in neural networks. This approach builds on the idea behind ResNets, which have enabled the construction of deeper networks since 2016 thanks to shortcut connections that keep information flowing without loss.
Saqib Qamar, the author of an article on Medium, describes how traditional transformers operate like a single-lane highway where information flows linearly. Researchers tried to expand this into multiple "lanes" using hyper-connections (HC), which would make it possible to process more data simultaneously without additional computational overhead. Unfortunately, these parallel lanes led to chaos: signals either became amplified by as much as 3,000 times or vanished completely. This caused training to fail at around 12,000 steps, resulting in the loss of weeks of work and millions in computing costs.
DeepSeek solved this problem by applying mathematical constraints based on manifolds. Instead of freely mixing information between lanes, mHC forces signals to behave like balanced averages. According to tests conducted on models ranging from 3 billion to 27 billion parameters (the adjustable values learned by the model), signal amplification stabilized at approximately 1.6 times instead of experiencing extreme fluctuations. The results? A 7.2% performance improvement in complex reasoning and a 6.9% improvement in reading comprehension across eight benchmarks. Moreover, thanks to GPU optimizations, it added only 6.7% more training time, even though four parallel streams are processed instead of one.
Grassmann Flows: An Alternative to the Attention Mechanism
Another article on The Neuron connects DeepSeek's discovery with a paper by Zhang Chong titled "Attention Is Not What You Need." This research proposes replacing the attention mechanism, which has been the foundation of all large AI models since 2017, with something completely different—Grassmann flows. Attention helps models understand relationships between words in a text, for example, connecting "the cat" with "was tired" in the sentence "The cat sat down on the mat because it was tired." But as Harvey explains, attention has its limits, especially when processing long texts.
The problem with attention lies in its quadratic complexity: If you have 1,000 tokens (a token is roughly a word or part of a word), the model must calculate the relationships between every pair, resulting in a million connections. Double the length to 2,000 tokens? The computation quadruples. This makes processing millions of tokens (about 750,000 words, equivalent to 10 novels) extremely expensive. Zhang Chong proposes Grassmann flows, which represent relationships between tokens geometrically on a Grassmann manifold using Plücker coordinates. Instead of an enormous attention matrix, tokens are processed linearly—doubling the length doubles the computation rather than quadrupling it.
Tests showed that a model with 13 to 18 million parameters based entirely on Grassmann flows achieved results only 10–15% worse than a standard transformer on tasks such as language modeling on Wikitext-2 and natural language inference on SNLI. On SNLI, it even slightly outperformed the transformer. The Neuron emphasizes that this approach makes models more interpretable because relationships are described by mathematical rules rather than random weights learned through gradient descent.
How These Innovations Complement Each Other
Both approaches address mathematical chaos in transformers, but in different ways. DeepSeek's mHC adds "guardrails" to the existing architecture to prevent signal explosions and uses techniques such as the Sinkhorn-Knopp algorithm for efficiency. According to Rohan Paul, quoted in Harvey's article, this enables stable training without sacrificing performance. Grassmann flows, by contrast, go further and argue that attention is unnecessarily complex—they replace it with geometry, reducing complexity from quadratic to linear.
Harvey notes that these methods are not in conflict. They could be combined: Grassmann flows for local relationships and mHC for deeper layers. DeepSeek, founded by Liang Wenfeng, often publishes such research ahead of major releases, such as last year's R1 model. A new flagship model is expected to arrive soon, while Grassmann flows will be tested at larger scales.
These discoveries make AI more reliable and efficient, which is crucial at a time when training costs tens of millions and takes months.
