Scientists Achieve Superlinear Speedup in Training a 100-Billion-Parameter Model

A New Era of Heterogeneous Machine Learning

Researchers from the Shanghai Artificial Intelligence Laboratory, led by Ding Tang, Jiecheng Zhou, Jiakai Hu, Shengwei Li, Huihuang Zheng, Zhilin Pei, Hui Wang, and Xingcheng Zhang, introduced a groundbreaking framework called H2 (HyperHetero), which enables the efficient training of large language models on clusters containing more than 1,000 heterogeneous chips from different manufacturers. This technology addresses one of the most significant challenges in current artificial intelligence development—how to efficiently utilize the computing resources of different chip types that have distinct characteristics in terms of computing performance, memory, and communication capabilities.

Conventional approaches to distributed training of large models were designed for homogeneous environments, where all chips come from a single manufacturer and have similar characteristics. However, the current situation in the industry requires the use of diversified hardware resources, as relying solely on one type of chip may be insufficient due to manufacturing capacity constraints or its specific suitability for certain computing tasks. Hyper-heterogeneous computing environments are characterized by three key features: substantially different hardware specifications without clear patterns, software isolation between chips from different manufacturers, and a highly imbalanced distribution of different chip types within a single cluster.

Overcoming Technical Isolation

The H2 framework addresses the problem of technical isolation between different chip types through three main components. The first component is DiTorch, a unified PyTorch-compatible interface that standardizes operator libraries and runtime environments across heterogeneous chips. DiTorch uses PyTorch as a unified programming layer and implements a bottom-up accuracy alignment pipeline that first enforces numerical consistency at the operator level across different chips and then achieves end-to-end accuracy alignment at the model level.

DiTorch integrates two main strategies for improving interoperability across different AI hardware platforms. Torch Adapter can be supplied by the chip manufacturer, or the Device-Independent Process Unit (DIPU) can be used directly to harmonize the underlying runtime systems from different manufacturers into unified APIs. In addition, the Device-Independent Operator Interface (DIOPI) connects the unified Torch operator API with operator libraries tailored to specific manufacturers, ensuring consistent semantic interpretation across operators. More than 300 standardized operator interfaces are currently implemented in DIOPI.

The second component is DiComm, a unified communication library that overcomes isolation and inefficiency in complex heterogeneous environments. DiComm uses the libibverbs library and enables RDMA communication across different chip architectures, supporting both homogeneous and heterogeneous chip-to-chip interactions. The library implements two distinct communication models: CPU-mediated and device-direct. In the device-direct method, each chip registers its local memory regions with the RDMA driver, allowing the RDMA-enabled NIC to map physical addresses for remote access. Experimental results show that device-direct RDMA reduces average latency by 9.94× compared with the conventional TCP/IP scheme.

Heterogeneous Parallelism and Automated Strategy Search

The third key component is the HeteroPP framework together with HeteroAuto, an adaptive pipeline parallelism strategy that dynamically balances computational load, memory constraints, and communication overhead. HeteroPP extends pipeline parallelism to heterogeneous AI chip clusters for distributed LLM training. Each pipeline stage consists exclusively of a homogeneous chip type, while heterogeneous nodes are strategically distributed across different stages. Chips with greater memory capacity are mapped to earlier pipeline stages, while chips with less memory are allocated to later stages.

HeteroAuto provides an automated search method for optimizing parallel strategy configurations in heterogeneous chip environments. The system uses a depth-first search (DFS) methodology and heuristic search to derive the optimal layer distribution. The algorithm first identifies an appropriate degree of data parallelism and then divides large groups of homogeneous chips into smaller subgroups that are processed as distinct heterogeneous entities. To improve training efficiency in heterogeneous settings, a topology-aware activation resharding strategy was developed that configures each chip server to assign a dedicated communication NIC to each chip based on its NIC affinity.

Experimental Validation and Results

The researchers validated their framework on a 100-billion-parameter model with an architecture based on the structural principles of LLaMA. The model contains 96 layers, 64 attention heads, a hidden size of 8,192, and an intermediate size of 36,864, with a maximum sequence length of 4,096 tokens. Testing was conducted with four different types of AI chips designated Chip-A, Chip-B, Chip-C, and Chip-D, which differed in computing performance, memory capacity, and the number of chips per node.

The experimental results demonstrate the exceptional efficiency of the H2 framework. In a configuration with 768 chips of three types, the system achieved a HeteroSpeedupRatio of 109.03%, while with 1,024 chips of four types, it achieved 104.29%. Even with a constant global batch size, the system achieved 89.56% and 77.45%, respectively. These results show that heterogeneous training can achieve competitive—and, in some cases, even superior—efficiency compared with the baseline throughput of homogeneous training.

The superlinear performance improvement can be explained by the fact that conventional 3D parallel training tends to overlook imbalanced resource requirements across different computing tasks, while the HeteroPP framework with HeteroAuto capitalizes on these imbalances through intelligent allocation of chip workloads and fine-tuning of training hyperparameters based on specific resource requirements. In practical production environments, lower-specification chips typically have significantly lower prices and reduced power consumption compared with high-specification chips, making the framework economically advantageous.

Future Prospects and Significance

The H2 framework represents a significant advance in distributed machine learning by enabling the efficient integration of diverse hardware resources into a unified computing system. This technology has the potential to revolutionize how organizations approach the training of large language models by allowing them to use existing heterogeneous infrastructure instead of having to invest in homogeneous clusters. The results of experiments on the 100-billion-parameter model confirm the feasibility and efficiency of hyper-heterogeneous training at an unprecedented scale, opening up new possibilities for developing even larger and more complex AI systems using diversified computing resources.