Distributed training is the systems discipline of spreading a neural network's training across many accelerators, thousands to hundreds of thousands of GPUs at the LLM frontier, so that models too large for any single device can be trained in acceptable wall-clock time. Every frontier model on this wiki, from GPT-3 and GPT-4 to the open runs of Meta AI and DeepSeek, exists because of this stack.1
The parallelism taxonomy
Data parallelism replicates the model and splits the batch, synchronizing gradients each step. Tensor (model) parallelism splits individual weight matrices across devices, as systematized by NVIDIA's Megatron-LM. Pipeline parallelism assigns contiguous layer blocks to different devices and streams microbatches through them. Expert parallelism distributes the experts of mixture-of-experts models. Frontier runs compose all four ("4D parallelism"), tuned to cluster topology and interconnect bandwidth.1
Memory-sharding schemes, DeepSpeed's ZeRO and PyTorch's FSDP, partition optimizer state, gradients, and parameters across data-parallel ranks, collapsing the memory redundancy of naive replication.2 Expert parallelism entered the canon with Google's GShard, which trained 600-billion-parameter sparse translation models across 2048 TPU cores and established the routing-and-sharding patterns later MoE systems inherit.3
Practice
The discipline's realities shape everything downstream: Transformer architecture choices favor operations that parallelize cleanly; checkpoint-and-restart engineering determines how training survives hardware failures, which at 100,000-GPU scale occur continuously; and utilization metrics (MFU) are the field's efficiency scoreboard. Published system descriptions from Google DeepMind (Pathways), Meta's Llama reports, and DeepSeek's infrastructure papers are the canonical public documentation, while labs like OpenAI and Anthropic disclose comparatively little; figures here carry source attribution accordingly.12
See also
References
-
Shoeybi, M., et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism," arXiv:1909.08053, September 2019. ↩↩↩
-
Rajbhandari, S., et al., "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models," arXiv:1910.02054, October 2019. ↩↩
-
Lepikhin, D., et al., "GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding," arXiv:2006.16668, June 2020. ↩