Contents

Turing-NLG (Turing Natural Language Generation, also T-NLG) is a 17-billion-parameter language model announced by Microsoft on February 13, 2020.1 At announcement it was the largest published language model, roughly doubling NVIDIA's 8.3-billion-parameter Megatron-LM,2 a distinction it held only until OpenAI's 175-billion-parameter GPT-3 was announced in May 2020.3 The model was built by Project Turing, Microsoft's applied deep-learning group, and was never released openly; it served as the debut demonstration of the DeepSpeed training library and its ZeRO optimizer, which Microsoft open-sourced the same day.4

Architecture and training

Turing-NLG is a decoder-only Transformer of the same generative family as GPT-2, with 78 layers, a hidden size of 4256, and 28 attention heads.1 Training ran on 256 NVIDIA V100 GPUs in DGX-2 nodes linked by InfiniBand.1 Microsoft attributed the run's feasibility to ZeRO, a memory-partitioning scheme in the DeepSpeed library that shards optimizer states across data-parallel workers, the first stage of a design that also partitions gradients and parameters; per the company, this first stage (ZeRO-OS) cut the required model-parallelism degree from 16 to 4, allowed a fourfold larger batch size, and tripled training throughput.4 Microsoft reported state-of-the-art results on the WikiText-103 perplexity and LAMBADA completion benchmarks at the time.1

Access and deployment

Turing-NLG was never made publicly available. Microsoft offered a private demo to a small set of academic users for testing and feedback, and stated that the underlying work was being integrated into multiple Microsoft products including Bing, Office, and Xbox; abstractive summarization and question answering were the showcased capabilities.1 The February 2020 blog post remained the primary technical disclosure; Microsoft published no dedicated paper and no weights.

Legacy

Turing-NLG marked an intermediate point in language-model scaling, between the 8.3-billion-parameter Megatron-LM of September 2019 and the 175-billion-parameter GPT-3 of May 2020.23 In October 2021, Microsoft and NVIDIA combined DeepSpeed with the Megatron-LM training stack to produce the 530-billion-parameter Megatron-Turing NLG, its direct successor.5 Large-scale internal training at Microsoft later continued under the Microsoft AI division, which trains the MAI-1 models.

References

  1. Corby Rosset, "Turing-NLG: A 17-billion-parameter language model by Microsoft," Microsoft Research Blog, February 2020. ↩↩↩↩↩

  2. Shoeybi et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism," arXiv:1909.08053, September 2019. ↩↩

  3. Brown et al., "Language Models are Few-Shot Learners," arXiv:2005.14165, May 2020. ↩↩

  4. Microsoft Research, "ZeRO & DeepSpeed: New system optimizations enable training models with over 100 billion parameters," Microsoft Research Blog, February 2020. ↩↩

  5. Ali Alvi and Paresh Kharya, "Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, the World's Largest and Most Powerful Generative Language Model," Microsoft Research Blog, October 2021. ↩