Category: 2019 model releases

This category contains 7 pages.

• ALBERT · A 2019 parameter-efficient variant of BERT from Google Research that used factorized embeddings and cross-layer parameter sharing to cut model size.
• DistilBERT · Hugging Face's 2019 distilled version of BERT: roughly 40% smaller and 60% faster while retaining about 97% of BERT's language-understanding performance.
• GPT-2 · OpenAI's 1.5-billion-parameter 2019 model; demonstrated multitask behavior from pure language modeling and started the staged-release debate.
• Megatron-LM · NVIDIA's 2019 research project and open-source framework for training multi-billion-parameter Transformer language models with model parallelism.
• RoBERTa · Facebook AI's July 2019 replication study of BERT that showed the original model was significantly undertrained, setting new benchmark records with an optimized pretraining recipe.
• T5 · Google's 2019 Text-to-Text Transfer Transformer, an open-weight encoder-decoder family that cast every NLP task as text-to-text.
• XLNet · A June 2019 generalized autoregressive pretraining model from Google Brain and Carnegie Mellon University that outperformed BERT on many benchmarks.