Category: 2018 model releases

This category contains 4 pages.

• BERT · Google's 2018 encoder-only Transformer, pretrained with masked language modeling; defined the pretrain-then-finetune era in NLP.
• ELMo · 2018 contextualized word embedding model from AI2 and the University of Washington, built on a bidirectional LSTM language model.
• GPT-1 · OpenAI's June 2018 Generative Pre-trained Transformer, the 117-million-parameter model that established the pretrain-then-finetune recipe behind the GPT series.
• ULMFiT · Howard and Ruder's 2018 three-stage transfer-learning recipe for text, a direct precursor to the pretrain-and-fine-tune paradigm of BERT and GPT.