Harald Sack: "Next leg in our brief history …" - Sigmoid Social

Recent searches

Search options

Only available when logged in.

Harald Sack @lysander07@sigmoid.social

Next leg in our brief history of (Large) #language models is then advent of the first (real) pretrained #LLMs: ElMO (Allan Institute for AI, 2017), GPT (OpenAI, 2018) and BERT (Google, 2018).
Slides: https://drive.google.com/file/d/1atNvMYNkeKDwXP3olHXzloa09S5pzjXb/view?usp=drive_link
ELMO: https://aclanthology.org/N18-1202.pdf
GPT: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf
BERT: https://aclanthology.org/N19-1423
#AI @fizise #languagemodel #nlp #artificialintelligence #generativeai #lecture #ise2023

Slide from ISE 2023 lecture, Brief History of LLMs:
2017, Elmo (Embeddings from Language Models)
developed by Allen Institute for AI, trained on 5.5B words, 1B parameters
2018, Generative Pre-trained Transformers (GPT) introduced by OpenAI, a pre-trained language model that can be fine-tuned for various NLP applications. GPT was trained on 1.1GB text (book corpus).
2018, Bidirectional Encoder Representations from Transformers (BERT), a family of pre-trained language models for NLP tasks is released by Google. BERT-Large comprises 340M parameters, pre-trained on book corpus and English Wikipedia.
Bibliography:
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. (2018). Deep Contextualized Word Representations. In Proc of the 2018 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1, pp. 2227–2237, ACL.
Radford, A., Narasimhan, K., Salimans, T. & Sutskever, I. (2018). Improving language understanding by generative pre-training.
J. Devlin, M. Chang, K. Lee, and K. Toutanova. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc of the 2019 Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol 1, pp. 4171–4186, ACL.

Aug 13, 2023, 07:01 AM··Web

3boosts·5favorites

Drag & drop to upload