Differences

This shows you the differences between two versions of the page.

--- nlp:language_model [2026/07/30 18:29] – jmflanig
+++ nlp:language_model [2026/08/03 03:17] (current) – jmflanig
@@ Line 56: / Line 56: @@
   * [[https://arxiv.org/pdf/1602.02410.pdf|Jozefowicz et al 2016 - Exploring the Limits of Language Modeling]] It's interesting to see how far we've come since 2016.
   * GPT: [[https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf|Radford et al 2018 - Improving Language Understanding by Generative Pre-Training]] [[https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf|old link]]
-  * GPT-2: [[https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf|Radford et al 2019 - Language Models are Unsupervised Multitask Learners]] [[https://github.com/openai/gpt-2|original github]] [[https://amaarora.github.io/2020/02/18/annotatedGPT2.html|Annotated GPT-2]] [[https://jalammar.github.io/illustrated-gpt2/|Illustrated GPT-2]] Interestingly, GPT-2 does //not// include a bias term in the final linear layer for the vocab, see [[https://github.com/openai/gpt-2/blob/master/src/model.py#L171|here]] and [[https://github.com/huggingface/transformers/blob/v4.19.2/src/transformers/models/gpt2/modeling_gpt2.py#L951|here]].
+  * GPT-2: [[https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf|Radford et al 2019 - Language Models are Unsupervised Multitask Learners]] [[https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf|old link]] [[https://github.com/openai/gpt-2|original github]] [[https://amaarora.github.io/2020/02/18/annotatedGPT2.html|Annotated GPT-2]] [[https://jalammar.github.io/illustrated-gpt2/|Illustrated GPT-2]] Interestingly, GPT-2 does //not// include a bias term in the final linear layer for the vocab, see [[https://github.com/openai/gpt-2/blob/master/src/model.py#L171|here]] and [[https://github.com/huggingface/transformers/blob/v4.19.2/src/transformers/models/gpt2/modeling_gpt2.py#L951|here]].
   * GPT-3: [[https://arxiv.org/pdf/2005.14165.pdf|Brown et al 2019 - Language Models are Few-Shot Learners]] [[https://beta.openai.com/|OpenAI]] [[https://github.com/openai/openai-cookbook|cookbook]]
   * [[https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/|Turing-NLG: A 17-billion-parameter language model by Microsoft]]
@@ Line 73: / Line 73: @@
 ^ Model ^ Year ^ Parameters ^ Training Data ^ Public? ^ Link ^
 | [[https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf|GPT]] | 2018 | | BooksCorpus | Yes | [[https://github.com/openai/finetune-transformer-lm|github]] [[https://huggingface.co/openai-gpt|Huggingface]] |
-| [[https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf|GPT-2]] | 2019 | 1.5B | Webtext (closed, see [[language_model#datasets]] below) | Yes | [[https://github.com/openai/gpt-2|github]] [[https://huggingface.co/gpt2|Huggingface]] |
+| [[https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf|GPT-2]] | 2019 | 1.5B | Webtext (closed, see [[language_model#datasets]] below) | Yes | [[https://github.com/openai/gpt-2|github]] [[https://huggingface.co/gpt2|Huggingface]] |
 | [[https://arxiv.org/pdf/2005.14165.pdf|GPT-3]] | 2020 | 175B | CommonCrawl, Webtext2, Books 1&2, Wikipedia | API | [[https://beta.openai.com/|OpenAI]] [[https://github.com/openai/openai-cookbook|cookbook]] |
 | [[https://arxiv.org/pdf/2112.10684.pdf|MoE]] | 2021 | 1.1T (13B) | CC100, CC-News, CC-Stories, OpenWebText, BookCorpus, Wikipedia | Yes | [[https://github.com/facebookresearch/fairseq/tree/main/examples/moe_lm|github]] [[https://huggingface.co/KoboldAI/fairseq-dense-6.7B|HuggingFace]] |