NLP Wiki

User Tools

Site Tools

Trace: • classification • retrieval-augmented_methods • normalization • zero-shot_learning • automatic_theorem_proving • gpu_deep_learning • distribution_shift • software • unsupervised_methods • knowledge-enhanced_methods
nlp:language_model

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
nlp:language_model [2025/09/23 23:53] – [Large Language Models] jmflanignlp:language_model [2026/08/03 03:17] (current) – jmflanig
Line 26: Line 26:
       * **[[https://arxiv.org/pdf/2303.18223.pdf|Zhao et al 2023 - A Survey of Large Language Models]]**       * **[[https://arxiv.org/pdf/2303.18223.pdf|Zhao et al 2023 - A Survey of Large Language Models]]**
       * [[https://arxiv.org/pdf/2404.09022|Weng 2024 - Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategies]]       * [[https://arxiv.org/pdf/2404.09022|Weng 2024 - Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategies]]
 +      * **[[https://arxiv.org/pdf/2501.17805|2025 - International AI Safety Report]]** (Has a good non-technical overview of AI, ML & LLMs)
 +      * [[https://arxiv.org/pdf/2601.02907|Gan et al 2026 - Beyond the Black Box: A Survey on the Theory and Mechanism of Large Language Models]]
   * **Language models in the news, etc**   * **Language models in the news, etc**
     * [[https://www.wired.com/story/ai-text-generator-gpt-3-learning-language-fitfully/|Wired - GPT-3]]     * [[https://www.wired.com/story/ai-text-generator-gpt-3-learning-language-fitfully/|Wired - GPT-3]]
Line 54: Line 56:
   * [[https://arxiv.org/pdf/1602.02410.pdf|Jozefowicz et al 2016 - Exploring the Limits of Language Modeling]] It's interesting to see how far we've come since 2016.   * [[https://arxiv.org/pdf/1602.02410.pdf|Jozefowicz et al 2016 - Exploring the Limits of Language Modeling]] It's interesting to see how far we've come since 2016.
   * GPT: [[https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf|Radford et al 2018 - Improving Language Understanding by Generative Pre-Training]] [[https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf|old link]]   * GPT: [[https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf|Radford et al 2018 - Improving Language Understanding by Generative Pre-Training]] [[https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf|old link]]
-  * GPT-2: [[https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf|Radford et al 2019 - Language Models are Unsupervised Multitask Learners]] [[https://github.com/openai/gpt-2|original github]] [[https://amaarora.github.io/2020/02/18/annotatedGPT2.html|Annotated GPT-2]] [[https://jalammar.github.io/illustrated-gpt2/|Illustrated GPT-2]] Interestingly, GPT-2 does //not// include a bias term in the final linear layer for the vocab, see [[https://github.com/openai/gpt-2/blob/master/src/model.py#L171|here]] and [[https://github.com/huggingface/transformers/blob/v4.19.2/src/transformers/models/gpt2/modeling_gpt2.py#L951|here]].+  * GPT-2: [[https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf|Radford et al 2019 - Language Models are Unsupervised Multitask Learners]] [[https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf|old link]] [[https://github.com/openai/gpt-2|original github]] [[https://amaarora.github.io/2020/02/18/annotatedGPT2.html|Annotated GPT-2]] [[https://jalammar.github.io/illustrated-gpt2/|Illustrated GPT-2]] Interestingly, GPT-2 does //not// include a bias term in the final linear layer for the vocab, see [[https://github.com/openai/gpt-2/blob/master/src/model.py#L171|here]] and [[https://github.com/huggingface/transformers/blob/v4.19.2/src/transformers/models/gpt2/modeling_gpt2.py#L951|here]].
   * GPT-3: [[https://arxiv.org/pdf/2005.14165.pdf|Brown et al 2019 - Language Models are Few-Shot Learners]] [[https://beta.openai.com/|OpenAI]] [[https://github.com/openai/openai-cookbook|cookbook]]   * GPT-3: [[https://arxiv.org/pdf/2005.14165.pdf|Brown et al 2019 - Language Models are Few-Shot Learners]] [[https://beta.openai.com/|OpenAI]] [[https://github.com/openai/openai-cookbook|cookbook]]
   * [[https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/|Turing-NLG: A 17-billion-parameter language model by Microsoft]]   * [[https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/|Turing-NLG: A 17-billion-parameter language model by Microsoft]]
Line 71: Line 73:
 ^ Model ^ Year ^ Parameters ^ Training Data ^ Public? ^ Link ^ ^ Model ^ Year ^ Parameters ^ Training Data ^ Public? ^ Link ^
 | [[https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf|GPT]] | 2018 | | BooksCorpus | Yes | [[https://github.com/openai/finetune-transformer-lm|github]] [[https://huggingface.co/openai-gpt|Huggingface]] | | [[https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf|GPT]] | 2018 | | BooksCorpus | Yes | [[https://github.com/openai/finetune-transformer-lm|github]] [[https://huggingface.co/openai-gpt|Huggingface]] |
-| [[https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf|GPT-2]] | 2019 | 1.5B | Webtext (closed, see [[language_model#datasets]] below) | Yes | [[https://github.com/openai/gpt-2|github]] [[https://huggingface.co/gpt2|Huggingface]] |+| [[https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf|GPT-2]] | 2019 | 1.5B | Webtext (closed, see [[language_model#datasets]] below) | Yes | [[https://github.com/openai/gpt-2|github]] [[https://huggingface.co/gpt2|Huggingface]] |
 | [[https://arxiv.org/pdf/2005.14165.pdf|GPT-3]] | 2020 | 175B | CommonCrawl, Webtext2, Books 1&2, Wikipedia | API | [[https://beta.openai.com/|OpenAI]] [[https://github.com/openai/openai-cookbook|cookbook]] | | [[https://arxiv.org/pdf/2005.14165.pdf|GPT-3]] | 2020 | 175B | CommonCrawl, Webtext2, Books 1&2, Wikipedia | API | [[https://beta.openai.com/|OpenAI]] [[https://github.com/openai/openai-cookbook|cookbook]] |
 | [[https://arxiv.org/pdf/2112.10684.pdf|MoE]] | 2021 | 1.1T (13B) | CC100, CC-News, CC-Stories, OpenWebText, BookCorpus, Wikipedia | Yes | [[https://github.com/facebookresearch/fairseq/tree/main/examples/moe_lm|github]] [[https://huggingface.co/KoboldAI/fairseq-dense-6.7B|HuggingFace]] | | [[https://arxiv.org/pdf/2112.10684.pdf|MoE]] | 2021 | 1.1T (13B) | CC100, CC-News, CC-Stories, OpenWebText, BookCorpus, Wikipedia | Yes | [[https://github.com/facebookresearch/fairseq/tree/main/examples/moe_lm|github]] [[https://huggingface.co/KoboldAI/fairseq-dense-6.7B|HuggingFace]] |
Line 120: Line 122:
     * [[https://arxiv.org/pdf/2307.03172|Liu 2023 - Lost in the Middle: How Language Models Use Long Contexts]]     * [[https://arxiv.org/pdf/2307.03172|Liu 2023 - Lost in the Middle: How Language Models Use Long Contexts]]
       * [[https://link.springer.com/chapter/10.1007/978-3-031-88708-6_16|Hutter et al 2025 - Lost but Not Only in the Middle]]       * [[https://link.springer.com/chapter/10.1007/978-3-031-88708-6_16|Hutter et al 2025 - Lost but Not Only in the Middle]]
 +
 +==== LLM Benchmarks ====
 +  * [[https://artificialanalysis.ai/models]]
  
 ==== Origin of Capabilities ==== ==== Origin of Capabilities ====
Line 130: Line 135:
   * **Overviews**   * **Overviews**
     * [[https://arxiv.org/pdf/2307.03109|Chang et al 2023 - A Survey on Evaluation of Large Language Models]]     * [[https://arxiv.org/pdf/2307.03109|Chang et al 2023 - A Survey on Evaluation of Large Language Models]]
 +    * For common evaluation datasets for LLMs, see recent LLM system description papers such as the [[https://arxiv.org/pdf/2407.21783|LLama 3 paper]] (table 2) or [[https://www.anthropic.com/news/claude-sonnet-4-5|Claude Sonnet 4.5]] (evaluation table).
   * lm-evaluation-harness: [[https://github.com/EleutherAI/lm-evaluation-harness|LM Evaluation Harness (EleutherAI)]] (Released May 2021)   * lm-evaluation-harness: [[https://github.com/EleutherAI/lm-evaluation-harness|LM Evaluation Harness (EleutherAI)]] (Released May 2021)
   * [[https://arxiv.org/pdf/2401.00595|Mizrahi et al 2024 - State of What Art? A Call for Multi-Prompt LLM Evaluation]]   * [[https://arxiv.org/pdf/2401.00595|Mizrahi et al 2024 - State of What Art? A Call for Multi-Prompt LLM Evaluation]]
Line 137: Line 143:
   * **Effects of Length and Irrelevant Context**   * **Effects of Length and Irrelevant Context**
     * [[https://arxiv.org/pdf/2402.14848|Levy et al 2024 - Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models]]     * [[https://arxiv.org/pdf/2402.14848|Levy et al 2024 - Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models]]
 +
 +===== Tool-Use in LLMs =====
 +See also [[prompting#Chained or Tool-based Prompting]].
 +  * **Overviews and Background**
 +    * [[https://modelcontextprotocol.io/docs/getting-started/intro|Model Contex Protocol]]
 +
 +===== Retrieval-Augmented Generation (RAG) =====
 +See [[Retrieval-Augmented Methods]].
  
 ===== Limitations of Current LLMs ===== ===== Limitations of Current LLMs =====
Line 171: Line 185:
   * Extracting Training Data   * Extracting Training Data
     * [[https://arxiv.org/pdf/2012.07805.pdf|Carlini et al 2020 - Extracting Training Data from Large Language Models]] [[https://github.com/ftramer/LM_Memorization|github]]     * [[https://arxiv.org/pdf/2012.07805.pdf|Carlini et al 2020 - Extracting Training Data from Large Language Models]] [[https://github.com/ftramer/LM_Memorization|github]]
 +    * [[https://arxiv.org/pdf/2601.02671|Ahmed et al 2026 - Extracting Books from Production Language Models]]
   * Membership Inference for Training Data   * Membership Inference for Training Data
     * (Decide if some sample data is in the training data or not)     * (Decide if some sample data is in the training data or not)
nlp/language_model.1758671632.txt.gz · Last modified: 2025/09/23 23:53 by jmflanig
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki