This page is about natural language task descriptions and prompting language models. Both these methods use natural language to describe a task which is to be performed.

• Liu et al 2021 - Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

• Zero-shot
  • Radford et al 2019 - Language Models Are Unsupervised Multitask Learners GPT-2
  • Wei et al 2021 - Finetuned Language Models Are Zero-Shot Learners
• Few-shot aka In-Context Learning
  • Schick & Schütze 2020 - It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
  • Schick & Schütze 2021 - Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference Introduces PET, pre-dates GTP-3
  • Brown et al 2021 - Language Models are Few-Shot Learners GPT-3
  • Gao et al 2021 - Making Pre-trained Language Models Better Few-shot Learners
  • Schick & Schütze 2020 - Few-Shot Text Generation with Natural Language Instructions GenPET, prompting for natural language generation
• Soft-Prompting, etc
  • Qin & Eisner 2021 - Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
  • Tang et al 2022 - Context-Tuning: Learning Contextualized Prompts for Natural Language Generation
• Prompt design
  • Mishra et al 2022 - Reframing Instructional Prompts to GPTk’s Language
  • Sorensen et al 2022 - An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels
• Data-Augmentation Prompting
  • Wang et al 2022 - PromDA: Prompt-based Data Augmentation for Low-Resource NLU Tasks
• Chain of Thought Prompting
  • Wei et al 2022 - Chain of Thought Prompting Elicits Reasoning in Large Language Models
  • Wang et al 2022 - Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Dohan et al 2022 - Language Model Cascades
• More papers
  • Scao & Rush 2021 - How Many Data Points is a Prompt Worth? Prompts are very helpful in small data regimes, and are worth 100's of datapoints.

• Wang et al 2022 - Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

• Garg et al 2022 - What Can Transformers Learn In-Context? A Case Study of Simple Function Classes

• Goldwasser & Roth 2014 - Learning From Natural Instructions
• Mishra et al 2021 - Natural-Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions

• Invited Talk @ NAACL 2021: Humans Learn From Task Descriptions and So Should Our Models - Hinrich Schütze

• Timo Schick

• Few-Shot Learning
• Zero-Shot Learning