====== Prompt Engineering ======

===== Introductions and Overviews =====
  * Guides
    * [[https://www.datacamp.com/tutorial/a-beginners-guide-to-chatgpt-prompt-engineering|A Beginner's Guide to ChatGPT Prompt Engineering]] Good concise intro (Accessed March 2025)
    * **[[https://big-picture.com/media/the_prompt_engineering_cheat_sheet.pdf|The Prompt Engineering Cheat Sheet]]** Great
    * [[https://www.promptingguide.ai/|Prompt Engineering Guide]] This one is pretty good
    * [[https://cloud.google.com/discover/what-is-prompt-engineering|Google Cloud - Prompt engineering: overview and guide]]
  * **Overview Papers**
    * [[https://aclanthology.org/2023.findings-emnlp.618.pdf|Leidinger et al 2023 - The language of prompting: What linguistic properties make a prompt successful?]]
  * Slides
  * Blogs:
    * [[https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/|Lil'log Prompt Engineering]]
    * Github: [[https://github.com/brexhq/prompt-engineering|BREX's Prompt Engineering Guide]]
    * Github: [[https://github.com/dair-ai/Prompt-Engineering-Guide|DAIR AI's Prompt Engineering Guide]]
    * Courses
      * [[https://learnprompting.org/docs/intro|learnprompting.org]]
  * [[https://github.com/f/awesome-chatgpt-prompts|Awesome ChatGPT Prompts]]

===== Papers =====
Note: some of these papers should be moved to automatic prompt engineering.

  * [[https://aclanthology.org/2022.findings-acl.50.pdf|Mishra et al 2022 - Reframing Instructional Prompts to GPTk’s Language]]
  * [[https://aclanthology.org/2022.acl-long.60.pdf|Sorensen et al 2022 - An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels]]
  * [[https://arxiv.org/pdf/2109.01247.pdf|Webson & Pavlick 2021 - Do Prompt-Based Models Really Understand the Meaning of Their Prompts?]]
  * **[[https://arxiv.org/pdf/2212.10560.pdf|Wang et al 2022 - Self-Instruct: Aligning Language Model with Self Generated Instructions]]**
  * [[https://arxiv.org/pdf/2301.07085|Webson et al 2023 - Are Language Models Worse than Humans at Following Prompts? It's Complicated]]
  * [[https://arxiv.org/pdf/2301.08721.pdf|Cheng et al 2023 - Batch Prompting: Efficient Inference with Large Language Model APIs]] Batch prompting to save money
  * [[https://arxiv.org/pdf/2306.01150.pdf|Yin et al 2023 - Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning]]
  * [[https://arxiv.org/pdf/2305.19500.pdf|Chen et al 2023 - Exploring Lottery Prompts for Pre-trained Language Models]] Searches to see if there is a prompt for each example that produces the correct answer
  * [[https://arxiv.org/pdf/2308.10819.pdf|Li et al 2023 - Do you really follow me? Adversarial Instructions for Evaluating the Robustness of Large Language Models]]
  * [[https://arxiv.org/abs/2308.05342|Wang & Zhao 2023 - Metacognitive Prompting Improves Understanding
in Large Language Models]]
  * [[https://arxiv.org/pdf/2503.10084|Zhang et al 2025 - Why Prompt Design Matters and Works: A Complexity Analysis of Prompt Search Space in LLMs]] Shows that naive CoT prompting, like “think step by step,"can severely hinder performance.
  * **What makes a good prompt?**
    * [[https://aclanthology.org/2023.findings-emnlp.618.pdf|Leidinger et al 2023 - The language of prompting: What linguistic properties make a prompt successful?]]
    * [[https://aclanthology.org/2023.findings-emnlp.679.pdf|Gonen et al 2023 - Demystifying Prompts in Language Models via Perplexity Estimation]]

==== Automatic Prompt Engineering ====
  * [[https://arxiv.org/pdf/2010.15980.pdf|Shin et al 2020 - AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts]]
  * [[https://arxiv.org/pdf/2205.12548.pdf|Deng et al 2022 - RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning]]
  * [[https://arxiv.org/pdf/2302.03668.pdf|Wen et al 2023 - Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery]]
  * [[https://arxiv.org/pdf/2305.03495.pdf|Pryzant et al 2023 - Automatic Prompt Optimization with "Gradient Descent" and Beam Search]]
  * **[[https://arxiv.org/pdf/2309.03409.pdf|Yang et al 2023 - Large Language Models as Optimizers]]** Figures out a good prompt for the task
  * [[https://arxiv.org/pdf/2309.16797.pdf|Ferdando et al 2023 - Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution]]
  * **Metaprompting**
    * Blog posts, etc
      * [[https://www.prompthub.us/blog/a-complete-guide-to-meta-prompting|A Complete Guide to Meta Prompting]]
      * [[https://www.promptlayer.com/glossary/meta-prompting|Meta-prompting]]
      * [[https://community.openai.com/t/meta-prompting-concept-asking-chat-gpt-for-the-best-prompt-for-your-desired-completion-then-to-revise-it-before-using-it/248619|Meta-Prompting Concept: Asking Chat-GPT for the best prompt for your desired completion, then to revise it before using it]]
    * [[https://aclanthology.org/2022.coling-1.287v2.pdf|2022 - MetaPrompting: Learning to Learn Better Prompts]]
    * [[https://arxiv.org/pdf/2401.12954|2024 - Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding]]


===== Examples of Prompts =====
Examples of clever or interesting prompts.
  * See appendix of [[https://arxiv.org/pdf/2307.16789.pdf]]
  * [[https://arxiv.org/pdf/2310.01714.pdf|Yasunaga et al 2023 - Large Language Models as Analogical Reasoners]]
  * [[https://arxiv.org/pdf/2210.02441.pdf|Arora et al 2022 - Ask Me Anything: A Simple Strategy for Prompting Language Models]]
  * [[https://github.com/f/awesome-chatgpt-prompts|Awesome ChatGPT Prompts]]
  * Prompt to break down sentences into independent facts:
{{media:facts-prompt-arxiv-2305.14251.png}}
(from [[https://arxiv.org/pdf/2305.14251.pdf|Min 2023]])
  * [[https://arxiv.org/pdf/2404.07972|Xie et al 2024 - OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments ]] See the prompts in appendix C


===== Related Pages =====
  * [[Prompting]]