• Paper: Devlin et al 2018 - BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• Blogs
  • Demystifying BERT: A Comprehensive Guide to the Groundbreaking NLP Framework
  • The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning)
• Textbooks
  • SLP Ch 11 (especially 11.2)
• Training from scratch
  • Izsak et al 2021 - How to Train BERT with an Academic Budget
• Retrospective Analyssis
  • Nityasya et al 2023 - On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research

• Wang & Cho 2019 - BERT has a Mouth, and It Must Speak:BERT as a Markov Random Field Language Model WARNING: Mistake in this paper, it's not an MRF.
• Goyal et al 2021 - Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis–Hastings

Summary: Rogers et al 2020 - A Primer in BERTology: What we know about how BERT works. See also Neural Network Psychology.

• 2019 - What Does BERT Look At?An Analysis of BERT’s Attention Also points out that BERT looks at the SEP token as a no-op attention.
• Wiedemann et al 2019 - Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings
• Tenney et al 2019 - What do you learn from context? Probing for sentence structure in contextualized word representations
• Tenney et al 2019 - BERT Rediscovers the Classical NLP Pipeline
• Rogers et al 2020 - A Primer in BERTology: What we know about how BERT works
• 2022 - Visualization of position embeddings in BERT and GPT-2 (Twitter)
• Papadimitriou et al 2022 - When classifying grammatical role, BERT doesn’t care about word order... except when it matters

• Sun et al 2019 - How to Fine-Tune BERT for Text Classification? Exhaustive study investigating different fine-tuning methods of BERT on text classification tasks and provides a general strategy for BERT fine-tuning.
• Shi & Lin 2019 - Simple BERT Models for Relation Extraction and Semantic Role Labeling
• Liu 2019 - Fine-tune BERT for Extractive Summarization
• Zhang et al 2019 - BERTScore: Evaluating Text Generation with BERT

• Mulyar et al 2020 - MT-Clinical BERT: Scaling Clinical Information Extraction with Multitask Learning

• Shen et al 2019 - Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• Sanh et al 2019 - DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

• Pretraining
• Transformers