Neural Network Psychology

“Neural network psychology” is the study of what neural networks learn and why they make the predictions they do. See below for examples.

• Shi et al 2016 - Why Neural Translations are the Right Length
• Shi et al 2016 - Does String-Based Neural MT Learn Source Syntax?
• Linzen et al 2016 - Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
• Tenney et al 2019 - What do you learn from context? Probing for sentence structure in contextualized word representations Introduces the technique of “edge probing”
• Tenney et al 2019 - BERT Rediscovers the Classical NLP Pipeline Uses “edge probing”
• Jawahar et al 2019 - What does BERT learn about the structure of language?
• Raganato et al 2018 - An Analysis of Encoder Representations in Transformer-Based Machine Translation
• Langedijk et al 2023 - DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformers

• I believe Kevin Knight came up with this term

• BlackboxNLP

• BERTology
• Explainability
• Mechanistic Interpretability
• Probing Experiments