Catastrophic Forgetting

Catastrophic forgetting is when a neural network is trained to do one task but loses that ability when trained to do a second task.

• Kirkpatrick et al 2016 - Overcoming catastrophic forgetting in neural networks
• Wang et al 2023 - A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning
• 2023 - Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy

• EWC: Kirkpatrick et al 2016 - Overcoming catastrophic forgetting in neural networks A famous method
• Mallya Lazebnik 2017 - PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning

• Li et al 2022 - Overcoming Catastrophic Forgetting During Domain Adaptation of Seq2seq Language Generation
• 2023 - Understanding Catastrophic Forgetting in Language Models via Implicit Inference
• Zhang & Wu 2024 - Dissecting learning and forgetting in language model finetuning
• Zheng et al 2025 - Spurious Forgetting in Continual Learning of Language Models Argues that the drop in performance while doing continued training “often reflects a decline in task alignment rather than knowledge loss”

• Continual Learning