NLP Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Catastrophic Forgetting ======
//Catastrophic forgetting// is when a neural network is trained to do one task but loses that ability when trained to do a second task.

===== Overviews =====
  * [[https://arxiv.org/pdf/1612.00796.pdf|Kirkpatrick et al 2016 - Overcoming catastrophic forgetting in neural networks]]
  * [[https://arxiv.org/pdf/2307.09218|Wang et al 2023 - A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning]]
  * [[https://arxiv.org/pdf/2312.10549|2023 - Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy]]

===== Papers =====

  * EWC: [[https://arxiv.org/pdf/1612.00796|Kirkpatrick et al 2016 - Overcoming catastrophic forgetting in neural networks]] A famous method
  * [[https://arxiv.org/pdf/1711.05769|Mallya Lazebnik 2017 - PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning]]

==== In NLP ====
  * [[https://aclanthology.org/2022.naacl-main.398.pdf|Li et al 2022 - Overcoming Catastrophic Forgetting During Domain Adaptation of Seq2seq Language Generation]]
  * [[https://arxiv.org/pdf/2309.10105|2023 - Understanding Catastrophic Forgetting in Language Models via Implicit Inference]]
  * [[https://openreview.net/pdf?id=tmsqb6WpLz|Zhang & Wu 2024 - Dissecting learning and forgetting in language model finetuning]]
  * [[https://arxiv.org/pdf/2501.13453|Zheng et al 2025 - Spurious Forgetting in Continual Learning of Language Models]] Argues that the drop in performance while doing continued training "often reflects a decline in task alignment rather than knowledge loss"

===== Related Pages =====
  * [[Continual Learning]]