Differences

This shows you the differences between two versions of the page.

--- nlp:domain_adaptation [2022/01/24 04:46] – [Related Pages] jmflanig
+++ nlp:domain_adaptation [2023/06/15 07:36] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
 ====== Domain Adaptation ======
-See [[https://en.wikipedia.org/wiki/Domain_adaptation|Wikipedia - Domain Adaptation]].  Usually in NLP, domain adaptation becomes necessary if the training data is from a different genre than the testing data - you train on newswire, test on medical domain, for example.  Since natural language is so varied, often the domains are quite different, with different lexical items, semantics, and syntactic patterns.  For a definition of what a domain is, see  [[https://www.aclweb.org/anthology/P15-2092.pdf|van der Wees 2015 - What’s in a Domain? Analyzing Genre and Topic Differences in Statistical Machine Translation]] or references in [[https://arxiv.org/pdf/2004.10964.pdf|Gururangan 2020]].
+See [[https://en.wikipedia.org/wiki/Domain_adaptation|Wikipedia - Domain Adaptation]].  Usually in NLP, domain adaptation becomes necessary if the training data is from a different genre than the testing data - you train on newswire, test on medical domain, for example.  Since natural language is so varied, often the domains are quite different, with different lexical items, syntactic patterns, and semantics.  For a definition of what a domain is, see  [[https://www.aclweb.org/anthology/P15-2092.pdf|van der Wees 2015 - What’s in a Domain? Analyzing Genre and Topic Differences in Statistical Machine Translation]] or references in [[https://arxiv.org/pdf/2004.10964.pdf|Gururangan 2020]].
 ===== Overviews =====
   * [[https://arxiv.org/pdf/1812.02849.pdf|2018 - A Survey of Unsupervised Deep Domain Adaptation]] Unsupervised domain adaptation differs from regular domain adaptation in that you don't get to see labeled examples in the domain of interest, only unlabeled examples.
   * [[https://arxiv.org/pdf/1806.00258.pdf|Chu & Wang 2018 - A Survey of Domain Adaptation for Neural Machine Translation]]
+  * [[https://arxiv.org/pdf/2006.00632.pdf|Ramponi & Plank 2020 - Neural Unsupervised Domain Adaptation in NLP - A Survey]]
+===== Domain Adaptation (Outside of NLP) =====
+  * [[https://proceedings.mlr.press/v37/ganin15.pdf|Ganin & Lempitsky 2015 - Unsupervised Domain Adaptation by Backpropagation]]
+  * [[https://arxiv.org/pdf/1505.07818.pdf|Ganin et al 2016 - Domain-Adversarial Training of Neural Networks]]
 ===== Papers (General Domain Adaptation in NLP) =====
-See also [[https://www.aminer.org/topic/5faa648192c7f9be21f70f88|Awesome Neural Adaptation in NLP]] A curated list of unsupervised domain adaption papers in NLP (not including MT).
+See also [[https://github.com/bplank/awesome-neural-adaptation-in-NLP|Awesome Neural Adaptation in NLP]] or [[https://www.aminer.org/topic/5faa648192c7f9be21f70f88|here (old)]] A curated list of unsupervised domain adaption papers in NLP (not including MT).
   * [[https://arxiv.org/pdf/0907.1815.pdf|Daumé III 2009 - Frustratingly Easy Domain Adaptation]]  A seminal paper, the baseline that you should always try (for linear models).
   * The obvious baseline for neural networks is to fine-tune a pre-trained network on the new domain.  I don't know any papers looking into this method and the trade-off associated with it, but there should be one
+  * [[https://arxiv.org/pdf/1505.07818.pdf|Ganin et al 2016 - Domain-Adversarial Training of Neural Networks]]
   * [[https://www.aclweb.org/anthology/C16-1038.pdf|Kim et al 2016 - Frustratingly Easy Neural Domain Adaptation]]  Not a very good paper. Didn't compare to sensible baselines such as fine-tuning on the new domain.
   * [[https://www.aclweb.org/anthology/K17-1029.pdf|2017 - Neural Domain Adaptation for Biomedical Question Answering]]
+  * [[https://aclanthology.org/P18-1096.pdf|Ruder & Plank 2018 - Strong Baselines for Neural Semi-Supervised Learning under Domain Shift]] Proposes multi-task tri-training, which trains one model for tri-training on one task.
   * [[https://arxiv.org/pdf/1911.10768.pdf|Nishida et al 2019 - Unsupervised Domain Adaptation of Language Models for Reading Comprehension]] LREC paper
   * [[https://arxiv.org/pdf/2004.10964.pdf|Gururangan et al 2020 - Don't Stop Pretraining: Adapt Language Models to Domains and Tasks]] Note: measures similarity between domains by counting n-gram overlap. Has nice references for what is a domain.
   * [[https://arxiv.org/pdf/2104.07078.pdf|Karouzos et al 2021 - UDALM: Unsupervised Domain Adaptation through Language Modeling]] NAACL Main paper.
+  * [[https://aclanthology.org/2021.adaptnlp-1.22.pdf|Xu et al 2021 - Gradual Fine-Tuning for Low-Resource Domain Adaptation]]
+  * [[https://aclanthology.org/2021.emnlp-main.566.pdf|Kulshreshtha et al 2021 - Back-Training excels Self-Training at Unsupervised Domain Adaptation of Question Generation and Passage Retrieval]] Back-training: generating noisy inputs given outputs (vs self-training: generating noisy outputs given inputs).
+  * [[https://aclanthology.org/2022.naacl-main.96.pdf|Chronopoulou et al 2022 - Efficient Hierarchical Domain Adaptation for Pretrained Language Models]]
+  * [[https://arxiv.org/pdf/2302.03169.pdf|Xie et al 2023 - Data Selection for Language Models via Importance Resampling]]
+  * [[https://arxiv.org/pdf/2302.03194.pdf|Malik et al 2023 - UDApter - Efficient Domain Adaptation Using Adapters]]
+{{media:domain_adaptation_table2020.png}}\\
+Table from [[https://arxiv.org/pdf/2006.00632.pdf|Ramponi & Plank 2020]].
 ===== Domain Adaptation in NLP Tasks =====
@@ Line 23: / Line 38: @@
 ===== Related Pages =====
+  * [[Data Augmentation]]
+  * [[ml:Fine-Tuning]]
   * [[ml:Semi-supervised Learning]]