Differences

This shows you the differences between two versions of the page.

--- nlp:robustness_in_nlp [2021/04/29 07:03] – jmflanig
+++ nlp:robustness_in_nlp [2023/06/15 07:36] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
 ====== Robustness and Brittleness in NLP (and Deep Learning) ======
+For an overview, read [[https://arxiv.org/pdf/1707.07328.pdf|Jia 2017]] and [[https://arxiv.org/pdf/1907.11932.pdf|Jin 2019]].
 ===== Papers =====
-  * [[https://arxiv.org/pdf/1707.07328.pdf|Jia and Liang 2017 - Adversarial Examples for Evaluating Reading Comprehension Systems]]
+  * **[[https://arxiv.org/pdf/1707.07328.pdf|Jia and Liang 2017 - Adversarial Examples for Evaluating Reading Comprehension Systems]]**
   * [[https://arxiv.org/pdf/1803.02324.pdf|Gururangan et al 2018 - Annotation Artifacts in Natural Language Inference Data]]
   * [[https://arxiv.org/pdf/1805.01042.pdf|Poliak et al 2018 - Hypothesis Only Baselines in Natural Language Inference]]
+  * [[https://arxiv.org/pdf/1808.05326.pdf|Zellers et al 2018 - SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference]] Introduces Adversarial Filtering to try to eliminate dataset bias
   * [[https://arxiv.org/pdf/1902.01007.pdf|McCoy et al 2019 - Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference]]
   * [[https://www.aclweb.org/anthology/P19-1610.pdf|Wee and Ng 2019 - Improving the Robustness of Question Answering Systems to Question Paraphrasing]]
+  * **[[https://arxiv.org/pdf/1907.11932.pdf|Jin et al 2019 - Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment]]**
+  * [[http://proceedings.mlr.press/v119/srivastava20a/srivastava20a.pdf|Srivastava et al 2020 - Robustness to Spurious Correlations via Human Annotations]]
+  * [[https://arxiv.org/pdf/2007.06778.pdf|Tu et al 2020 - An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models]]
+  * [[https://arxiv.org/pdf/2002.00293.pdf|Bartolo et al 2020 - Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension]] Applies adversarial filtering to QA
   * [[https://arxiv.org/pdf/2004.14004.pdf|Si et al 2020 - Benchmarking Robustness of Machine Reading Comprehension Models]]
-  * [[https://arxiv.org/pdf/1907.11932.pdf|Jin et al 2019 - Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment]]
+  * **[[https://arxiv.org/pdf/2005.04118.pdf|Ribeiro et al 2020 - Beyond Accuracy: Behavioral Testing of NLP Models with CheckList]]**
-  * [[http://proceedings.mlr.press/v119/srivastava20a/srivastava20a.pdf|Srivastava 2020 - Robustness to Spurious Correlations via Human Annotations]]
+  * [[https://arxiv.org/pdf/2010.03656.pdf|Rosenman et al 2020 - Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data]] Shows that deep learning relation extraction systems usually rely on shallow heuristics
+  * [[https://aclanthology.org/2021.acl-short.43.pdf|Lin et al 2021 - Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models]]
 ===== Conferences, Workshops, and Shared Tasks =====
+  * [[https://www.aclweb.org/anthology/W17-5401.pdf|Ettinger et al 2017 - Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task]]
   * [[https://bibinlp.umiacs.umd.edu/|Build It, Break It The Language Edition]]
   * [[https://generalizablenlp.weebly.com/|EMNLP 2017 Workshop - Building Linguistically Generalizable NLP Systems]]
@@ Line 17: / Line 26: @@
 ===== People =====
   * [[https://scholar.google.com/citations?user=PbEw81gAAAAJ&hl=en|Hal Daumé III]]
+  * [[https://scholar.google.com/citations?user=0rskDKgAAAAJ&hl=en|Yoav Goldberg]]
   * [[https://scholar.google.com/citations?user=pouyVyUAAAAJ&hl=en|Percy Liang]]
   * [[https://scholar.google.com/citations?user=sFyrSa8AAAAJ&hl=en|Ellie Pavlick]]
+===== Related Pages =====
+  * [[Bias#Dataset Bias (Annotation Artifacts)]]
+  * [[ml:Distribution Shift]]
+  * [[Evaluation#Robust Evaluation]]