NLP Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Dataset Creation ======

===== Annotation =====
For annotation tools, see [[nlp:software#annotation_tools|Software - Annotation Tools]]. Annotation can often be greatly sped up by building your own annotation tool with exactly the features you want for your application. This is can be a worthwhile time investment, since a well-designed tool can speed up annotation.\\
To annotate data manually (without using [[crowdsourcing]]), practitioners generally follow these steps:
  * **Gather data**: Decide the data source and gather some data to annotate.  Be sure to consider any ethical issues.  If you want to be able to release the data publicly, check for potential copyright or privacy violations.
  * **Decide what to annotate**: Look at a portion of the data and decide a rough idea of what you want to annotate -- that is the phenomena you want to capture and at what granularity.  Write a document describing the preliminary annotation scheme.
  * **Pilot annotation**: Try annotating some data by yourself or with some collegues using the annotation scheme.  (Annotate the same data). Compare annotations and decide on edge cases (decide what to do on the difficult boardline cases).  Decide if you want to simplify or extend the annotation scheme.
  * **Refine annotation scheme (iterative)**: Refine your annotation scheme until you're happy with it and it is easy to annotate.  This may take several rounds of pilot annotation. 
  * **Compute inter-annotator agreement**: Make sure to doubly annotate a subset of the data you annotate so you can compute inter-annotator agreement
  * **Full-scale annotation**: Annotate a bunch of data yourself or using annotators you've trained (usually it helps to have these annotators involved in developing the annotation scheme during the pilot annotation).  Depending on the complexity of the annotation task, you may need to have regular meetings during this time to decide on edge cases as they come up.
  * **Release**: Release it with a document or published paper describing the annotation scheme and a [[experimental_method#reproducibility_checklists_datasheets_and_model_cards|datasheet]]. Make sure there are no copyright or privacy violations when releasing the data.  You can create a project page such as [[https://rowanzellers.com/hellaswag/|this one]] or [[http://style.cs.umass.edu/|this one]].
  * **Updates**: You can release another version of the data to annotate more data or fix errors.  You can also have a bug report form (like [[https://amr.isi.edu/report-bug.html|this one]]) for the dataset to allow others to fix errors.

==== Annotation Agreement ====
  * Measures of inter-annotator agreement
    * [[https://en.wikipedia.org/wiki/Cohen%27s_kappa|Cohen's kappa]] Better than percent agreement, see slides [[https://www.inf.ed.ac.uk/teaching/courses/fnlp/lectures/10_slides-2x2.pdf|here]]
    * [[https://www.real-statistics.com/reliability/interrater-reliability/weighted-cohens-kappa/|Weighted Cohen's kappa]]
    * [[https://en.wikipedia.org/wiki/Fleiss%27_kappa|Fleiss' kappa]] More general than Cohen's kappa
  * Software
    * R's [[https://cran.r-project.org/web/packages/irr/index.html|Inter-Annotator Reliability Package]] (IRR) is great. [[https://cran.r-project.org/web/packages/irr/irr.pdf|docs]] [[https://www.andywills.info/rminr/irr.html|example]]


==== Building Your own Annotation Tool ====
  * For simple projects, annotation can be done in a spreadsheet
  * When building your own annotation tool, here are some things to consider
    * The purpose of the tool is to make the annotation faster.  Think carefully about what interface will be fastest for trained annotators.
    * To speed up development, use whatever language and API you are familiar with or find easiest.
    * Think very carefully about ways to reduce unnecessary mouse clicks, typing, reading text, etc.  Every mouse click counts.  Aggressively remove anything that is unnecessary, like typing escape or enter to save.  Instead, automatically save when you go to the next example, etc.
    * Plan on doing some iterations on the tool.  You will need to try it, and change it based on your experience.
    * It doesn't need to be perfect, it just needs to be fast to use.  It's ok to have bugs in the annotation tool if it's not widely used, and they don't slow down annotation.
    * Don't make it full-featured.  You just need the features that make annotation fast.

===== Dataset and Data Selection Issues =====
  * [[Bias]]
  * [[Data Preparation#Creating a Train/Dev/Test Split]]
  * [[https://ai.googleblog.com/2021/06/data-cascades-in-machine-learning.html|Data Cascades]]

===== Data Validation =====
See [[ml:Data Cleaning and Validation]].

===== Crowdsourcing =====
See [[Crowdsourcing]].

===== Alternative Methods =====
=== Methods of Faster or Cheaper Annotation ===
  * [[https://www.aclweb.org/anthology/N13-1014.pdf|Garrette & Baldridge 2013 - Learning a Part-of-Speech Tagger from Two Hours of Annotation]] Talks efficiency of annotating types vs tokens
  * [[https://arxiv.org/pdf/1306.2091.pdf|Schneider et al 2013 - A Framework for (Under)specifying Dependency Syntax without Overloading Annotators]]
  * [[https://arxiv.org/pdf/1605.07723.pdf|Ratner et al 2016 - Data Programming: Creating Large Training Sets, Quickly]]
  * See also [[Prompting]] and [[https://arxiv.org/pdf/2103.08493.pdf|Scao & Rush 2021 - How Many Data Points is a Prompt Worth?]] Prompts are very helpful in small data regimes, and are worth 100's of datapoints.

=== Methods of Avoiding Dataset Bias or Improving Robustness ===
  * **Adversarial Filtering**
    * [[https://arxiv.org/pdf/1808.05326.pdf|Zellers et al 2018 - SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference]] Introduced adversarial filtering
    * [[https://arxiv.org/pdf/1905.07830.pdf|Zellers et al 2019 - HellaSwag: Can a Machine Really Finish Your Sentence?]]

===== Reducing Bias =====
See [[Bias#Reducing Annotation Artifacts During Dataset Creation]]

===== Documentation =====
  * Good to document the data with [[experimental_method#reproducibility_checklists_datasheets_and_model_cards|data cards]], such as [[https://quac.ai/datasheet.pdf|this]] or [[https://gem-benchmark.com/data_cards/web_nlg|this]] (or the [[https://web.archive.org/web/20210613075644/https://gem-benchmark.com/data_cards/WebNLG|old version]])

===== Related Pages =====
  * [[nlp:software#Annotation Tools]]
  * [[Bias]]
  * [[Crowdsourcing]]
  * [[ml:Data Cleaning and Validation]]
  * [[Data Augmentation]]
  * [[Data Preparation]]
  * [[Ethics]]
  * [[Robustness in NLP]]