Differences

This shows you the differences between two versions of the page.

--- nlp:dataset_creation [2023/11/22 22:19] – [Related Pages] jmflanig
+++ nlp:dataset_creation [2023/12/10 06:18] (current) – [Building Your own Annotation Tool] jmflanig
@@ Line 2: / Line 2: @@
 ===== Annotation =====
-For annotation tools, see [[nlp:software#annotation_tools|Software - Annotation Tools]].\\
+For annotation tools, see [[nlp:software#annotation_tools|Software - Annotation Tools]]. Annotation can often be greatly sped up by building your own annotation tool with exactly the features you want for your application. This is can be a worthwhile time investment, since a well-designed tool can speed up annotation.\\
 To annotate data manually (without using [[crowdsourcing]]), practitioners generally follow these steps:
   * **Gather data**: Decide the data source and gather some data to annotate.  Be sure to consider any ethical issues.  If you want to be able to release the data publicly, check for potential copyright or privacy violations.
@@ Line 20: / Line 20: @@
   * Software
     * R's [[https://cran.r-project.org/web/packages/irr/index.html|Inter-Annotator Reliability Package]] (IRR) is great. [[https://cran.r-project.org/web/packages/irr/irr.pdf|docs]] [[https://www.andywills.info/rminr/irr.html|example]]
+==== Building Your own Annotation Tool ====
+  * For simple projects, annotation can be done in a spreadsheet
+  * When building your own annotation tool, here are some things to consider
+    * The purpose of the tool is to make the annotation faster.  Think carefully about what interface will be fastest for trained annotators.
+    * To speed up development, use whatever language and API you are familiar with or find easiest.
+    * Think very carefully about ways to reduce unnecessary mouse clicks, typing, reading text, etc.  Every mouse click counts.  Aggressively remove anything that is unnecessary, like typing escape or enter to save.  Instead, automatically save when you go to the next example, etc.
+    * Plan on doing some iterations on the tool.  You will need to try it, and change it based on your experience.
+    * It doesn't need to be perfect, it just needs to be fast to use.  It's ok to have bugs in the annotation tool if it's not widely used, and they don't slow down annotation.
+    * Don't make it full-featured.  You just need the features that make annotation fast.
 ===== Dataset and Data Selection Issues =====