See also NLP Progress, Wikipedia List of datasets, and nlp-datasets. Also data preparation.

• BNC corpus
• Gigaword
• Common crawl
• Bookcorpus (Used in BERT)

• GLUE: paper - Warning: Has an issue with QQP and WNLI due to dev and tests sets not coming from the same distribution. See FAQ here.
• SuperGLUE: paper - A more difficult version of GLUE.
• CLUE: paper - Like GLUE, but for Chinese
• MMMLU: Hendrycks et al 2020 - Measuring Massive Multitask Language Understanding This dataset is a popular dataset for LLMs to evaluate on (for example GPT-4, etc). However, it has two serious issues. 1) the test set is available on the web, which means LLMs are likely contaminated, and 2) the datasets has no in-domain training data, and can only be evaluated in a few-shot manner. This make is impossible to properly compare to prior fine-tuned methods.

• Survey on Multilingual NLP Datasets: List of Datasets and Paper

Some standard seq2seq datasets.

• Microsoft Research Paraphrase Corpus, paper