Curriculum learning (CL) is where a neural network is trained on easier examples before training on harder examples. The method for deciding which examples to train on at different times is called the curriculum, and uses a measure of difficulty for the data points. As an example, one can train on shorter sentences before adding longer sentences to the training. CL can help the model learn features that generalize better, and help the model learn faster. For an overview, see Bengio 2009.

• Wang et al 2020 - A Comprehensive Survey on Curriculum Learning
• Soviany et al 2021 - Curriculum Learning: A Survey

• Bengio et al 2009 - Curriculum Learning
• Graves 2017 - Automated Curriculum Learning for Neural Networks
• Hacohen & Weinshall 2019 - On The Power of Curriculum Learning in Training Deep Networks
• Press et al 2020 - Shortformer: Better Language Modeling using Shorter Inputs Points out that BERT also uses curriculum learning.
• 2020 - Curriculum Learning for Natural Language Understanding

• 2017 - On Convergence Property of Implicit Self-paced Objective
• 2017 - A theoretical understanding of self-paced learning
• 2015 - What Objective Does Self-paced Learning Indeed Optimize?

Curriculum learning is widely used in computer vision.

• 2014 - Learning to Execute