Ensembling combines the probabilistic predictions of several models to improve generalization performance. For example, ensembling models trained with different random seeds almost always improves performance. This technique is often used when performance is the main object, such as in competitions like WMT. However, in papers, because it often gives a large improvement, researchers usually compare non-ensembling methods to other non-ensembled methods, and ensembled methods to ensembled methods. See for example of this see Gehring et al 2017.

• Ganaie et al 2021 - Ensemble Deep Learning: A Review

• Singh et al 2016 - Swapout: Learning an ensemble of deep architectures

• Fort et al 2019 - Deep Ensembles: A Loss Landscape Perspective

• Knowledge Distillation