Ensembling models trained with different random seeds always improves performance. It is often used in competitions like WMT. However, in papers, because it always gives an improvement, researchers usually compare non-ensembling methods to other non-ensembled methods, and ensembled methods to ensembled methods. See for example of this see Gehring et al 2017.

• Ganaie et al 2021 - Ensemble Deep Learning: A Review

• Singh et al 2016 - Swapout: Learning an ensemble of deep architectures

• Fort et al 2019 - Deep Ensembles: A Loss Landscape Perspective

• Knowledge Distillation