Differences

This shows you the differences between two versions of the page.

--- ml:ensembling [2021/10/29 12:28] – [Overviews] jmflanig
+++ ml:ensembling [2023/06/15 07:36] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
 ====== Ensembling ======
-Ensembling models trained with different random seeds always improves performance.  It is often used in competitions like WMT.  However, in papers, because it always gives an improvement, researchers usually compare non-ensembling methods to other non-ensembled methods, and ensembled methods to ensembled methods.  See for example of this see [[https://arxiv.org/pdf/1705.03122.pdf|Gehring et al 2017]].
+Ensembling combines several models to improve generalization performance.  For example, ensembling models trained with different random seeds almost always improves performance.  This technique is often used when performance is the main object, such as in competitions like [[https://machinetranslate.org/wmt|WMT]].  However, in papers, because it often gives a large improvement, researchers usually compare non-ensembling methods to other non-ensembled methods, and ensembled methods to ensembled methods.  As an example of this see [[https://arxiv.org/pdf/1705.03122.pdf|Gehring et al 2017]].
+==== Basic Method ====
+For models trained with cross-entropy, //the standard method of ensembling in NLP is to just average the probabilities of the models at test time and predict using this probability//.  There are two standard ways to create the different models for ensembling:
+  * **Multirun ensembling**: models come from training runs with different random seeds
+  * **Checkpoint ensembling**: models come from different checkpoints of a single training run. This has the advantage that a only single training run is needed
+See [[https://www.amazon.com/Neural-Machine-Translation-Philipp-Koehn/dp/1108497322|Koehn 2020]] p. 148 or [[http://mt-class.org/jhu/assets/nmt-book.pdf#page=67|pdf here]].
 ===== Overviews =====
   * [[https://arxiv.org/pdf/2104.02395.pdf|Ganaie et al 2021 - Ensemble Deep Learning: A Review]]
+===== Papers =====
+  * [[https://proceedings.neurips.cc/paper/2016/file/c51ce410c124a10e0db5e4b97fc2af39-Paper.pdf|Singh et al 2016 - Swapout: Learning an ensemble of deep architectures]]
 ===== Theory =====
   * [[https://arxiv.org/pdf/1912.02757.pdf|Fort et al 2019 - Deep Ensembles: A Loss Landscape Perspective]]
+===== Related Pages =====
+  * [[Knowledge Distillation]]