NLP Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Regularization ======

===== Regularization in Deep Learning =====
==== Dropout ====
  * Dropout: [[https://arxiv.org/pdf/1207.0580.pdf|Hinton et al 2012 - Improving Neural Networks by Preventing Co-adaptation of Feature Detectors]]
  * DropConnect: [[http://proceedings.mlr.press/v28/wan13.pdf|Wang et al 2013 - Regularization of Neural Networks using DropConnect]]
  * [[https://nlp.stanford.edu/pubs/sidaw13fast.pdf|Wang & Manning 2013 - Fast dropout training]] Shows that dropout is an approximation to an objective, and directly optimizes a fast approximation to this objective. "We show how to do fast dropout training by sampling from or integrating a Gaussian approximation, instead of doing Monte Carlo optimization of this objective. This approximation, justified by the central limit theorem and empirical evidence, gives an order of magnitude speedup and more stability"
  * [[https://papers.nips.cc/paper/2013/file/7b5b23f4aadf9513306bcd59afb6e4c9-Paper.pdf|Adaptive Dropout]]  Learns a dropout network during training ([[https://paperswithcode.com/method/adaptive-dropout|summary]])
  * [[https://arxiv.org/pdf/1606.01305.pdf|Zoneout (Krueger et al 2016)]] (For regularizing RNNs) "At each timestep, zoneout stochastically forces some hidden units to maintain their previous values"
  * [[https://arxiv.org/pdf/1909.11299.pdf|Lee et al 2019 - Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models]]
  * LayerDrop: [[https://arxiv.org/pdf/1909.11556.pdf|Fan et al 2019 - Reducing Transformer Depth on Demand with Structured Dropout]] Regularizes networks to allow extraction of smaller networks of any depth at test time without needing to finetune.
  * [[https://arxiv.org/pdf/2101.01761.pdf|Pham & Le 2021 - AutoDropout: Learning Dropout Patterns to Regularize Deep Networks]]  Shows an improvement of 1-2 BLEU for machine translation. Downside: computationally expensive.
  * [[https://arxiv.org/pdf/2106.14448.pdf|Liang et al 2021 - R-Drop: Regularized Dropout for Neural Networks]] Universally works, except see the github issues about reproducibility

==== Early-stopping ====

==== Weight decay =====
Although L2 regularization (weight decay) wasn't popular in early deep learning models in NLP, it has become popular in pre-trained transformer models.  For example, BERT and RoBERTa both use a weight decay of .01 ([[https://arxiv.org/pdf/1907.11692.pdf|RoBERTa paper]]).  Although Adam with weight decay isn't L2 regularization, it can be fixed using [[https://arxiv.org/pdf/1711.05101.pdf|WAdam]] (see below).
  * WAdam: [[https://arxiv.org/pdf/1711.05101.pdf|Loshchilov & Hutter 2017 - Decoupled Weight Decay Regularization]] Fixes weight decay in Adam so that it correctly implements L2 regularization.  Applied to the Transformer in [[https://arxiv.org/pdf/2006.00719.pdf|Yao 2020]]

==== Max-Norm Regularization ====

==== $L_p$ Regularization =====
$L_p$ regularization is regularization with an $L_p$ norm term.  Popular choices of $p$ are  $L_2$ (weight decay), $L_1$ (Lasso, induces sparsity while still being convex), $L_0$ (very non-convex: counts the number of non-zero parameters. difficult to optimize), and $L_\infty$ (max-norm regularization).  $L_1$ regularization combined with $L_2$ regularization (adding both regularization terms to the objective) is called an elastic net, and sometimes performs better than either one separately.

  * $L_0$ regularization
    * [[https://arxiv.org/pdf/1712.01312.pdf|Louizos et al 2018 - Learning Sparse Neural Networks through L0 Regularization]]

==== Sparsity-Inducing Regularizers ====
  * **Structured Sparsity**
    * [[https://www.di.ens.fr/~fbach/STS394.pdf|Bach et al 2012 - Structured Sparsity through Convex Optimization]]
    * [[https://arxiv.org/pdf/1608.03665.pdf|Wen et al 2016 - Learning Structured Sparsity in Deep Neural
Networks]]

==== Label Smoothing ====
Introduced here: [[https://arxiv.org/pdf/1512.00567.pdf|Szegedy et al 2015 - Rethinking the Inception Architecture for Computer Vision]], and used in BERT.
  * [[https://arxiv.org/pdf/1906.02629.pdf|2019 - When Does Label Smoothing Help?]]
  * [[http://proceedings.mlr.press/v119/lukasik20a/lukasik20a.pdf|2020 - Does Label Smoothing Mitigate Label Noise?]]

==== Batch Size ====
Smaller batch sizes usually generalize better, and batch size is a hyperparameter that can be tuned. From [[https://arxiv.org/pdf/2002.12915.pdf|Wei et al 2020]]: "The implicit regularization effect
of stochasticity in SGD has been empirically studied in the context of small v.s. large batch training Keskar et al. (2016), where it is observed that noisier small-batch SGD converges to “flatter” local minima which generalize better, whereas large-batch SGD converges “sharper” local minima which generalize more poorly."  There's also theory about this going back to the 90's. 

==== Batch Normalizaton ====
Batch normalization has been observed to have a regularizing effect.

==== Learning Rate Schedule ====
**Regularizing effect:** A large initial learning rate can have a regularizing effect, and is commonly used for training Transformers.  From [[https://arxiv.org/pdf/1907.04595.pdf|Li et al 2019]]: "... a large initial learning rate is required to successfully train a deep network even though it slows down optimization of the train loss. Modern state-of-the-art architectures
typically start with a large learning rate and anneal it at a point when the model’s fit to the training data plateaus [25, 32, 17, 42]. Meanwhile, models trained using only small learning rates have been found to
generalize poorly despite enjoying faster optimization of the training loss."

===== Theory =====
==== Dropout ====
From [[https://arxiv.org/pdf/1609.08017.pdf|Ma et al 2017]]:
<blockquote>
In their pioneering work, Hinton et al. (2012) and Srivastava et al. (2014) interpreted dropout as
an extreme form of model combination (aka. model ensemble) with extensive parameter/weight
sharing, and they proposed to learn the combination through minimizing an appropriate expected loss.
Interestingly, they also pointed out that for a single logistic neural unit, the output of dropout is in
fact the geometric mean of the outputs of the model ensemble with shared parameters. Subsequently,
many theoretical justifications of dropout have been explored, and we can only mention a few here due
to space limits. Building on the weight sharing perspective, Baldi & Sadowski (2013; 2014) analyzed
the ensemble averaging property of dropout in deep non-linear logistic networks, and supported
the view that dropout is equivalent to applying stochastic gradient descent on some regularized
loss function. Wager et al. (2013) treated dropout as an adaptive regularizer for generalized linear
models (GLMs). Helmbold & Long (2016) discussed the differences between dropout and traditional
weight decay regularization. In terms of statistical learning theory, Gao & Zhou (2014) studied the
Rademacher complexity of different types of dropout, showing that dropout is able to reduce the
Rademacher complexity polynomially for shallow neural networks (with one or no hidden layers) and
exponentially for deep neural networks. This latter work (Gao & Zhou, 2014) formally demonstrated
that dropout, due to its regularizing effect, contributes to reducing the inherent model complexity, in
particular the variance component in the generalization error.

Seen as a model combination technique, it is intuitive that dropout contributes to reducing the variance
of the model performance. Surprisingly, dropout has also been shown to play some role in reducing
the model bias. For instance, Jain et al. (2015) studied the ability of dropout training to escape local
minima, hence leading to reduced model bias. Other studies (Chen et al., 2014; Helmbold & Long,
2014; Wager et al., 2014) focus on the effect of the dropout noise on models with shallow architectures.
We noted in passing that there are also some work (Kingma et al., 2015; Gal & Ghahramani, 2015;
2016) trying to understand dropout from the Bayesian perspective.
</blockquote>
=== Papers ===
  * [[http://proceedings.mlr.press/v28/wang13a.pdf|Wang & Manning 2013 - Fast Dropout Training]]
  * [[https://arxiv.org/pdf/1506.02142.pdf|Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning]] and slides [[https://tensorchiefs.github.io/bbs/files/dropouts-brownbag.pdf|here]].
  * [[https://arxiv.org/pdf/2002.12915.pdf|Wei et al 2020 - The Implicit and Explicit Regularization Effects of Dropout]]

==== Learning Rate Schedule ====
  * [[https://arxiv.org/pdf/1907.04595.pdf|Li et al 2019 - Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks]]

==== L2 Regularization ====
  * L2 Regularization is almost equivalent to early-stopping.  See slides 18-21 [[https://www.stat.cmu.edu/~ryantibs/convexopt/lectures/stochastic-gd.pdf|here]] (2020 version).

===== Related Pages =====
  * [[Ensembling]]
  * [[ml:theory:Generalization in Deep Learning]]