Normalization

Normalization can improve the optimizer's ability to train a neural network. There are two main categories of normalization procedures: activation normalization and weight normalization (Shen 2020).

• Blog post: 2019 - Normalization Techniques in Deep Neural Networks

Batch normalization is popular in computer vision, but not usually used in NLP because it doesn't work well. Layer normalization is usually used instead (see Shen 2020).

• Ioffe & Szegedy 2015 - Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Issues with RNNs
  • Laurent et al 2015 - Batch Normalized Recurrent Neural Networks
  • Cooijmans et al 2016 - Recurrent Batch Normalization
• Bjorck et al 2018 - Understanding Batch Normalization. See also section 3 of De & Smith 2020 - Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks for a different perspective.
• Shen et al 2020 - PowerNorm: Rethinking Batch Normalization in Transformers

• Layer Normalization
• RMSNorm. Improvement to layer normalization. Computationally more efficient, and gives improved invariance properties. Shown to work well for Transformers by Narang et al 2021.

• Weight normalization is billed as an alternative to batch normalization. Salimans & Kingma 2016 - Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks “…improve the conditioning of the optimization problem and we speed up convergence of stochastic gradient descent. Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also be applied successfully to recurrent models such as LSTMs and to noise-sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited.” See section 2.2 Relation to batch normalization.

• Training Setups in the Literature