Layer Normalization

Weight Normalization, alternative to batch normalization. Salimans & Kingma 2016 - Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks “…improve the conditioning of the optimization problem and we speed up convergence of stochastic gradient descent. Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also be applied successfully to recurrent models such as LSTMs and to noise-sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited.” See section 2.2 Relation to batch normalization.

RMSNorm. Improvement to layer normalization. Computationally more efficient, and gives improved invariance properties. Confirmed by Narang et al 2021 to work well.