ml:nn_initialization

This is an old revision of the document!

Neural Network Initialization

Glorot (Xavier) initialization: Glorot & Bengio 2010 - Understanding the Difficulty of Training Deep Feedforward Neural Networks (use with sigmoid activations)
He initialization: He et al 2015 (use with ReLu activations)
ADMIN: Liu et al 2020 - Very Deep Transformers for Neural Machine Translation For use with very deep Transformers

Resources

https://stats.stackexchange.com/questions/319323/whats-the-difference-between-variance-scaling-initializer-and-xavier-initialize

ml/nn_initialization.1613590693.txt.gz · Last modified: 2023/06/15 07:36 (external edit)