NLP Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Neural Network Training ======
===== Overviews =====
  * [[https://ucsc.primo.exlibrisgroup.com/permalink/01CDL_SCR_INST/1kt68tt/alma991025070453104876|Chapter 11 of Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow (UCSC login required)]]  Excellent introduction to training neural networks
  * [[https://www.deeplearningbook.org/contents/optimization.html|Deep Learning Chapter 8: Training Deep Models]]
  * [[https://arxiv.org/pdf/1803.09820.pdf|Smith 2018 - A Disciplined Approach to Neural Network Hyper-parameters]] Leslie's opinon, but has some good insights
  * [[http://karpathy.github.io/2019/04/25/recipe/|Karpathy 2019 - A Recipe for Training Neural Networks]] Good advice, especially looking at the data
  * [[https://classes.soe.ucsc.edu/nlp202/Winter22/slides/nn-training.pdf#page=9|NLP 202 Winter 2022 - Training Neural Networks (slides)]]

===== Topics =====
  * [[nlp:Data Preparation]] and [[nlp:Tokenization]]
  * [[Hyperparameter Tuning]]
    * Random search is a good strategy, see [[https://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf|Bergstra & Bengio 2012 - Random Search for Hyper-Parameter Optimization]]
  * [[NN Initialization|Initialization]]
  * [[Normalization]]
  * [[Optimizers]]
  * [[Learning Rate]]
    * [[https://www.jeremyjordan.me/nn-learning-rate/|Blog Post: Setting the learning rate of your neural network]]
  * [[Loss Functions]]
  * [[Regularization]]
  * [[Fine-Tuning]] and [[nlp:Pretraining]]
  * **[[NN Tricks|Neural Network Tricks]]**
    * Tricks such as [[Curriculum Learning]], etc
    * [[nlp:Transformers#Training|Transformer Training Tricks]]
    * Residual connections, [[https://arxiv.org/pdf/2003.04887.pdf|ReZero]]
    * [[https://arxiv.org/pdf/1710.03740|Mixed Precision Training]] (also [[https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html|Train With Mixed Precision - NVIDIA Docs]], see other papers as well)
  * [[Large-Scale]] and [[Distributed Training]]

===== Training Setups in the Literature =====
Training setups have evolved over time. Here are some from the literature.

  * [[https://arxiv.org/pdf/1409.0473.pdf|Bahdanau 2014]]: Minibatch stochastic gradient descent (SGD) with Adadelta, trained for 5 days.  Recurrent weight matrices initialized as random orthogonal matrices. For feedforward weight matrices, initialized by sampling each element from the Gaussian distribution of mean 0 and variance 0.00.  Biases initialized to 0.
  * [[https://arxiv.org/pdf/1603.01354.pdf|Ma & Hovy 2016]]: Minibatch SGD (batch size 10) with momentum (.9), gradient clipping (5.0) and learning rate decay (this setup performed similarly to Adam for them). Early-stopping on validation set. Dropout with rate = .5.
  * [[https://arxiv.org/pdf/1705.03122.pdf|Gehring et al 2017 - Convolutional Sequence to Sequence Learning]] They used a lot of tricks. Worth taking a look.
  * **[[https://arxiv.org/pdf/1706.03762.pdf|Transformer (Vaswani et al 2017)]]**: Uses Adam with warmup, residual dropout, and label smoothing. Adam parameters β1 = 0.9, β2 = 0.98 and epsilon = 10−9.  Warmup increases "the learning rate linearly for the first warmup_steps training steps, and decreasing it thereafter proportionally to the inverse square root of the step number." Trained for 100,000 steps (12 hours).
  * Transformer: [[https://arxiv.org/pdf/1804.00247.pdf|Popel & Bojar 2018 - Training Tips for the Transformer Model]]
  * **[[https://arxiv.org/pdf/1810.04805.pdf|BERT (Devlin et al 2019)]]**: Uses Adam with linear warmup, and linearly decaying stepsize.  Regularization: dropout and weight decay (L2 regularizer). Does not use label smoothing like the Transformer. "BERT is optimized with Adam (Kingma and Ba,2015) using the following parameters: β1 = 0.9, β2 = 0.999, espilon = 1e-6 and L2 weight decay of 0.01 with steps to a peak value of 1e-4, and then linearly decayed. BERT trains with a dropout of 0.1 on all layers and attention weights, and a GELU activation function (Hendrycks and Gimpel, 2016). Models are pretrained for S = 1,000,000 updates, with minibatches containing B = 256 sequences of maximum length T = 512 tokens." (from [[https://arxiv.org/pdf/1907.11692.pdf|Liu et al 2020]]) Trained for  1,000,000 steps.  A search over hyperparameters for training BERT is given in [[https://arxiv.org/pdf/1904.00962.pdf|You et al 2020]]. As noted in [[https://arxiv.org/pdf/2012.15832.pdf|Press 2020]], trains on shorter sentences (<128 tokens) for 90% of the training, before training on longer sentences (<512) for the last 10%.
  * TODO: GPT-1 (p. 5)
  * Low-resource NMT system: [[https://arxiv.org/pdf/1905.11901.pdf|Sennrich & Zhang 2019 - Revisiting Low-Resource Neural Machine Translation: A Case Study]] Uses bideep RNN, label smoothing, different dropout rates for output word embeddings, input word embeddings and hidden layers, tied embeddings, layer normalization, tuned BPE vocabulary size (reduced from larger data scenarios). Trained with Adam with early-stopping on a dev set using BLEU.
  * TODO: BART
  * [[https://aclanthology.org/2021.emnlp-main.831.pdf|Academic Budget BERT]]
  * [[https://arxiv.org/pdf/2201.11990.pdf|Megatron-Turing NLG]]
  * [[https://arxiv.org/pdf/2204.02311.pdf|PaLM]]

^ Paper ^ Architecture ^ Optimizer ^ Optimizer Hyperparameters ^ Initialization ^ Normalization ^ Regularizer ^ Learning Schedule ^ Stopping Criterion ^ Activation Function ^ Tokenization ^ Extras ^
| [[https://arxiv.org/pdf/1409.0473.pdf|Bahdanau 2014]] | Seq2seq BiLSTM + attention | Adadelta + gradient clipping (on norm) | espilon = 10−6, ρ = 0.95, gradient clipping = 1 | Random orthogonal + Gaussian | none | none | (no learning rate, set by Adadelta) | 5 days | LSTM & Tahn | Moses | |
| [[https://arxiv.org/pdf/1603.01354.pdf|Ma & Hovy 2016]] | BiLSTM +CNN word embds | SGD w/ momentum + gradient clipping | momentum = .9, gradient clipping = 5 |  GloVe & Uniform [-sqrt(3/dim),sqrt(3/dim)] | | Dropout (.5) | Rate decay | Early-stopping | Sigmoid & Tahn | Tokens| | 
| [[https://arxiv.org/pdf/1705.03122.pdf|Gehring et al 2017]] | CNN seq2seq | Nesterov + gradient clipping (on norm) | | | | | | | | | |
| [[https://arxiv.org/pdf/1706.03762.pdf|Vaswani et al 2017]] | Transformer | Adam | β1 = 0.9, β2 = 0.999, espilon = 1e-6 | Glorot/fan_avg | Layer normalization | Dropout, label smoothing | Linear warm-up + 1/sqrt(step_number) |100,000 steps | ReLu | WordPiece |
| [[https://arxiv.org/pdf/1810.04805.pdf|BERT (Devlin et al 2019)]] | Transformer | Adam | β1 = 0.9, β2 = 0.999, espilon = 1e-6 | (Glorot/fan_avg?) | Layer normalization | L2 weight decay of 0.01 | Linear warmup + linear decay | 1,000,000 | GELU | WordPiece | Trains on short sentences first |
| | | | | | | | | | | | |

===== Alternative Training Methods =====
See [[Alternative Training Methods|Neural Networks: Alternative Training Methods]]

===== Related Pages =====
  * [[Alternative Training Methods]]
  * [[nlp:Data Preparation]]
  * [[Hyperparameter Tuning]]
  * [[NN Initialization]]
  * [[NN Tricks]]
  * [[Optimizers]]
  * [[Regularization]]