NLP Wiki

User Tools

Site Tools

Trace: • question_generation • data_augmentation • semantic_representations • non-autoregressive_seq2seq • alignment • gradient_clipping • visual_question_answering • morphological_analysis • generalization_in_deep_learning • tokenization
ml:nn_tricks

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
ml:nn_tricks [2023/06/15 07:36] – external edit 127.0.0.1ml:nn_tricks [2023/10/11 22:19] (current) – jmflanig
Line 12: Line 12:
     * [[Curriculum Learning]]     * [[Curriculum Learning]]
     * Overcoming [[Catastrophic Forgetting]]     * Overcoming [[Catastrophic Forgetting]]
-    * Adjust the batch size, or use gradient accumulation to simulate larger batch sizes+    * Adjust the batch size, or use gradient accumulation (see [[https://kozodoi.me/blog/20210219/gradient-accumulation|this blog]], for example) to simulate larger batch sizes
     * Try a different [[optimizers#modern_deep_learning_optimizers|optimizer]], such as [[ https://arxiv.org/pdf/1908.03265.pdf|RAdam]]     * Try a different [[optimizers#modern_deep_learning_optimizers|optimizer]], such as [[ https://arxiv.org/pdf/1908.03265.pdf|RAdam]]
     * Adjust [[https://arxiv.org/pdf/2011.02150.pdf|epsilon]] in Adam     * Adjust [[https://arxiv.org/pdf/2011.02150.pdf|epsilon]] in Adam
ml/nn_tricks.1686814574.txt.gz · Last modified: 2023/06/15 07:36 by 127.0.0.1
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki