NLP Wiki

User Tools

Site Tools

Trace: • optimization_in_deep_learning
ml:optimization_in_deep_learning

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
ml:optimization_in_deep_learning [2023/12/01 01:20] – jmflanigml:optimization_in_deep_learning [2025/03/25 00:49] (current) – [Effects on Optimization] jmflanig
Line 6: Line 6:
   * Weight normalization   * Weight normalization
     * Improves the conditioning of the optimization problem ([[https://arxiv.org/pdf/1602.07868.pdf|Salimans & Kingma 2016]])     * Improves the conditioning of the optimization problem ([[https://arxiv.org/pdf/1602.07868.pdf|Salimans & Kingma 2016]])
 +  * Lipschitz Constant
 +    * [[https://arxiv.org/pdf/2306.09338|Qi et al 2023 - Understanding Optimization of Deep Learning via Jacobian Matrix and Lipschitz Constant]] Talks about the effect of the Lipschitz constant on optimizing deep neural networks.
  
 ===== On Global Optimization of Neural Networks ===== ===== On Global Optimization of Neural Networks =====
Line 41: Line 43:
  
 ===== Miscellaneous Topics ===== ===== Miscellaneous Topics =====
 +
 +==== Effect of Skip Connections ====
   * [[https://arxiv.org/pdf/1702.08591.pdf|Balduzzi et al 2017 - The Shattered Gradients Problem: If resnets are the answer, then what is the question?]]   * [[https://arxiv.org/pdf/1702.08591.pdf|Balduzzi et al 2017 - The Shattered Gradients Problem: If resnets are the answer, then what is the question?]]
 +  * [[https://arxiv.org/pdf/1701.09175.pdf|Orhan & Pitkow 2017 - Skip Connections Eliminate Singularities]]
  
 ===== People ===== ===== People =====
ml/optimization_in_deep_learning.1701393653.txt.gz · Last modified: 2023/12/01 01:20 by jmflanig
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki