NLP Wiki

User Tools

Site Tools

Trace: • nn_initialization • quantum_machine_learning • conditional_computation • optimization • regularization
ml:optimization

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
ml:optimization [2022/05/09 08:07] – [Lectures and Books] jmflanigml:optimization [2024/03/06 21:57] (current) – jmflanig
Line 10: Line 10:
 ===== Theory ===== ===== Theory =====
   * [[http://proceedings.mlr.press/v37/agarwal15.pdf|Agarwal, Leon Bottou 2015 - A Lower Bound for the Optimization of Finite Sums]]   * [[http://proceedings.mlr.press/v37/agarwal15.pdf|Agarwal, Leon Bottou 2015 - A Lower Bound for the Optimization of Finite Sums]]
 +  * [[https://arxiv.org/pdf/1309.5549.pdf|Ghadimi & Lan 2013 - Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming]] Cited by Jin 2021 for Theorem 2.10
 +  * [[https://dl.acm.org/doi/pdf/10.1145/3418526|Jin et al 2021 - On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points]] Overview of results
  
 ===== Papers ===== ===== Papers =====
Line 15: Line 17:
   * [[https://arxiv.org/pdf/1506.01186.pdf|2015 - Cyclical Learning Rates for Training Neural Networks]]   * [[https://arxiv.org/pdf/1506.01186.pdf|2015 - Cyclical Learning Rates for Training Neural Networks]]
   * [[https://arxiv.org/pdf/2002.12414.pdf|2020 - On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings]]   * [[https://arxiv.org/pdf/2002.12414.pdf|2020 - On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings]]
 +  * [[https://proceedings.mlr.press/v202/mei23a/mei23a.pdf|Mei et al 2023 - Stochastic Gradient Succeeds for Bandits]]
  
 ===== Courses ===== ===== Courses =====
-See also [[:Courses and Seminars]].+See also [[:Courses]] and [[:Seminars]].
   * [[http://www.princeton.edu/~yc5/ele522_optimization/|Large-Scale Optimization for Data Science @ Princeton]]   * [[http://www.princeton.edu/~yc5/ele522_optimization/|Large-Scale Optimization for Data Science @ Princeton]]
   * [[http://www.princeton.edu/~yc5/reading_group/index.html|Mathematical Data Science Reading Group @ Princeton]]   * [[http://www.princeton.edu/~yc5/reading_group/index.html|Mathematical Data Science Reading Group @ Princeton]]
Line 27: Line 30:
   * [[https://medium.com/intuitionmachine/the-peculiar-behavior-of-deep-learning-loss-surfaces-330cb741ec17|2017 - The Two Phases of Gradient Descent in Deep Learning]]   * [[https://medium.com/intuitionmachine/the-peculiar-behavior-of-deep-learning-loss-surfaces-330cb741ec17|2017 - The Two Phases of Gradient Descent in Deep Learning]]
   * [[https://medium.com/inveterate-learner/deep-learning-book-chapter-8-optimization-for-training-deep-models-part-i-20ae75984cb2|Optimization For Training Deep Models Part I]]   * [[https://medium.com/inveterate-learner/deep-learning-book-chapter-8-optimization-for-training-deep-models-part-i-20ae75984cb2|Optimization For Training Deep Models Part I]]
 +
 +===== People =====
 +  * [[https://scholar.google.com/citations?user=xaQuPloAAAAJ&hl=en|Dale Schuurmans]]
  
 ===== Related Pages ===== ===== Related Pages =====
 +  * [[application_optimization|Application: Optimization]]
   * [[NN Training]]   * [[NN Training]]
   * [[Optimizers]]   * [[Optimizers]]
   * [[Optimization in Deep Learning]]   * [[Optimization in Deep Learning]]
  
ml/optimization.1652083621.txt.gz · Last modified: 2023/06/15 07:36 (external edit)
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki