• Duchi - Introductory Lectures on Stochastic Optimization
• Bottou et al 2016 - Optimization Methods for Large-Scale Machine Learning Survey paper that includes a bunch of theory results with proofs (also rate of convergence of SGD on non-convex objectives)
• Black-box Optimization
• Boyd & Vandenberghe - Convex Optimization
• Luenberger - Linear and Nonlinear Programming Great book

• Agarwal, Leon Bottou 2015 - A Lower Bound for the Optimization of Finite Sums

• 2018 - Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates
• 2015 - Cyclical Learning Rates for Training Neural Networks
• 2020 - On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings

See also Courses and Seminars.

• Large-Scale Optimization for Data Science @ Princeton
• Mathematical Data Science Reading Group @ Princeton
• EPFL Course - Optimization for Machine Learning
• Convex Optimization and Approximation @ Berkeley

• 2018 - Setting the learning rate of your neural network
• 2017 - The Two Phases of Gradient Descent in Deep Learning
• Optimization For Training Deep Models Part I

• NN Training
• Optimizers
• Optimization in Deep Learning