NLP Wiki

User Tools

Site Tools

Trace:
ml:optimizers

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
ml:optimizers [2025/03/26 20:02] – [Second-Order Optimizers] jmflanigml:optimizers [2026/07/31 23:09] (current) – [Modern Deep Learning Optimizers] jmflanig
Line 39: Line 39:
     * Background on norms: [[https://arxiv.org/pdf/2409.20325|Bernstein & Newhouse 2024 - Old Optimizer, New Norm: An Anthology]]     * Background on norms: [[https://arxiv.org/pdf/2409.20325|Bernstein & Newhouse 2024 - Old Optimizer, New Norm: An Anthology]]
     * Applied to larger scale LLM training: [[https://arxiv.org/pdf/2502.16982|Liu et al 2025 - Muon is Scalable for LLM Training]]     * Applied to larger scale LLM training: [[https://arxiv.org/pdf/2502.16982|Liu et al 2025 - Muon is Scalable for LLM Training]]
 +    * Muon is used in Deepseek V4 and Kimi 3 training
  
 ==== Provably Linearly-Convergent Optimizers ==== ==== Provably Linearly-Convergent Optimizers ====
ml/optimizers.1743019366.txt.gz · Last modified: 2025/03/26 20:02 by jmflanig
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki