NLP Wiki

User Tools

Site Tools

Trace:
ml:optimizers

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
ml:optimizers [2026/07/22 06:25] – [Modern Deep Learning Optimizers] jmflanigml:optimizers [2026/07/31 23:09] (current) – [Modern Deep Learning Optimizers] jmflanig
Line 39: Line 39:
     * Background on norms: [[https://arxiv.org/pdf/2409.20325|Bernstein & Newhouse 2024 - Old Optimizer, New Norm: An Anthology]]     * Background on norms: [[https://arxiv.org/pdf/2409.20325|Bernstein & Newhouse 2024 - Old Optimizer, New Norm: An Anthology]]
     * Applied to larger scale LLM training: [[https://arxiv.org/pdf/2502.16982|Liu et al 2025 - Muon is Scalable for LLM Training]]     * Applied to larger scale LLM training: [[https://arxiv.org/pdf/2502.16982|Liu et al 2025 - Muon is Scalable for LLM Training]]
-    * Muon is used in Deepseek V4 training+    * Muon is used in Deepseek V4 and Kimi 3 training
  
 ==== Provably Linearly-Convergent Optimizers ==== ==== Provably Linearly-Convergent Optimizers ====
ml/optimizers.txt · Last modified: 2026/07/31 23:09 by jmflanig
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki