NLP Wiki

User Tools

Site Tools

Trace: • infinite_neural_networks
ml:infinite_neural_networks

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
ml:infinite_neural_networks [2022/08/26 21:24] – [Neural Tangent Kernel] jmflanigml:infinite_neural_networks [2023/06/15 07:36] (current) – external edit 127.0.0.1
Line 5: Line 5:
   * Neural Tangent Kernel   * Neural Tangent Kernel
     * [[https://rajatvd.github.io/NTK/|Understanding the Neural Tangent Kernel (blog post)]]     * [[https://rajatvd.github.io/NTK/|Understanding the Neural Tangent Kernel (blog post)]]
 +    * [[https://lilianweng.github.io/posts/2022-09-08-ntk/|Some Math behind Neural Tangent Kernel (blog post)]]
   * [[https://arxiv.org/pdf/2007.15801.pdf|Lee et al 2020 - Finite Versus Infinite Neural Networks: an Empirical Study]]   * [[https://arxiv.org/pdf/2007.15801.pdf|Lee et al 2020 - Finite Versus Infinite Neural Networks: an Empirical Study]]
  
Line 20: Line 21:
   * [[https://arxiv.org/pdf/1902.06720.pdf|Lee et al 2019 - Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent]]   * [[https://arxiv.org/pdf/1902.06720.pdf|Lee et al 2019 - Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent]]
   * [[https://arxiv.org/pdf/1904.11955.pdf|Arora et al 2019 - On Exact Computation with an Infinitely Wide Neural Net]]   * [[https://arxiv.org/pdf/1904.11955.pdf|Arora et al 2019 - On Exact Computation with an Infinitely Wide Neural Net]]
 +  * **[[https://arxiv.org/pdf/1910.08013.pdf|Aitchison 2019 - Why bigger is not always better: on finite and infinite neural networks]]**
   * [[https://arxiv.org/pdf/1912.13053.pdf|Xiao et al 2019 - Disentangling Trainability and Generalization in Deep Neural Networks]]   * [[https://arxiv.org/pdf/1912.13053.pdf|Xiao et al 2019 - Disentangling Trainability and Generalization in Deep Neural Networks]]
   * [[https://arxiv.org/pdf/1912.02803.pdf|Novak et al 2019 - Neural Tangents: Fast and Easy Infinite Neural Networks in Python]] [[https://github.com/google/neural-tangents|github]] [[https://colab.research.google.com/github/google/neural-tangents/blob/master/notebooks/neural_tangents_cookbook.ipynb|Colab notebook]]   * [[https://arxiv.org/pdf/1912.02803.pdf|Novak et al 2019 - Neural Tangents: Fast and Easy Infinite Neural Networks in Python]] [[https://github.com/google/neural-tangents|github]] [[https://colab.research.google.com/github/google/neural-tangents/blob/master/notebooks/neural_tangents_cookbook.ipynb|Colab notebook]]
Line 25: Line 27:
   * [[https://arxiv.org/pdf/2001.07301.pdf|Sohl-Dickstein et al 2020 - On the Infinite Width Limit of Neural Networks with a Standard Parameterization]]   * [[https://arxiv.org/pdf/2001.07301.pdf|Sohl-Dickstein et al 2020 - On the Infinite Width Limit of Neural Networks with a Standard Parameterization]]
   * [[https://arxiv.org/pdf/2006.14548.pdf|Yang 2020 - Tensor Programs II: Neural Tangent Kernel for Any Architecture]]   * [[https://arxiv.org/pdf/2006.14548.pdf|Yang 2020 - Tensor Programs II: Neural Tangent Kernel for Any Architecture]]
 +    * [[https://arxiv.org/pdf/2011.14522.pdf|Yang & Hu 2020 - Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks]]
     * [[https://arxiv.org/pdf/2105.03703.pdf|Yang & Littwin 2021 - Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics]]     * [[https://arxiv.org/pdf/2105.03703.pdf|Yang & Littwin 2021 - Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics]]
 +    * [[https://arxiv.org/pdf/2203.03466.pdf|Yang et al 2022 - Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer]]
   * **[[https://arxiv.org/pdf/2007.15801.pdf|Lee et al 2020 - Finite Versus Infinite Neural Networks: an Empirical Study]]**   * **[[https://arxiv.org/pdf/2007.15801.pdf|Lee et al 2020 - Finite Versus Infinite Neural Networks: an Empirical Study]]**
 +  * [[https://arxiv.org/abs/2010.01092|Liu et al 2020 - On the linearity of large non-linear models: when and why the tangent kernel is constant]]
 +  * [[https://arxiv.org/pdf/2012.00152.pdf|Domingos 2020 - Every Model Learned by Gradient Descent Is Approximately a Kernel Machine]]
   * [[https://arxiv.org/pdf/2206.08720.pdf|Novak et al 2022 - Fast Finite Width Neural Tangent Kernel]] [[https://youtu.be/8MWOhYg89fY?t=10984|video]] [[https://github.com/google/neural-tangents|github]] [[https://colab.research.google.com/github/google/neural-tangents/blob/main/notebooks/empirical_ntk_fcn.ipynb|code example]]   * [[https://arxiv.org/pdf/2206.08720.pdf|Novak et al 2022 - Fast Finite Width Neural Tangent Kernel]] [[https://youtu.be/8MWOhYg89fY?t=10984|video]] [[https://github.com/google/neural-tangents|github]] [[https://colab.research.google.com/github/google/neural-tangents/blob/main/notebooks/empirical_ntk_fcn.ipynb|code example]]
- +  * **[[https://openreview.net/pdf?id=tUMr0Iox8XW|Yang et al 2022 - Efficient Computation of Deep Nonlinear Infinite-Width Neural Networks that Learn Features]]**
- +
  
 ===== Notes ===== ===== Notes =====
Line 37: Line 41:
 ===== Software ===== ===== Software =====
   * [[https://github.com/google/neural-tangents|Neural Tangents]] [[https://arxiv.org/pdf/1912.02803.pdf|paper]] [[https://colab.research.google.com/github/google/neural-tangents/blob/master/notebooks/neural_tangents_cookbook.ipynb|Colab notebook]]   * [[https://github.com/google/neural-tangents|Neural Tangents]] [[https://arxiv.org/pdf/1912.02803.pdf|paper]] [[https://colab.research.google.com/github/google/neural-tangents/blob/master/notebooks/neural_tangents_cookbook.ipynb|Colab notebook]]
 +
 +===== People =====
 +  * [[https://scholar.google.com/citations?user=Xz4RAJkAAAAJ&hl=en|Greg Yang]]
  
ml/infinite_neural_networks.1661549077.txt.gz · Last modified: 2023/06/15 07:36 (external edit)
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki