Differences

This shows you the differences between two versions of the page.

--- ml:infinite_neural_networks [2022/08/26 20:34] – [Neural Tangent Kernel] jmflanig
+++ ml:infinite_neural_networks [2023/06/15 07:36] (current) – external edit 127.0.0.1
@@ Line 5: / Line 5: @@
   * Neural Tangent Kernel
     * [[https://rajatvd.github.io/NTK/|Understanding the Neural Tangent Kernel (blog post)]]
+    * [[https://lilianweng.github.io/posts/2022-09-08-ntk/|Some Math behind Neural Tangent Kernel (blog post)]]
+  * [[https://arxiv.org/pdf/2007.15801.pdf|Lee et al 2020 - Finite Versus Infinite Neural Networks: an Empirical Study]]
 ===== Papers =====
-See also related work [[https://github.com/google/neural-tangents#references|here]] and [[https://github.com/google/neural-tangents/wiki/Overparameterized-Neural-Networks:-Theory-and-Empirics|here]].
-  * [[https://arxiv.org/pdf/1902.06720.pdf|Lee et al 2019 - Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent]]
-  * [[https://arxiv.org/pdf/1912.13053.pdf|Xiao et al 2019 - Disentangling Trainability and Generalization in Deep Neural Networks]]
-  * [[https://arxiv.org/pdf/2007.15801.pdf|Lee et al 2020 - Finite Versus Infinite Neural Networks: an Empirical Study]]
   * Unbounded Depth NNs: [[https://proceedings.mlr.press/v162/nazaret22a/nazaret22a.pdf|Nazaret & Blei 2022 - Variational Inference for Infinitely Deep Neural Networks]]
 ==== Neural Tangent Kernel ====
-See also the related work [[https://github.com/google/neural-tangents#references|here]].
+See also related work [[https://github.com/google/neural-tangents#references|here]] and [[https://github.com/google/neural-tangents/wiki/Overparameterized-Neural-Networks:-Theory-and-Empirics|here]].
+  * [[https://www.cs.toronto.edu/~radford/ftp/pin.pdf|Neal 1994 - Priors for Infinite Networks]] [[https://www.cs.toronto.edu/~radford/pin.abstract.html|Other versions]]
   * [[https://papers.nips.cc/paper/1996/file/ae5e3ce40e0404a45ecacaaf05e5f735-Paper.pdf|Williams 1996 - Computing with infinite networks]]
   * [[https://arxiv.org/pdf/1711.00165.pdf|Lee et al 2017 - Deep Neural Networks as Gaussian Processes]]
   * [[https://arxiv.org/pdf/1806.07572.pdf|Jacot et al 2018 - Neural Tangent Kernel: Convergence and Generalization in Neural Networks]]
+  * [[https://arxiv.org/pdf/1902.06720.pdf|Lee et al 2019 - Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent]]
   * [[https://arxiv.org/pdf/1904.11955.pdf|Arora et al 2019 - On Exact Computation with an Infinitely Wide Neural Net]]
+  * **[[https://arxiv.org/pdf/1910.08013.pdf|Aitchison 2019 - Why bigger is not always better: on finite and infinite neural networks]]**
+  * [[https://arxiv.org/pdf/1912.13053.pdf|Xiao et al 2019 - Disentangling Trainability and Generalization in Deep Neural Networks]]
   * [[https://arxiv.org/pdf/1912.02803.pdf|Novak et al 2019 - Neural Tangents: Fast and Easy Infinite Neural Networks in Python]] [[https://github.com/google/neural-tangents|github]] [[https://colab.research.google.com/github/google/neural-tangents/blob/master/notebooks/neural_tangents_cookbook.ipynb|Colab notebook]]
   * [[https://papers.nips.cc/paper/2020/file/0b1ec366924b26fc98fa7b71a9c249cf-Paper.pdf|He et al 2020 - Bayesian Deep Ensembles via the Neural Tangent Kernel]]
   * [[https://arxiv.org/pdf/2001.07301.pdf|Sohl-Dickstein et al 2020 - On the Infinite Width Limit of Neural Networks with a Standard Parameterization]]
+  * [[https://arxiv.org/pdf/2006.14548.pdf|Yang 2020 - Tensor Programs II: Neural Tangent Kernel for Any Architecture]]
+    * [[https://arxiv.org/pdf/2011.14522.pdf|Yang & Hu 2020 - Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks]]
+    * [[https://arxiv.org/pdf/2105.03703.pdf|Yang & Littwin 2021 - Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics]]
+    * [[https://arxiv.org/pdf/2203.03466.pdf|Yang et al 2022 - Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer]]
+  * **[[https://arxiv.org/pdf/2007.15801.pdf|Lee et al 2020 - Finite Versus Infinite Neural Networks: an Empirical Study]]**
+  * [[https://arxiv.org/abs/2010.01092|Liu et al 2020 - On the linearity of large non-linear models: when and why the tangent kernel is constant]]
+  * [[https://arxiv.org/pdf/2012.00152.pdf|Domingos 2020 - Every Model Learned by Gradient Descent Is Approximately a Kernel Machine]]
   * [[https://arxiv.org/pdf/2206.08720.pdf|Novak et al 2022 - Fast Finite Width Neural Tangent Kernel]] [[https://youtu.be/8MWOhYg89fY?t=10984|video]] [[https://github.com/google/neural-tangents|github]] [[https://colab.research.google.com/github/google/neural-tangents/blob/main/notebooks/empirical_ntk_fcn.ipynb|code example]]
+  * **[[https://openreview.net/pdf?id=tUMr0Iox8XW|Yang et al 2022 - Efficient Computation of Deep Nonlinear Infinite-Width Neural Networks that Learn Features]]**
 ===== Notes =====
@@ Line 32: / Line 41: @@
 ===== Software =====
   * [[https://github.com/google/neural-tangents|Neural Tangents]] [[https://arxiv.org/pdf/1912.02803.pdf|paper]] [[https://colab.research.google.com/github/google/neural-tangents/blob/master/notebooks/neural_tangents_cookbook.ipynb|Colab notebook]]
+===== People =====
+  * [[https://scholar.google.com/citations?user=Xz4RAJkAAAAJ&hl=en|Greg Yang]]