Infinite Neural Networks

Infinite neural networks are neural networks that have an infinite number of hidden units or an infinite number of layers.

• Neural Tangent Kernel
  • Understanding the Neural Tangent Kernel (blog post)
  • Some Math behind Neural Tangent Kernel (blog post)
• Lee et al 2020 - Finite Versus Infinite Neural Networks: an Empirical Study

• Unbounded Depth NNs: Nazaret & Blei 2022 - Variational Inference for Infinitely Deep Neural Networks

See also related work here and here.

• Neal 1994 - Priors for Infinite Networks Other versions
• Williams 1996 - Computing with infinite networks
• Lee et al 2017 - Deep Neural Networks as Gaussian Processes
• Jacot et al 2018 - Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Lee et al 2019 - Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Arora et al 2019 - On Exact Computation with an Infinitely Wide Neural Net
• Aitchison 2019 - Why bigger is not always better: on finite and infinite neural networks
• Xiao et al 2019 - Disentangling Trainability and Generalization in Deep Neural Networks
• Novak et al 2019 - Neural Tangents: Fast and Easy Infinite Neural Networks in Python github Colab notebook
• He et al 2020 - Bayesian Deep Ensembles via the Neural Tangent Kernel
• Sohl-Dickstein et al 2020 - On the Infinite Width Limit of Neural Networks with a Standard Parameterization
• Yang 2020 - Tensor Programs II: Neural Tangent Kernel for Any Architecture
  • Yang & Hu 2020 - Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks
  • Yang & Littwin 2021 - Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics
  • Yang et al 2022 - Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
• Lee et al 2020 - Finite Versus Infinite Neural Networks: an Empirical Study
• Liu et al 2020 - On the linearity of large non-linear models: when and why the tangent kernel is constant
• Domingos 2020 - Every Model Learned by Gradient Descent Is Approximately a Kernel Machine
• Novak et al 2022 - Fast Finite Width Neural Tangent Kernel video github code example
• Yang et al 2022 - Efficient Computation of Deep Nonlinear Infinite-Width Neural Networks that Learn Features

Jeff's thoughts: Although objective functions for training finite neural networks are usually non-convex, for neural networks with an infinite number of hidden units (infinitely wide) they are usually convex (this is because the space of infinite neural networks is linear: any infinite NN is just a linear combination of all possible parameters in the parameter space).

• Neural Tangents paper Colab notebook

• Greg Yang