Bayesian Methods

Bayesian methods are a sub-field of statistics (bayesian statistics), and are often used in machine learning. Bayesian methods assume a prior over the space of models (the prior belief), and after observing the data, update the belief over the space of models (the posterior belief). The posterior belief can be used to make predictions, etc. (This is in contrast to frequentist methods that are “designed to create procedures with certain frequency gaurantees (consistency, coverage, minimaxity etc)” (Nonparametric Bayesian Methods, Chapter 8)). Bayesian methods often have very good frequentist properties, and often perform well in practice.

There is recent interest in combining bayesian methods with deep learning. For examples, see the Bayesian Deep Learning Workshop at NeurIPS. Bayesian methods such as bayesian optimization are also used in hyperparameter tuning.

• Johnson et al 2006 - Bayesian Inference for PCFGs via Markov Chain Monte Carlo
• Johnson 2007 - Why doesn’t EM find good HMM POS-taggers?

• Blog post: Intro to Bayesian Deep Learning
• Wilson 2020 - The Case for Bayesian Deep Learning
• Jospin et al 2020 - Hands-on Bayesian Neural Networks – a Tutorial for Deep Learning Users
• Goan & Fookes 2020 - Bayesian Neural Networks: An Introduction and Survey
• Charnock et al 2020 - Bayesian Neural Networks (Draft book chapter)

• Cai et al 2018 - VIBNN: Hardware Acceleration of Bayesian Neural Networks
• Shridhar et al 2019 - A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• He et al 2020 - Bayesian Deep Ensembles via the Neural Tangent Kernel
• Karaletsos & Bui 2020 - Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights

• Tran et al 2021 - A Practical Tutorial on Variational Bayes

• Wang & Blei - Frequentist Consistency of Variational Bayes

Bayesian nonparametric methods are a sub-area of bayesian statistics, and are also commonly used in machine learning. They were very popular in NLP before the deep learning era started in 2014.

• Orbanz & Teh 2010 - Bayesian Nonparametric Models
• Wasserman - Nonparametric Bayesian Methods Notes from a statistics course

• Talks (with videos)
  • Talk by Michael Jordan: Dirichlet Processes, Chinese Restaurant Processes, and All That
  • Yee Whye Teh's talks. Video Another video Slides: Introduction to Bayesian Nonparametrics
  • Peter Orbanzs' Resources on Bayesian Nonparametrics
• Wikipedia - Dirichlet process
• Wikipedia - Chinese restaurant process
• Wikipedia - Pitman–Yor process

• Blei & Jordan 2004 - Variational Inference for Dirichlet Process Mixtures
• Goldwater et al 2005 - Interpolating Between Types and Tokens by Estimating Power-Law Generators “We show that taking a particular stochastic process – the Pitman-Yor process – as an adaptor justifies the appearance of type frequencies in formal analyses of natural language.”
• Teh 2006 - A Bayesian Interpretation of Interpolated Kneser-Ney
• Johnson et al 2006 - Adaptor Grammars: A Framework for Specifying Compositional Nonparametric Bayesian Models
• Teh 2007 - Hierarchical Dirichlet Processes See also Wikipedia - Hierarchical Dirichlet process Viswanathan & Faruqui 2014
• Shindo et al 2012 - Bayesian Symbol-Refined Tree Substitution Grammars for Syntactic Parsing 92.4 on PTB. Was SOTA until Dyer 2016 surpassed it.
• Chahuneau et al 2013 - Knowledge-Rich Morphological Priors for Bayesian Language Models Combines a finite-state guesser with Bayesian non-parametrics

• Lee et al 2020 - A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
  
• Nalisnick et al 2020 - Approximate Inference for Deep Latent Gaussian Mixtures

• David Blei
• Sharon Goldwater
• Mark Johnson
• Michael Jordan
• Yee Whye Teh

• Graphical Models Bayesian methods often use techniques from graphical models, such as MCMC and variational inference, as well as representing likelihood and prior as a graphical model