Bayesian Non-Parametric Models

Here is a question I raised in my Feynman Notebook:

1. Will we learn about Gaussian Processes/Neural Networks in this course?
   • This is a type of Bayesian Non-parametric and we don’t cover these in the specialization. However Abel Rodriguez, the instructor of the third course on mixture model has a short course
   • Herbert Lee wrote a Bayesian Nonparametrics via Neural Networks on the subject.
   • Athanasios Kottas of UCSC has made the following notes available on his website:
     
   • Tutorial on Nonparametric Bayesian density regression: modeling methods and applications
   • Short course on Applied Bayesian Nonparametric Mixture Modeling with references (16 pages) with 52 of them being his own papers.

This is not available as a course on Coursera and isn’t a part of the specialization which ended in the last course. So this notes are my own personal notes gathered from tutorials and courses I found on the web. - Tamara Broderick’s Gaussian Processes for Regression tutorials from - 2025 slides video and code - 2024 slides - Tamara Broderick

Overview

• In this course we learn to:
  • Demonstrate a wide range of skills and knowledge in Bayesian statistics.
  • Explain essential concepts in Bayesian statistics.
  • Apply what you know to real-world data.
• We will build the following skills:
  • Probability Distribution (Dirichlet, Beta)
  • Bayesian Statistics
• There are currently five modules planned in this course:
  1. Gaussian Processes for Regression: We will focus on Gaussian processes as a flexible prior distribution for regression problems, allowing us to capture complex relationships in the data.
     1. Gaussian process model slides
     2. Gaussian process regression slides
     3. Squared exponential kernel and observation noise slides
     4. What uncertainty are we quantifying? slides
     5. A list of resources: slide
  2. Dirichlet process: We will explore the Dirichlet process as a prior distribution over probability measures, allowing for flexible modeling of unknown distributions.
  • The Beta distribution
  • The Dirichlet Distribution
  • Dirichlet process
  • Polya urn scheme
  • Stick breaking representation
  • Dirichlet process mixture models
  • Hierarchical Dirichlet processes
  3. Chinese restaurant process: We will introduce the Chinese restaurant process as a metaphor for the Dirichlet process, providing an intuitive understanding of how it works.
  4. Indian buffet process: We will discuss the Indian buffet process as a model for representing the distribution of features in a dataset, allowing for flexible and scalable modeling of complex data structures.
  5. Polya tree: We will explore the Polya tree as a nonparametric prior distribution for modeling probability distributions, allowing for flexible and adaptive modeling of complex data structures.

Prerequisite skill checklist 🗒️

NotePrerequisite skill checklist

Bayesian Statistics

• Interpret and specify the components of Bayesian statistical models: likelihood, prior, posterior.
• Explain the basics of sampling algorithms, including sampling from standard distributions and using MCMC to sample from non-standard posterior distributions.
• Assess the performance of a statistical model and compare competing models using posterior samples.
• Coding in R to achieve the above tasks.

Mixture Models

• Define mixture model.
• Explain the likelihood function associated with a random sample from a mixture distribution.
• Derive Markov chain Monte Carlo algorithms for fitting mixture models.
• Coding in R to achieve the above tasks.

Time Series Analysis

• Define time series and stochastic processes (univariate, discrete-time, equally-spaced)
• Define strong and weak stationarity
• Define the auto-covariance function, the auto-correlation function (ACF), and the partial autocorrelation function (PACF)
• Definition of the general class of autoregressive processes of order p.

Some References:

1. Gaussian Processes Rasmussen and Williams (2006)
2. Surrogates Gramacy (2020)
3. A Tutorial on Bayesian Optimization Frazier (2018)

• It builds a surrogate for the objective and quantifies the uncertainty in that surrogate using a Bayesian machine learning technique, Gaussian process regression, and then uses an acquisition function defined from this surrogate to decide where to sample
• three common acquisition functions:
  • expected improvement
  • entropy search
  • knowledge gradient

Frazier, Peter I. 2018. “A Tutorial on Bayesian Optimization.” https://arxiv.org/abs/1807.02811.

Gramacy, Robert B. 2020. Surrogates: Gaussian Process Modeling, Design, and Optimization for the Applied Sciences. Chapman & Hall/CRC the r Series. CRC Press. https://bookdown.org/rbg/surrogates/.

Rasmussen, Carl Edward, and Christopher K. I. Williams. 2006. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning Series. MIT Press. https://gaussianprocess.org/gpml/.