Bayesian Optimization

Books:

1. (Garnett 2023) Bayesian Optimization
2. (Nguyen 2024) Bayesian Optimization in Action

Libraries

• GPyTorch - A Gaussian Process library implemented in PyTorch.
• BoTorch - A library for Bayesian optimization built on top of GPyTorch and PyTorch.
• scikit-optimize - A simple and efficient library for Bayesian optimization in Python.

Overview

• In this course we learn to:
  • Demonstrate a wide range of skills and knowledge in Bayesian statistics.
  • Explain essential concepts in Bayesian statistics.
  • Learn how to optimize hyperparameter tuning,
  • A/B testing
• We will build the following skills:
  • Gaussian Processes
  • Modeling with Gaussian Processes
  • Model Assessment, Selection, and Averaging
  • Decision Theory for Optimization
  • Utility Functions for Optimization
  • Common Bayesian Optimization Policies
  • Computing Policies with Gaussian Processes
  • Implementation
  • Theoretical Analysis
  • Extensions and Related Settings
  • A Brief History of Bayesian Optimization
• There are five modules in this course:
  1. Bayesian conjugate analysis for the autoregressive Time Series Models: We will focus on AR(p) models that fit in the Normal conjugate family, which means that the prior and likelihood lead to a posterior that is also in the Normal family.
  2. Model Selection Criteria: We will review AIC, BIC and DIC theory and Code for use in selecting the order of the AR(p) model as well as the number of components in the mixture model which we will develop in the next module.
  3. Bayesian location mixture of AR(p) model: This module covers a new kind of model, in which we extend the AR(p) model from using a single location to use a location mixture. These lead to conditionaly conjugate components, keeping everything more or less normal but with the many benefits arising from using a mixture model.
  4. The peer reviewed data analysis project: We will develop the model and then evaluate other students’ models.

I also explored his personal website and found that is contains some very interesting material.

1. He has written a book called Cracking The First Year Exam This book is a collection of questions from 5 first year statistics courses. I noticed that some of the instructors from the specialization feature in this book.

2. He has made extensive notes for a course on Stochastic Processes. This is one of the of the frist year MA courses I haven’t had the oppertunity to take and I will go over these notes as soon as I complete this course. I wanted to explore the connections between stochastic processes and Bayesian statistics, as I believe they can complement each other well. I also found that he has taught a short course on Bayesian Causal Inference which is based primarily on research papers. And it seems

3. In Classical Inference he covers material from a frequentist type statistical inference course which follows Casella and Berger (2002). This material is covered to some extent in the first course of the specialization but not in any great detail. So it may be useful to review this material as well. In fact some of it fills gaps in the material I felt missing and added as appendices like a proof of the central limit theorem and number of results related the law of large numbers. I wish I could say that this is easy reading after taking this specialization but it should look fairly familiar. A few other topics that I find of interest are chapter 9 - on the method of moments and

4. In Bayesian Causal Inference - slides and Bayesian Causal Inference - report he reviews the material from papers on Bayesian Causal Inference.

Overall I think he was on one of the best explainers of the complex material. One downside is that is english is a little broken but I’ve had much worse. However you can see from the first slide that it covers the key concepts in a clear and concise manner leaving out very little.

Also it seems to me that Jizhou Kang still remembers how hard this material is at first glance which is a great asset for teaching it effectively. You may notice that we revisit some old material this is because this course makes a great effort to be self contained.

Many parts of this course are steps towards completing the capstone project. And perhaps just as exciting we will also cover a new type of model - a mixture version of the autoregressive model from the previous course.

We will cover the Bayesian conjugate analysis for autoregressive time series models.

This sounds a bit bombastic, all it means is that we will be using a likelihood and a prior that leads to a conjugate posterior, keeping everything within one distributional family!

Prerequisite skill checklist 🗒️

NotePrerequisite skill checklist

Bayesian Statistics

• Interpret and specify the components of Bayesian statistical models: likelihood, prior, posterior.
• Explain the basics of sampling algorithms, including sampling from standard distributions and using MCMC to sample from non-standard posterior distributions.
• Assess the performance of a statistical model and compare competing models using posterior samples.
• Coding in R to achieve the above tasks.

Mixture Models

• Define mixture model.
• Explain the likelihood function associated with a random sample from a mixture distribution.
• Derive Markov chain Monte Carlo algorithms for fitting mixture models.
• Coding in R to achieve the above tasks.

Time Series Analysis

• Define time series and stochastic processes (univariate, discrete-time, equally-spaced)
• Define strong and weak stationarity
• Define the auto-covariance function, the auto-correlation function (ACF), and the partial autocorrelation function (PACF)
• Definition of the general class of autoregressive processes of order p.

Casella, G., and R. L. Berger. 2002. Statistical Inference. Duxbury Advanced Series in Statistics and Decision Sciences. Thomson Learning. http://home.ustc.edu.cn/~zt001062/MaStmaterials/George%20Casella&Roger%20L.Berger--Statistical%20Inference.pdf.

Garnett, Roman. 2023. Bayesian Optimization. Cambridge University Press.

Nguyen, Q. 2024. Bayesian Optimization in Action. In Action. Manning. https://books.google.co.il/books?id=RIfZEAAAQBAJ.