Appendix J — Bayesian Books in R & Python

Author

Oren Bochman

A number of books on Bayesian statistics and time series analysis are available in both R and Python. A number of these are introduced in the specilization and many others are worth mentioning.

There are many books in R and Python that can help you learn more about these languages and how to use them for data analysis.

Here are some of the most popular books on R and Python:

J.1 Introduction to Probability

J.2 Books in R

J.3 Books in Python

J.4 Bayesian Statistics: Techniques and Models

At the end of the course there is a handout called further reading. In this course the following titles are recommended for further reading and reference, unfortunately some of the links were out of date.

Figure J.1: Doing Bayesian Data Analysis

(Kruschke 2011) Doing Bayesian Data Analysis this is the favorite on Bayesian statistics. It is a great book for learning Bayesian statistics and covers a wide range of topics in Bayesian modeling, computation, and inference. The book is suitable for graduate students and researchers in statistics, computer science, and related fields. This text is recommended for further reading in the course 2

Figure J.2: Statistical Rethinking

(McElreath 2015) Statistical Rethinking: A Bayesian Course with Examples in R and Stan by Richard McElreath is, the runner up in terms of a popular book on Bayesian Statistics. This book is a great resource for learning Bayesian statistics and covers a wide range of topics in Bayesian modeling, computation, and inference. The book is suitable for graduate students and researchers in statistics, computer science, and related fields. McElreath is a gifted and entertaining explainer but he is not a full-fledged statistician. I find that in most cases the examples in this book are rather weak and challenging to adapt to real life settings. On the other hand his use of metaphors and his ability to pass on his intuition is unparalleled and has helped to shape my own thinking about statistics as is evidenced by the many references to his work in these notes. This text is recommended for further reading in the course 2

Figure J.3: Bayesian Data Analysis

(Gelman et al. 2013) Bayesian Data Analysis by Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, Donald B. Rubin. This book is considered the Graduate level text on Bayesian statistics and covers a wide range of topics in Bayesian data analysis. The book is suitable for graduate students and researchers in statistics, computer science, and related fields. This book is harder to read than the previous two books. I’d recommend reading and viewing McElreath’s book and videos first and only then moving to this book later. This text is recommended for further reading in the course 2 but with a cautionary caveat that a solid background in calculus-based probability will be useful, as many details are left to the reader. I took an undergraduate course in probability and statistics that was based on measure theory but I’m not sure this was the kind of background these authors have in mind.

Figure J.4: The BUGS book

(Lunn et al. 2012) The BUGS book: A Practical Introduction to Bayesian Analysis is a classic text on Bayesian statistics and covers a wide range of topics in Bayesian modeling, computation, and inference. IT focuses on simulation using BUGS. This text is recommended for further reading in the course 2

Some addition books that were mentioned in the course bibliography:

(Carlin and Louis 2008) Bayesian Methods for Data Analysis

(Agresti 2012) This is a graduate level book on categorical data which often poses extra challenges for Bayesian statistics. § 17.5 covers Bayesian methods for Categorical Data.

(Spiegelhalter et al. 2002) “Measures of Model Complexity and Fit” by David Spiegelhalter, Andrew Thomas, Nicky Best, and W. Brian Gilks. This paper considers the problem of model selection in Bayesian Hierarchical Models and proposes a new measure of model complexity and fit, the Deviance Information Criterion (DIC).

(Banerjee, Gelfand, and Carlin 2026) Hierarchical Modeling and Analysis for Spatial Data by Sudipto Banerjee, Bradley P. Carlin, and Alan E. Gelfand. This book is a comprehensive introduction to hierarchical modeling and analysis for spatial data and covers a wide range of topics in spatial statistics, computation, and inference. The book is suitable for graduate students and researchers in statistics, computer science, and related fields. data etc

(Carlin and Louis 2008) Bayesian Methods for Data Analysis

(Gelman and Hill 2006) This is too is a classic book on data analysis using regression and multilevel/hierarchical models. I read this one may years ago and is fairly approachable and recommended Unless the next edition has come out?

J.5 Mixture Models

Figure J.5: Finite Mixture and Markov Switching Models

There were no books recommended in this course but (Frühwirth-Schnatter 2006) was in the bibliography from the previous course. Finite Mixture and Markov Switching Models by Sylvia Frühwirth-Schnatter. This book is a comprehensive introduction to finite mixture and Markov switching models and covers a wide range of topics in mixture modeling, computation, and inference. The book is suitable for graduate students and researchers in statistics, computer science, and related fields.

(Fruhwirth-Schnatter, Celeux, and Robert 2019) Handbook of Mixture Analysis by Christian P. Robert, Gilles Celeux, Sylvia Fruhwirth-Schnatter

(McLachlan and Peel 2004) Finite Mixture Models by Geoffrey J. McLachlan and David Peel. A comprehensive account of major issues in finite mixture modeling. Advanced text on finite mixture models. Lacks exercises and requires deep diving into papers.

(Chen 2023) Statistical Inference Under Mixture Models by Jiahua Chen. A more recent introduction to mixture models with recent developments in testing hypothesis for the order of the mixture and insights on inference for mixture models.

(Yao and Xiang 2024) Mixture Models: Parametric, Semiparametric, and New Directions by Weixin Yao and Sijia Xiang.

(Visser and Speekenbrink 2022) Mixture and Hidden Markov Models with R by Visser, I. and Speekenbrink, M.

J.6 Non-parametric Bayesian Statistics

(Bayesian Nonparametrics 2010) “Bayesian Nonparametrics” by Peter M. Hjort, Chris Holmes, Maria E. Müller, and Stephen G. Walker. This book is a comprehensive introduction to Bayesian nonparametric methods and covers a wide range of topics in nonparametric modeling, computation, and inference. The book is suitable for graduate students and researchers in statistics, computer science, and related fields.

J.7 Bayesian Time Series Bibliography

We start with some books from the course, I collected here both the recommended books and some others that I found useful.

Figure J.6: Bayesian Forecasting and Dynamic Models

(West and Harrison 2013) Bayesian Forecasting and Dynamic Models by Mike West and Jeff Harrison. This book is a classic text on Bayesian statistics and covers a wide range of topics in Bayesian forecasting and dynamic models. The book is suitable for graduate students and researchers in statistics, computer science, and related fields. This is a much easier read than the following book and is a great introduction to Bayesian time series analysis. It soon became my goto reference for understanding the more challenging aspects of the course. It provides simple explanations with better motivations.

I have some criticism. The authors sometimes become very meta discussing their models at very abstract level that is more relevant to philosophy majors than statistics. For example a dynamic model is a defined as a set of models whose parameters may change at any index of time series. That’s a very poor definition - its like asking us to use a 1:1 map for navigation. When I took a course on function approximation in RL we got a much neater exposition to parameterization of to very similar markov state space models. I found that the white noise model getting a very extensive treatment, perhaps, because it is easier to analyze. This is a random walk and the rudiments of which are easily taught in a single high-school lesson. They also digress into statistical war stories type anecdotes that rather than instil confidence that the authors know what they are talking about they are effectively making a case that they failed to communicate effectively with their clients with disastrous effect. Mathematically the book is very much self contained and this shows great integrity. I’ve seen some talks by Mike West in which he highlights research by his students and this are great if very challenging to follow.

The authors also published a second book [Applied Bayesian Forecasting and Time Series Analysis]](https://www2.stat.duke.edu/~mw/bats.html) this one is based on the BATS software package which is no longer maintained. However it does contain a large number of datasets

Figure J.7: Time Series: Modeling, Computation, and Inference

(Prado, Ferreira, and West 2023) Time Series: Modeling, Computation, and Inference by course instructor Raquel Prado and Marco A. R. Ferreira, Mike West. This book, now in its second edition is a comprehensive introduction to time series analysis and covers a wide range of topics in time series modeling, computation, and inference. The book is suitable for graduate students and researchers in statistics, computer science, and related fields.

While learning this course I found some of the material harder to follow than I expected. The book helped to clarify definitions and so on however the book is rather comprehensive and mathematically advanced unlike some other books on statistics.

The teacher frequently point out that many aspects of Times series and are beyond the scope of the course. Yet this book covers much more ground like unequally spaced time series and vector valued time series.

For example we look at EKG data which the authors have been working on for years. However we look at it in this course in terms of a univariate time series while in reality EKG is usually sampled at 12 sites simultaneously yielding a multi-variate time series.

Once this course is done I will probably want to dive deeper into the subject and try to devote more time to other models in the book.

c.f. (Nielsen 2019)

Figure J.8: Practical Times Series Analysis

(Nielsen 2019) Practical Time Series Analysis: Prediction with Statistics and Machine Learning by Aileen Nielsen. Is a good resource for practitioners getting started with time series analysis. I also recommend any videos by Aileen Nielsen on the subject. In many ways this is a recommended introductory book on time series analysis. It covers, finding and wrangle time series data. EDA for TS. How to store temporal data. Simulate time series data. Generate and select features for a time series. Estimate errors, evaluate accuracy and performance. Forecast and classify time series with machine or deep learning.

Figure J.9: Statistical Analysis in Climate Research

(Storch and Zwiers 2002) Statistical Analysis in Climate Research I came across this book while looking into the Durban-Levinson recursion and the Yule-Walker equations. So far I haven’t had time to read it but it looks promising. Here is the description from the publisher:

Climatology is, to a large degree, the study of the statistics of our climate. The powerful tools of mathematical statistics therefore find wide application in climatological research. The purpose of this book is to help the climatologist understand the basic precepts of the statistician’s art and to provide some of the background needed to apply statistical methodology correctly and usefully. The book is self contained: introductory material, standard advanced techniques, and the specialized techniques used specifically by climatologists are all contained within this one source. There are a wealth of real-world examples drawn from the climate literature to demonstrate the need, power and pitfalls of statistical analysis in climate research. Suitable for graduate courses on statistics for climatic, atmospheric and oceanic science, this book will also be valuable as a reference source for researchers in climatology, meteorology, atmospheric science, and oceanography.

Hans von Storch is Director of the Institute of Hydrophysics of the GKSS Research Centre in Geesthacht, Germany and a Professor at the Meteorological Institute of the University of Hamburg.

Francis W. Zwiers is Chief of the Canadian Centre for Climate Modelling and Analysis, Atmospheric Environment Service, Victoria, Canada, and an Adjunct Professor at the Department of Mathematics and Statistics of the University of Victoria.

Figure J.10: Bayesian Modeling and Computation in Python

(Martin, Kumar, and Lao 2021) Bayesian Modeling and Computation in Python by Osvaldo Martin is a great book for learning Bayesian statistics and covers a wide range of topics in Bayesian modeling, computation, and inference. I found the chapter on state space modeling and the Kalman filter particularly useful. The book is a great resource for translating what we learned in the course to Python. The book is suitable for undergraduate students in statistics, computer science, and related fields.

Figure J.11: Introductory Time Series with R

(Cowpertwait and Metcalfe 2009) Introductory Time Series with R by Cowpertwait and Metcalfe

Yearly global mean temperature and ocean levels, daily share prices, and the signals transmitted back to Earth by the Voyager space craft are all examples of sequential observations over time known as time series. This book gives you a step-by-step introduction to analyzing time series using the open source software R. Each time series model is motivated with practical applications, and is defined in mathematical notation. Once the model has been introduced it is used to generate synthetic data, using R code, and these generated data are then used to estimate its parameters. This sequence enhances understanding of both the time series model and the R function used to fit the model to data. Finally, the model is used to analyze observed data taken from a practical application. By using R, the whole procedure can be reproduced by the reader.

All the data sets used in the book are available on the website at datasets

The book is written for undergraduate students of mathematics, economics, business and finance, geography, engineering and related disciplines, and postgraduate students who may need to analyze time series as part of their taught programme or their research.

Paul Cowpertwait is an associate professor in mathematical sciences (analytics) at Auckland University of Technology with a substantial research record in both the theory and applications of time series and stochastic models.

Andrew Metcalfe is an associate professor in the School of Mathematical Sciences at the University of Adelaide, and an author of six statistics text books and numerous research papers. Both authors have extensive experience of teaching time series to students at all levels.

Figure J.12: Analysis of Integrated and Cointegrated Time Series with R

(Pfaff 2008) “Analysis of Integrated and Cointegrated Time Series with R” by Bernhard Pfaff. Its been a long time since I read this book and rather than do it an injustice I direct you to the review by Dirk Eddelbuettel in the Journal of Statistical Software is available at review. Or the book’s website at Analysis of Integrated and Cointegrated Time Series with R.

The analysis of integrated and co-integrated time series can be considered as the main methodology employed in applied econometrics. This book not only introduces the reader to this topic but enables him to conduct the various unit root tests and co-integration methods on his own by utilizing the free statistical programming environment R. The book encompasses seasonal unit roots, fractional integration, coping with structural breaks, and multivariate time series models. The book is enriched by numerous programming examples to artificial and real data so that it is ideally suited as an accompanying text book to computer lab classes.

The second edition adds a discussion of vector auto-regressive, structural vector auto-regressive, and structural vector error-correction models.

(Broemeling 2019) Bayesian Analysis of Time Series by Lyle D. Broemeling

In many branches of science relevant observations are taken sequentially over time. Bayesian Analysis of Time Series discusses how to use models that explain the probabilistic characteristics of these time series and then utilizes the Bayesian approach to make inferences about their parameters. This is done by taking the prior information and via Bayes theorem implementing Bayesian inferences of estimation, testing hypotheses, and prediction. The methods are demonstrated using both R and WinBUGS. The R package is primarily used to generate observations from a given time series model, while the WinBUGS packages allows one to perform a posterior analysis that provides a way to determine the characteristic of the posterior distribution of the unknown parameters.

The book covers pretty much the material in the course. It uses R and WinBUGS to demonstrate the models and methods. Models considered include: white noise, Wiener process (random walk), AR(p),ARMA(p,q), ARIMA, Regression, Regression with MA and Seasonal effects, DLM , TAR

Figure J.13: Bayesian Inference for Stochastic Processes by Lyle D. Broemeling

(Broemeling 2019) Bayesian Inference for Stochastic Processes by Lyle D. Broemeling is a comprehensive introduction to the analysis of stochastic processes using Bayesian methods. The book covers a wide range of topics in stochastic process modeling, computation, and inference. The book is suitable for graduate students and researchers in statistics, computer science, and related fields. The code for R and WinBUGS is available at code It is based on WinBUGS which is a bit dated but still useful and a bit dated but it covers a lot of the material in the course.

Figure J.14: Dynamic Time Series Models using R-INLA: An Applied Perspective

(Ravishanker, Raman, and Soyer 2022) Dynamic Time Series Models using R-INLA: An Applied Perspective is a new book that covers the use of the R-INLA package for fitting dynamic time series models. The book is available online gitbook

This is a very interesting book which covers a new approach to fitting time series models using the R-INLA package. INLA stands for Integrated Nested Laplace Approximation and is a method for fitting Bayesian models that is faster than MCMC. The book covers a wide range of topics in time series modeling, computation, and inference. The book is suitable for graduate students and researchers in statistics, computer science, and related fields.

Figure J.15: Statistics for Spatio-Temporal Data

(Cressie and Wikle 2011) Statistics for Spatio-Temporal Data is a book I came across when I tried to understand the NDLM model. NDLMs have a two level hierarchical form and it seems possible to extend this formulation will non-normally distributed shocks and possibly non linear relation. In this book the authors take an interesting approach of not only looking at NDLM as a hierarchical model but they also extend the time series model into a spatio-temporal model.

This book is a comprehensive introduction to the analysis of spatio-temporal data and covers a wide range of topics in spatio-temporal statistics. The book is suitable for graduate students and researchers in statistics, computer science, and related fields.

a newer title from the authors is:

Figure J.16: Spatio-Temporal Statistics with R

(Wikle, Zammit-Mangion, and Cressie 2019) Spatio-Temporal Statistics with R

Figure J.17: Bayesian Analysis of Stochastic Process Models

(Rios Insua, Ruggeri, and Wiper 2012) Bayesian Analysis of Stochastic Process Models by David Rios Insua, Fabrizio Ruggeri, Michael P. Wiper. This book is a comprehensive introduction to the analysis of stochastic process models using Bayesian methods. It covers a wide range of topics in stochastic process modeling, computation, and inference. The book is suitable for graduate students and researchers in statistics, computer science, and related fields.

There are also a number of books on NDLM that I’ve come across:

Dynamic linear model tutorial matlab

Forecasting, structural time series and the Kalman filter by Andrew C. Harvey

Dynamic Linear Models with R by Giovanni Petris Sonia Petrone Patrizia Campagnoli

Time Series Analysis by State Space Methods by J. Durbin and S.J. Koopman

Figure J.18: Machine Learning: A Bayesian and Optimization Perspective

(Theodoridis 2015) Machine Learning: A Bayesian and Optimization Perspective Has two chapters on bayesian learning which are summarized in this summer school slide deck. I came across this book while looking into the Durban-Levinson recursion and the Yule-Walker equations. So far I haven’t had time to read it but it looks like a good book on machine learning and the slide deck indicates it covers some of the essential material missing from this specialization like particle filtering.