Sick

I’ve been sick for about a week with the sniffles and a mild fever. It’s been a bit of a rough patch, but I’m starting to feel better now.

I’ve been resting a lot and drinking plenty of fluids. Hopefully, I’ll be back to my usual self soon!

I’ve watched many you tube videos on mathematical themes while resting.

The algorithm knows I’m interested in bayesian nonparametric statistics and it sometimes recommends videos on that topic most are not very useful but occasionally there is a gem. What constitutes a gem is of course subjective but let me try to spell it out.

First a bit about non parametric statistics which is probably not known to many readers though not really relevant to the main point of this post.

Non parametric statistics is a branch of statistics that does not assume a specific parametric form for the underlying population distribution. Instead, it relies on fewer assumptions and is often used when the data does not meet the requirements of parametric tests. Nonparametrics can models usually come with a parameter that controls the proclivity of the model to conform to the underlying distribution it has learned from the data.

Many of the videos are too technical about the math of a specific non parametric model or method - there are some effective expositions of the math but once you understand the math you care less about this aspect of BNP. The math is not just hard there are almost endless variations on the math depending on the specific model or method being discussed with researchers often engaged in tweaking the math to publish new papers rather than providing deep and useful new insights.

I think that the bayesian world there are three perhaps four key areas that are important to understand and improve deep learning and AI.

These are :

1. Hierarchical models (can capture multi scale structure and hierarchies in data),
2. Mixture models (are more flexible than single distribution models and can be combined with (1) to form mixtures of experts that arise naturally in machine learning.
3. Bayesian Nonparametrics - replace the well known distributions like the normal, gamma, beta with more flexible infinite dimensional generalizations like the Dirichlet process, Gaussian process, Indian buffet process, pitman yor process and so on.
4. Normalizing flows - these allow us to create complex distributions by transforming simple ones through a series of invertible functions. These are very useful in deep learning for tasks like density estimation and generative modeling.
5. Bayesian conjugate models based on the Metalog distributions. - This is the idea that if you use a Metalog distribution as your prior and likelihood you can get closed form posteriors that are easy to compute and work with and remain in the Metalog family. This is a combination of mixtures and nonparametrics. But more significantly a general purpose closed form conjugacy seems to be very useful tool for cases where the math is otherwise intractable and requires simulation based methods. Infact this might be tested in bayesian RL applications

Each of these brings something important to the table in the sense but ideally one would like a framework that combines all these aspects but this is a discussion for another day.

So what is interesting is being told about how to apply BNP models in practice, how to think and design in this paradigm (more so as it most of the math is fairly new and still evolving.) So often if you ask a question about a BNP the researcher would ask you what is your impressions as there is little in the literature about most practical applications. (How to infer the hyperparameters, how to design the model structure, how to do model checking and validation etc.) And yet is seems that theses models should let us specify more flexible models that represent precisely what we want and then we can use a deep neural net to approximate it. There are not many people with insights into this area.