Week 0: Feyman Notebook on Bayesian Time Series Analysis

Time Series Analysis

Notes on Bayesian Time Series Analysis
Coursera
notes
Bayesian Statistics
Time Series
Faynman Notebook
Author

Oren Bochman

Published

Tuesday, October 22, 2024

Keywords

time series

A Feynman Notebook - For Bayesian Time Series Analysis

TS Questions — A Feynman Notebook

Here is where I can collect questions about TS analysis that I have as I go through the course. Hopefully I will be better equipped to answer many of these them by the end of the course.

  1. How does Bayesian TS analysis differ from regular TS analysis?
  2. Fourier analysis is a powerful tool for time series analysis. How does it relate to Bayesian time series analysis?
  3. How and what type of prior knowledge into time series analysis?
  4. Are there models that are unique to Bayesian time series analysis?
  5. How does distributional thinking affect time series analysis?
  6. How do we represent uncertainty in Bayesian time series analysis?
  7. Will we learn about Gaussian Processes in this course?
  8. What BTS models are most useful in economics and finance?
  9. Is there a cleanup procedure for time series data?
  10. Is there an Bayesian EDA for time series?
    • we can use differencing to make the time series stationary
    • we can use the ACF and PACF
    • we can decompose the time series into trend, seasonal, and residual components
    • we can visualize autocorrelation using a correlogram
    • we can visualize periodicity using a periodogram and spectral density.
    • see (Nielsen 2019)
  11. How do we handle missing data in time series?
  12. How do we handle non-stationary time series?
    • By applying differencing we can make the time series stationary.
  13. Are there processes for long term memory in time series?
  14. Are there processes for power laws.
  • see https://wiki.santafe.edu/images/5/52/Powerlaws.pdf
  1. Can BTS handle dynamic systems in time series?
  2. Can we model time series with multiple scales?
  3. What TS models are useful for regime switching?
    • I recall this came up in Davidson-Pilon (2015)
    • This is also covered in module 3 of the course.
  4. How can we simulate time series data?
  5. How can we forecast time series data?
  6. How can we find periodicity in time series data?
  7. Is the Kalman Filter a state-space or dynamic linear model
  8. Particle filters AKA Sequential Monte Carlo methods in the book.
  9. Are there bayesian time series models of contagion?
  10. Are there bayesian time series models of epidemics?
  11. What are seasonal adjustment?
  12. How to do seasonal adjustment?
  13. What are the tools for wrangling and cleaning TS data.
  14. How to use the frequency domain as a key part of understanding and analyzing data?
  15. How to assess whether an existing trend should be predicted to continue or whether caution should be used in forecasts?
  16. Do you really understand AR models or are they just a black box?
  17. What is a unit root?
  18. How to use tools that would help you decide whether to use a seasonal model and/or include a unit root in models?
  19. How to use predictor variables to help forecast some variable such as sales? (i.e. regression on time series data)

Cointegration

  1. What are Cointegrated time series?
    • Cointegration is a technique used to find a long term correlation between time series processes that was introduced in 1987 by Nobel laureates Robert Engle and Clive Granger
    • also see (Pfaff 2008)
  2. What are some test for cointegration:
    • Engle-Granger,
    • Johansen Test,
    • the Phillips-Ouliaris test.

NN and Deep learning

  1. How to use neural network methods to forecast time series?
    • RNNs
    • lstms
    • convolutions
    • GRUs
    • transformers
    • TS foundations models
    • Neural Prophet citation needed
  2. Deep learning foundation models citation needed pre-train NN model with many time series. Is this a form of Bayesian time series analysis?
  3. How does this BTS relate to deep learning?
    • Diffusion models in DL are autoregressive citation needed
    • the recently the mamba architecture has been proposed which is an autoregressive state space model. citation needed

Web scraping

  1. Any tips on scaraping time series data?

Filtering & Smoothing

  1. What is smoothing in BTS
    • decomposing the series as a sum of two components: a smooth component, plus another component that includes all the features that are unexplained by the smooth component.
    • one way is to use a moving average.
    • in the bayesian context smoothing is the process of estimating the hidden states of a system given the observed data.
  2. What is filtering in Bayesian Time Series?

    • in the bayesian context filtering is the process of estimating the hidden states of a system given the observed data.

    • we want to sample p(/theta_t,k|\mathcal{D}_t)

  3. What is the Kalman filter?
  4. What is a particle filter?
  5. what is the Butterworth filter?
    • The Butterworth filter is a signal processing filter that has a flat response in passband (or as close as possible to flat)

Models:

  1. White noise
  2. Wiener process (random walk)
  3. AR(1): Autoregressive process order 1
  4. Ornstein–Uhlenbeck process a continuous-time version of the AR(1) process
  5. AR(p): Autoregressive process order p
  6. MA(q): Moving average process order q
  7. ARMA(p,q): Autoregressive moving average
  8. SARMA: Seasonal ARMA
  9. ARIMA: Autoregressive integrated moving average
  10. SARIMA: Seasonal ARIMA
  11. VAR: Vector autoregressive
  12. SVAR: structural vector autoregressive models (SVAR).
  13. VECM: Vector error correction models (VECM).
  14. GARCH: Generalized autoregressive conditional heteroskedasticity
  15. ARCH: Autoregressive conditional heteroskedasticity
  16. SMC: Sequential Monte Carlo
  17. MDM: Multiregression dynamic models
  18. LTMs: latent threshold models
  19. FFBS: Forward Filtering Backward Sampling
  20. DLM: Dynamic Linear Models
  21. BSTS: Bayesian Structural Time Series
    • https://drive.google.com/file/d/14US56VzanuLt03XBkoAGzLy0gDEreZUc/view
  22. TVAR: Time-varying autoregressive models
  23. DGLM: Dynamic Generalized Linear Models
magic trick

What is the relation between Dynamic linear time series models (DLTS) models and Bayesian structural time series models (BSTS) models?

Dynamic Linear Time Series (DLTS) models and Bayesian Structural Time Series (BSTS) models are both frameworks for modeling time series data, and they share a strong connection, particularly in the way they approach model formulation and uncertainty. Here’s a breakdown of their relationship:

  1. Dynamic Linear Time Series (DLTS) Models:
  • DLTS models, often referred to as Dynamic Linear Models (DLMs), are a class of models where the parameters (such as the intercept or slope) evolve over time according to a stochastic process. They can be written in a state-space form, consisting of:
  • Observation Equation: Relates the observed data to the hidden state.
  • State Equation: Describes how the hidden state evolves over time.
  • These models use Kalman filtering for inference and prediction.
  • DLTS models are flexible in handling non-stationarity and time-varying parameters.

In the course we primarily learned to use Bayesian methods to estimate the parameters of the model.

  1. Bayesian Structural Time Series (BSTS) Models:
  • BSTS models are a Bayesian approach to time series modeling, which generalizes the DLTS framework.
  • Like DLTS, they use a state-space form, where the time series is decomposed into different components (e.g., trend, seasonality, regression effects).
  • BSTS explicitly incorporates Bayesian inference, where prior distributions are placed on the model components and parameters, and inference is conducted using MCMC or other Bayesian methods.
  • One of the key advantages of BSTS is its ability to incorporate model uncertainty, allowing the user to specify structural components (such as trend or seasonality) with uncertainty about their presence or importance in the data.
  1. Relation Between DLTS and BSTS:

  • Bayesian Extension of DLTS: BSTS can be seen as a Bayesian extension of DLTS models. While DLTS uses Kalman filtering for deterministic inference, BSTS uses Bayesian methods to quantify and propagate uncertainty in model components and parameters.

  • Component Decomposition: Both models can represent the time series in terms of structural components (like trends, seasonal patterns, or covariates), but BSTS allows for more flexible modeling of these components using Bayesian priors and hierarchical structures.

  • Handling of Uncertainty: DLTS models provide point estimates for parameters using Kalman filters, while BSTS incorporates full probabilistic estimates, enabling better uncertainty quantification in the presence of small data, model misspecification, or structural breaks.

  • Model Complexity: BSTS models can handle more complex scenarios where the structure of the time series isn’t fully known (e.g., unknown seasonality or trends), whereas DLTS is typically used when the structure of the model (e.g., presence of trend or seasonality) is more defined.

  • for more information on BSTS

Software

  1. What are the most useful packages in R for time series analysis?
    • BOA Bayesian Output Analysis (BOA, Smith 2007) citation needed and
    • CODA Convergence Diagnosis and Output Analysis for MCMC (CODA, Plummer, Best, Cowles, and Vines 2006). citation needed
    • URCA Unit Root and Cointegration Tests for Time Series Data (URCA, Pfaff 2008)citation needed.
    • Vars VAR Modelling (Vars, Pfaff 2008). citation needed
    • BSTS Bayesian Structural Time Series (BSTS, Scott and Varian 2014). citation needed
    • CausalImpact Causal Impact Analysis (CausalImpact, Brodersen, Gallusser, Koehler, Remy, and Scott 2015). citation needed builds on BSTS and methods from this Inferring causal impact using Bayesian structural time-series models
    • KFAS Kalman Filter and Smoother for Exponential Family State Space Models (KFAS, Helske 2017). citation needed
    • MARSS Multivariate Autoregressive State-Space Models (MARSS, Holmes, Ward, and Scheuerell 2012). citation needed
    • MCMCpack Markov Chain Monte Carlo (MCMCpack, Martin, Quinn, and Park 2011). citation needed
    • MCMCglmm Markov Chain Monte Carlo Generalized Linear Mixed Models (MCMCglmm, Hadfield 2010). citation needed
    • R-INLA Integrated Nested Laplace Approximations (R-INLA, Rue, Martino, and Chopin 2009). citation needed used approximate Bayesian inference for Latent Gaussian Models that can be expressed as latent Gaussian Markov random fields (GMRF)
  2. What about in python?

References

Davidson-Pilon, C. 2015. Bayesian Methods for Hackers: Probabilistic Programming and Bayesian Inference. Addison-Wesley Data & Analytics Series. Pearson Education. https://books.google.co.il/books?id=rMKiCgAAQBAJ.
Nielsen, A. 2019. Practical Time Series Analysis: Prediction with Statistics and Machine Learning. O’Reilly Media. https://books.google.co.il/books?id=xNOwDwAAQBAJ.
Pfaff, B. 2008. Analysis of Integrated and Cointegrated Time Series with r. Use r! Springer New York. https://books.google.co.il/books?id=ca5MkRbF3fYC.
Prado, R., M. A. R. Ferreira, and M. West. 2023. Time Series: Modeling, Computation, and Inference. Chapman & Hall/CRC Texts in Statistical Science. CRC Press. https://books.google.co.il/books?id=pZ6lzgEACAAJ.
Woodward, W. A., B. P. Sadler, and S. Robertson. 2022. Time Series for Data Science: Analysis and Forecasting. Chapman & Hall/CRC Texts in Statistical Science. CRC Press. https://books.google.co.il/books?id=_W16EAAAQBAJ.

Reuse

CC SA BY-NC-ND

Citation

BibTeX citation:
@online{bochman2024,
  author = {Bochman, Oren},
  title = {Week 0: {Feyman} {Notebook} on {Bayesian} {Time} {Series}
    {Analysis}},
  date = {2024-10-22},
  url = {https://orenbochman.github.io/notes/bayesian-ts/module6.html},
  langid = {en}
}
For attribution, please cite this work as:
Bochman, Oren. 2024. “Week 0: Feyman Notebook on Bayesian Time Series Analysis.” October 22, 2024. https://orenbochman.github.io/notes/bayesian-ts/module6.html.