Bayesian Inference

In a frequentist setting, a statistician or econometrician typically commits to one specific parameter $\hat{\theta}$ after observing $\boldsymbol{y}$ and $\boldsymbol{X}$ and uses this parameter to dictate the probability of outcomes. Therefore, for an unobserved data $\tilde{y},\tilde{X}$ the likelihood is

However, in a dynamic real-world environment, it is often unrealistic to assume that the parameter remains fixed, given the continual changes and updates that influence underlying conditions.

Posterior Predictive

Instead of committing to one parameter to compute the likelihood of an unobserved data, the posterior predictve computes the weighted average of the likelihoods across all possible $\theta$, given its parameter $\alpha$.

Bayesian Linear Regression

Consider the following linear framework: $y = X\theta + U$

Note that $U\mid X \sim N(0, \sigma^2)$ implies $y\mid X \sim(X\theta, \sigma^2)$. Then, the posterior has the form

Gaussian Process

Gaussian Process is the distribution over functions. For a more general case that can be expanded to non-linear framework suppose $y = f(X) + U$. Then $f$ follows a Gaussian Process

Linear Basis Function

For example, $f$ can depend on the parameter $\theta$ where $\theta \sim N(0, \lambda^{-1}\boldsymbol{I})$. Consider the case $f(X) = \theta^\top \phi(X)$. Then, $m(X)$ is as follows

Kernel

The kernel $K$ determines the covariance over the functions, where $K(X, X') = cov(X, X')$. A common choice is the RBF kernel.

ref

Empirical Bayes

Instead of using cross validation to find the best hyperparmeter $\alpha$, which can potentially be biased towards the validation set, an alternative approach is to use Empirical Bayes.