Bayesian Estimation

Bayesian statistics provides formal methods for incorporating prior knowledge into the estimation of unknown parameters

In a non-Bayesian analysis, unknown parameters are treated as constants, while a Bayesian analysis treats them as random variables with a pdf

At the outset of an analysis the pdf assigned to a parameter is based on little information and is referred to as the prior distribution. After this pdf is updated with information, it is referred to as a posterior distribution

Definition: Let $W$ be a statistic dependent on a parameter $\theta$ with $f_W(w\mid \theta )$. Assume that $\theta$ is the value of a random variable $\Theta$ with prior distribution denoted $f_{\Theta }(\theta )$. The posterior distribution of $\Theta$, given that $W=w$, is

$$g_{\Theta }(\theta \mid W=w)=\frac{f_W(w\mid \theta )f_{\Theta }(\theta )}{{\int }_{-\infty }^{\infty }{f}_W(w\mid \theta )f_{\Theta }(\theta )d\theta }$$

The power of this is in its wide applicability

Now how do we use $g_{\Theta }(\theta \mid W=w)$ to actually calculate a point estimator $\hat{\theta }$? One technique is to find a maximum likelihood estimator, by differentiating the posterior distribution $\frac{d{g}_{\Theta }}{d\theta }=0$

However, the classic Bayesian approach is to minimize the risk associated with $\hat{\theta }$

Definition: Let $\hat{\theta }$ be an estimator for $\theta$ based on a statistic $W$. The loss function associated with $\hat{\theta }$ is denoted $L(\hat{\theta },\theta )$, where $L(\hat{\theta },\theta )\ge 0$ and $L(\theta ,\theta )=0$

Definition: The risk associated with $\hat{\theta }$ is the expected value of the loss function with respect to the posterior distribution of $\theta$,

$$\text{risk}={\int }_{\theta }^{}L(\hat{\theta },\theta )g_{\Theta }(\theta \mid W=w)d\theta$$

A Bayes estimate minimizes risk, which looks like differentiating the risk function

However, for L1 and L2 norm, there’s an easier method

Theorem: Let $g_{\Theta }(\theta \mid W=w)$ be the posterior distribution for the unknown parameter $\theta$

• If $L(\hat{\theta },\theta )=|\hat{\theta }-\theta |$ then the Bayes estimate is the median of $g_{\Theta }$
• If $L(\hat{\theta },\theta )=(\hat{\theta }-\theta {)}^2$ then the Bayes estimate is the mean of $g_{\Theta }$