Parameter Estimation

Obtaining Estimators

Our goal is to optimize the parameters of a probabilistic model given an experiment

Definition: A function of a random sample that approximates a parameter is called a statistic or an estimator. An estimator of $\theta$ is denoted $\hat{\theta }$. The maximum likelihood estimate is denoted ${\theta }_e$.

Definition: The probability of a trial is called the likelihood function $L(\theta )={\prod }_{i=1}^n{f}_{\theta }(x_i)$ (where $f_{\theta }$ is a pmf or pdf)

Poisson: $f_{\lambda }(x)=\frac{e^{-\lambda }{\lambda }^x}{x!}$, so $L(\lambda )=\frac{e^{-\lambda n}{\lambda }^{{\Sigma }_i{x}_i}}{{\prod }_i(x_i)!}$
Exponential: $f_{\lambda }(x)=\lambda e^{-\lambda x}$, so $L(\lambda )={\lambda }^n{e}^{-\lambda {\Sigma }_i{x}_i}$
Uniform: $L(\theta )=\frac{1}{{\theta }^n}$ if $0\le x_i\le \theta$ or $0$ otherwise

Example: To estimate $\theta$ on the uniform distribution $[0,\theta ]$, we simply take $\max x_i$
It’s pretty intuitive how this maximizes $L(\theta )$

Bernoulli is perhaps the most intuitive, $L(p)=p^{{\sum }_i^{}{X}_i}(1-p{)}^{n-{\sum }_i^{}{X}_i}$
If we get heads in an experiment of $n$ trials, then $\hat{p}=\frac{k}{n}$

We can do this more generally this by taking the derivative of out formula (use the product rule) and finding a local maxima

$X=\sum _{i=0}^n{X}_i$, $L(p)=p^X(1-p{)}^{n-X}$
$\frac{dL}{dp}=X{p}^{X-1}(1-p{)}^{n-X}-p^X(n-X)(1-p{)}^{n-X-1}=0$
$X{p}^{X-1}(1-p{)}^{n-X}=p^X(n-X)(1-p{)}^{n-X-1}$
$X(1-p)=p(n-X)$
$p=\frac{X}{n}$

Ta-Da! Math tells us something we already know but in a more complicated way :)

We can use regular calculus techniques to verify that setting the derivative of a function to $0$ results in a global maximum

We can also do something neat by maximizing the log-likelihood $e(p)=\log L(p)$, which gives us the same result (since it’s monotonic) in an easier way. This is smart, because it allows us to rewrite our ${\prod }_{}^{}$ expression into a ${\sum }_{}^{}$ expression, and moves the exponents to coefficients.

$e(p)=X\log p+(n-X)\log (1-p)$
$e^{\prime }(p)=\frac{X}{p}-\frac{n-X}{1-p}$
$X(1-p)=p(n-X)$
$p=\frac{X}{n}$

This is almost always a good technique for analyzing $L(\dots )$
When we do this analysis on Poisson, we also get the sample mean $\hat{\lambda }=\mu$

Recall the example earlier on the uniform distribution. Since this function is not differentiable, we had to take an order statistic on the trial, $\max x_i$. A similar idea for the exponential distribution, where ${\theta }_e=y_{\text{min}}$.

To do this in general with a multivariate distribution, we set all partial derivatives to $0$

The Method of Moments

The method of moments is often more tractable than the method of maximum likelihood when the underlying model has multiple parameters

Suppose we have $s$ unknown parameters ${\theta }_1,\dots ,{\theta }_s$. The first $s$ theoretical moments of $Y$, if they exist, are $E(Y^j)={\int }_{-\infty }^{\infty }{y}^j\cdot f_Y(y;{\theta }_1,\dots ,{\theta }_s)dy$;. This gives us $s$ equations $E(Y^i)=g_i({\theta }_1,\dots ,{\theta }_s)$

We also have sample moments $\frac{1}{n}{\sum }_{i=1}^n{y}_i^j$

The sample moments can be used as approximations for the theoretical moments, giving us a system of $s$ equations

Interval Estimations

The issue with point estimations is they do not tell us the reliability of our estimations. Confidence intervals let us quantify uncertainty in an estimator

We’ve already looked at confidence intervals in the context of CLT
If we want to find an average to fall into an interval with probability $\alpha$, we can use CLT to get $\left(\bar{y}-z_{\alpha /2}\cdot \frac{\sigma }{\sqrt{n}},\bar{y}+z_{\alpha /2}\cdot \frac{\sigma }{\sqrt{n}}\right)$ (where $P(Z\ge z_{\alpha })=\alpha$)

The same idea can apply to other distributions, like large-sample binomial random variables

Theorem: Let $k$ be the number of successes in $n$ independent trials (where $n$ is large). An $100(1-\alpha )\%$ confidence interval for $p$ is

$$\left[\frac{k}{n}-z_{\alpha /2}\sqrt{\frac{(k/n)(1-k/n)}{n}},\frac{k}{n}+z_{\alpha /2}\sqrt{\frac{(k/n)(1-k/n)}{n}}\right]$$

This is effectively the same equation listed above, as the standard deviation of a binary random variable is $\sqrt{p(1-p)}$

Note that when estimating intervals from data, we’re more accurate saying the chance that the interval aligns with the true parameters, not vice versa. In other words, the true value of the parameters is a fixed fact, while our estimation comes from a random procedure.

In popular press, estimates include a margin of error, which is typically half the maximum width of a $95\%$ confidence interval. The margin of error associated with an estimate $\frac{k}{n}$ is $d=\frac{z_{\alpha /2}}{2\sqrt{n}}$

It’s important to note that the values within a margin of error are not equal: If a political survey over $1000$ people reveals Bob is winning $50\%$ to $48\%$ with a $3\%$ margin of error, we cannot say that the race is a tie. With the numbers the text uses, Bob has a $90\%$ chance of winning.

How do we choose $n$ for an experiment with the binomial distribution? We’d like to minimize $n$ such that $P(|X/n-p|\le d)=1-\alpha$

We transform the inequality to obtain $\frac{d}{\sqrt{p(1-p)/n}}=z_{\alpha /2}$, implying $n=\frac{z_{\alpha /2}^{2}p(1-p)}{d^2}$. Since $p(1-p)\le \frac{1}{4}$, we can use $n=\frac{z_{\alpha /2}^2}{4d^2}$ as an acceptable bound

Properties of Estimators

Maximum likelihood and the method of moments often yield different estimators. Which should we use? What’s a good estimator and is there a best one?

Biasedness

Definition: An estimator $\hat{\theta }=h(Y_1,\dots ,Y_n)$ is unbiased if $E(\hat{\theta })=\theta$ for all $\theta$

If ${\lim }_{n\to \infty }E(\hat{\theta })=\theta$ then we say $\hat{\theta }$ is asymptotically unbiased

We can sometimes construct an unbiased estimator directly from a biased estimator

Example: Is ${\hat{\sigma }}^2=\frac{1}{n}{\sum }_{i=1}^n(Y_i-\hat{Y}{)}^2$ unbiased (normal distribution)?
$E({\hat{\sigma }}^2)=E\left[\frac{1}{n}{\sum }_{i=1}^n(Y_i-\bar{Y}{)}^2\right]$
$=E\left[\frac{1}{n}{\sum }_{i=1}^n(Y_i^2-2Y_i\bar{Y}+{\bar{Y}}^2)\right]$
$=E\left[\frac{1}{n}{\sum }_{i=1}^n{Y}_i^2-n{\bar{Y}}^2\right]$
$=\frac{1}{n}\left[{\sum }_{i=1}^{n}E(Y_i^2)-nE({\bar{Y}}^2)\right]$
$=\frac{1}{n}\left[{\sum }_{i=1}^n({\sigma }^2+{\mu }^2)-n\left(\frac{{\sigma }^2}{n}+{\mu }^2\right)\right]$
$=\frac{n-1}{n}{\sigma }^2$

So ${\hat{\sigma }}^2$ is biased (but not asymptotically)
The unbiased version is often called the sample variance denoted $S^2=\frac{1}{n-1}{\sum }_{i=1}^n(Y_i-\bar{Y}{)}^2$

The sample standard deviation is denoted $S=\sqrt{\frac{1}{n-1}{\sum }_{i=1}^n(Y_i-\bar{Y}{)}^2}$and is the most common estimator for $\sigma$, although ironically it is biased (taking the square root introduces a downward bias).

So great, we’ve figured out how to unbias an estimator. Now how do we choose between unbiased estimators?

Efficiency

Besides biasedness, we can also measure the precision of an estimator in terms of its variance

Definition: Between two unbiased estimators, if $\text{Var}({\hat{\theta }}_1)<\text{Var}({\hat{\theta }}_2)$ then we say ${\hat{\theta }}_1$ is more efficient than ${\hat{\theta }}_2$. The relative efficiency of ${\hat{\theta }}_1$ with respect to ${\hat{\theta }}_2$ is $\text{Var}({\hat{\theta }}_2)/\text{Var}({\hat{\theta }}_1)$

An obvious question (that Shahriar waved away in class for some reason) is whether we can identify an estimator with the smallest variance

Cramér-Rao Inequality: Let $f_Y(y;\theta )$ be a continuous pdf with continuous first-order and second-order derivatives. Also, suppose that the set of $y$ values, where $f_Y(y;\theta )\ne 0$, does not depend on $\theta$.

$$\text{Var}(\hat{\theta })\ge {\left\{nE\left[{\left(\frac{\partial \ln f_Y(Y;\theta )}{\partial \theta }\right)}^2\right]\right\}}^{-1}={\left\{-nE\left[\frac{{\partial }^2\ln f_Y(Y;\theta )}{\partial {\theta }^2}\right]\right\}}^{-1}$$

Definition: Let $\Theta$ be the set of all unbiased estimators for $f_Y(y;\theta )$. We say that ${\hat{\theta }}^{\ast }$ is a best or minimum-variance estimator if ${\hat{\theta }}^{\ast }\in \Theta$ and $\text{Var}({\hat{\theta }}^{\ast })\le \text{Var}(\hat{\theta })$ for all $\hat{\theta }\in \Theta$.

The unbiased estimator $\hat{\theta }$ is efficient if $\text{Var}(\hat{\theta })$ equals the Cramér-Rao lower bound for $f_Y(y;\theta )$. The efficiency of $\hat{\theta }$ is the ratio of the lower bound for $f_Y(y;\theta )$ to $\text{Var}(\hat{\theta })$.

Note that it’s possible for the best estimator to not be efficient (meaning no estimators can reach the Cramér-Rao lower bound)

Sufficiency

Suppose that a random sample of size $n$ is taken $X_1=k_1,\dots ,X_n=k_n$ from the Bernoulli pdf $p_X(k;p)=p^k(1-p{)}^{1-k}$. The maximum likelihood estimator is $\hat{p}=\left(\frac{1}{n}\right){\sum }_{i=1}^n{X}_i$, so our maximum likelihood estimate is $p_e=\left(\frac{1}{n}\right){\sum }_{i=1}^n{k}_i$

We’re interested in the conditional probability $P(X_1=k_1,\dots ,X_n=k_n\mid \hat{p}=p_e)=\frac{P(X_1=k_1,\dots ,X_n=k_n)}{P(\hat{p}=p_e)}$
$P(X_1=k_1,\dots ,X_n=k_n)=p^{n{p}_e}(1-p{)}^{n-n{p}_e}$
$P(\hat{p}=p_e)=P\left({\sum }_{i=1}^n{X}_i=n{p}_e\right)=\left(\genfrac{}{}{0ex}{}{n}{n{p}_e}\right)p^{n{p}_e}(1-p{)}^{n-n{p}_e}$
so our conditional probability equals $\frac{1}{\left(\genfrac{}{}{0ex}{}{n}{n{p}_e}\right)}$

Our estimator is sufficient, because $\frac{1}{\left(\genfrac{}{}{0ex}{}{n}{n{p}_e}\right)}$ is not a function of $p$
This conceptually tells us that everything our data can tell us about $p$ is contained in $p_e$, since otherwise the joint pdf $P(X_1=k_1,\dots ,X_n=k_n)$ is entirely a function of $k$ and $p$

Definition: The statistic $\hat{\theta }=h(X_1,\dots ,X_n)$ is sufficient for $\theta$ if $L(\theta )={\prod }_{i=1}^n{p}_X(k_i;\theta )=p_{\hat{\theta }}({\theta }_e;\theta )b(k_1,\dots ,k_n)$

A one-to-one function of a sufficient statistic is also sufficient, therefore we can construct an unbiased estimator for $\theta$ from a sufficient but biased statistic on $\theta$

Sometimes this definition is difficult to use, since our statistic might have a difficult pdf. The following theorem is an alternative factorization criterion

Theorem: The statistic $\hat{\theta }=h(X_1,\dots ,X_n)$ is sufficient if and only if there are functions $L(\theta )=g[h(k_1,\dots ,k_n);\theta ]\cdot b(k_1,\dots ,k_n)$

This works because we are able to “convert” the $g$ portion to include the pdf of $\hat{\theta }$

If a sufficient estimator ${\hat{\theta }}_S$ exists for $\theta$ then we know an MLE for $\theta$ will maximize $g({\theta }_e;\theta )$ and therefore be a function of ${\theta }_e$. It follows that maximum likelihood estimators are functions of sufficient estimators, which is an important justification for preferring them over method of moments estimators. (Would this imply they are often also sufficient, according to this statement above)

Rao-Blackwell: The variance of every unbiased estimator based on a sufficient estimator will necessarily be less than the variance of every unbiased estimator that is not a function of a sufficient estimator. This lets us restrict our search for estimators.

Consistency

It’s sometimes important to consider the asymptotic behavior of estimators, in case they have a problem that arises in the limit

For example, earlier we defined the term asymptotically unbiased

Definition: An estimator ${\hat{\theta }}_n=h(W_1,W_2,\dots ,W_n)$ is consistent for $\theta$ if it converges in probability to $\theta$, i.e. ${\lim }_{n\to \infty }P(|{\hat{\theta }}_n-\theta |<ϵ)=1$ for all $ϵ>0$

We can define this similarly by saying there exists an $n(ϵ,\delta )$ such that $P(|{\hat{\theta }}_n-\theta |<ϵ)>1-\delta$ for $n>n(ϵ,\delta )$ for all $ϵ>0$, $\delta >0$

This can be used as a tool to list the required number of samples given $\theta$, $ϵ$, and $\delta$

Chebyshev’s Inequality: Let $W$ be any random variable with mean $\mu$ and variance ${\sigma }^2$. For any $ϵ>0$,

$$P(|W-\mu |<ϵ)\ge 1-\frac{{\sigma }^2}{{ϵ}^2}$$

Proof: In the continuous case, $\text{Var}(Y)={\int }_{-\infty }^{\infty }(y-\mu {)}^2{f}_Y(y)dy$
$={\int }_{-\infty }^{\mu -ϵ}(y-\mu {)}^2{f}_Y(y)dy+{\int }_{\mu -ϵ}^{\mu +ϵ}(y-\mu {)}^2{f}_Y(y)dy+{\int }_{\mu +ϵ}^{\infty }(y-\mu {)}^2{f}_Y(y)dy$
$\ge {\int }_{-\infty }^{\mu -ϵ}(y-\mu {)}^2{f}_Y(y)dy+{\int }_{\mu +ϵ}^{\infty }(y-\mu {)}^2{f}_Y(y)dy$
$\ge {\int }_{|y-\mu |\ge ϵ}^{}(y-\mu {)}^2{f}_Y(y)dy$
$\ge {\int }_{|y-\mu |\ge ϵ}^{}{ϵ}^2{f}_Y(y)dy$

$\frac{{\sigma }^2}{{ϵ}^2}\ge P(|Y-\mu |\ge ϵ)$
Or as stated above, $P(|Y-\mu |<ϵ)\ge 1-\frac{{\sigma }^2}{{ϵ}^2}$

Chebyshev’s inequality is a useful tool for proving consistency (I believe specifically for unbiased estimators). We can use it to show that a bound can be established satisfying $\delta$

Related to Chebyshev’s is the weak law of large numbers says that the sample mean ${\hat{\mu }}_n$ is a consistent estimator for $\mu$

Also note that we can generally show that maximum likelihood estimators are consistent