Sampling Distributions

The normal distribution is pretty rad. We’ve already talked about the usefulness of $Z$-tests for Hypothesis Testing.md

The $Z$-tests we’ve considered so far have required us knowing the true variance ${\sigma }^2$. But what if we don’t know this?

The typical MLE estimator is $S^2=\frac{1}{n-1}{\sum }_{i=1}^n(Y_i-\bar{Y}{)}^2$
The logical question is whether we can then use $\frac{\bar{Y}-\mu }{S/\sqrt{n}}$ for our decision rules

When $n$ is large, we can basically treat them the same. However, when $n$ is smaller, changing $\sigma$ to $S$ makes a difference.

William Sealy Gossett graduated from Oxford in 1899 with First Class degrees in chemistry and mathematics. He took a position at Guinness and was tasked with making their recipes more scientific. This task had inherently small sample sizes and he became convinced that $\frac{\bar{Y}-\mu }{S/\sqrt{n}}$ had a different pdf. He derived the proper pdf and published it anonymously (under the name Student) in 1908, since Guinness forbid employees from publishing papers.

It took a while for anyone to realize the importance of this work. We now recognize it as the Student t distribution

In deriving this distribution, we will encounter several other sampling distributions, distributions that model the behavior of functions based on sets of random variables, used for inference

Theorem: $U={\sum }_{j=1}^m{Z}_j^2$ where $Z_i$ is a standard normal RV has a gamma distribution with $r=\frac{m}{2}$ and $\lambda =\frac{1}{2}$. $f_U(u)=\frac{1}{2^{m/2}\Gamma \left(\frac{m}{2}\right)}{u}^{m/2-1}{e}^{-u/2}$

Consider $m=1$, $F_{Z^2}(u)=P(Z^2\le u)=2P(0\le Z\le \sqrt{u})$
$=\frac{2}{\sqrt{2\pi }}{\int }_0^{\sqrt{u}}{e}^{-z^2/2}dz$
$f_{Z^2}(u)=\frac{2}{\sqrt{2\pi }}\frac{1}{2\sqrt{u}}{e}^{-u/2}=\frac{1}{2^{1/2}\Gamma \left(\frac{1}{2}\right)}{u}^{1/2-1}{e}^{-u/2}$
This is a gamma pdf with $r=\frac{1}{2}$, $\lambda =\frac{1}{2}$
The sum of $m$ gamma pdfs like this has $r=\frac{m}{2}$, $\lambda =\frac{1}{2}$

Definition: The pdf of $U={\sum }_{j=1}^m{Z}_j^2$ is called the chi square distribution with m degrees of freedom

Theorem: $S^2$ and $\bar{Y}$ are independent and $\frac{(n-1)S^2}{{\sigma }^2}=\frac{1}{{\sigma }^2}{\sum }_{i=1}^n(Y_i-\bar{Y}{)}^2$ has a chi square distribution with $n-1$ degrees of freedom

Definition: Suppose that $U$ and $V$ are independent chi square random variables with $n$ and $m$ degrees of freedom. $\frac{V/m}{U/n}$ is said to have an F distribution with m and n degrees of freedom

$F$ commemorates Sir Ronald Fisher (also involved with the Student $t$ distribution)

Theorem: $f_{F_{m,n}}(w)=\frac{\Gamma \left(\frac{m+n}{2}\right)m^{m/2}{n}^{n/2}{w}^{m/2-1}}{\Gamma \left(\frac{m}{2}\right)\Gamma \left(\frac{n}{2}\right)(n+mw{)}^{(m+n)/2}}$, $w\ge 0$
We derive this with two equations,
$f_{V/U}(w)={\int }_0^{\infty }|u|f_U(u)f_V(uw)du$ and
$f_{\frac{V/m}{U/n}}(w)=\frac{m}{n}{f}_{V/U}\left(\frac{m}{n}w\right)$

Definition: Let $Z$ be a standard normal random variable and let $U$ be a chi square random variable independent of $Z$ with $n$ degrees of freedom. The Student t ratio with n degrees of freedom is $T_n=\frac{Z}{\sqrt{U/n}}$

We often abbreviate degrees of freedom as df

$f_{T_n}(t)=f_{T_n}(-t)$

Theorem: The pdf for a Student $t$ random variable with $n$ degrees of freedom is $f_{T_n}(t)=\frac{\Gamma \left(\frac{n+1}{2}\right)}{\sqrt{n\pi }\Gamma \left(\frac{n}{2}\right){\left(1+\frac{t^2}{n}\right)}^{(n+1)/2}}$ for $-\infty <t<\infty$. This is often denoted as $f_t(t)$.

We use $T_n^2=\frac{Z^2}{U/n}$ (an $F$ distribution) to derive the pdf

Theorem: $T_{n-1}=\frac{\bar{Y}-\mu }{S/\sqrt{n}}$

The proof is simple, $\frac{\bar{Y}-\mu }{S/\sqrt{n}}=\frac{\frac{\bar{Y}-\mu }{\sigma /\sqrt{n}}}{\sqrt{\frac{(n-1)S^2}{{\sigma }^2(n-1)}}}=\frac{Z}{\sqrt{U/(n-1)}}$

Both $T_n$ and $Z$ are bell shaped and symmetric around zero. $T_n$ is flatter and has thicker tails. As $n$ increases, $T_n$ approaches $Z$.

Drawing Inferences

Now we can draw inferences about $\mu$ when $\sigma$ is not known

Theorem: Let $y_1,y_2,\dots ,y_n$ be a randoms ample from a normal distribution with unknown mean $\mu$. A $100(1-\alpha )\%$ confidence interval for $\mu$ is $\left(\bar{y}-t_{\alpha /2,n-1}\cdot \frac{s}{\sqrt{n}},\bar{y}+t_{\alpha /2,n-1}\cdot \frac{s}{\sqrt{n}}\right)$

The procedure for testing $H_0:\mu ={\mu }_0$ for unknown $\sigma$ is called the one-sampled t test

Theorem: Let $y_1,y_2,\dots ,y_n$ be a random sample from a normal distribution with unknown $\sigma$. $H_0:=\mu ={\mu }_0$. Let $t=\frac{\bar{y}-{\mu }_0}{s/\sqrt{n}}$

• Accept $H_1:=\mu >{\mu }_0$ if $t\ge t_{\alpha ,n-1}$
• Accept $H_1:=\mu <{\mu }_0$ if $t\le -t_{\alpha ,n-1}$
• Accept $H_1:\mu \ne {\mu }_0$ if $t\le -t_{\alpha /2,n-1}$ or $t\ge t_{\alpha /2,n-1}$

We prove this by showing that $t$ is a monotonic function of $\lambda$, satisfying GLRT

Of course, $t$ tests make the assumption that our samples are normally distributed. However,

1. The distribution of $\frac{\bar{Y}-\mu }{S/\sqrt{n}}$ is relatively unaffected by the pdf of $y_i$, provided $f_Y(y)$ is not too skewed and $n$ is not too small
2. As $n$ increases, $\frac{\bar{Y}-\mu }{S/\sqrt{n}}$ becomes increasingly similar to $f_{T_{n-1}}(t)$

This is awesome. Our $t$ test is robust, meaning it is not heavily dependent on its assumptions. Departures from normality are acceptable.

Sometimes we’d like to estimate ${\sigma }^2$ instead of $\mu$. We now have the tools to do this.

$\frac{(n-1)S^2}{{\sigma }^2}$ has a chi square distribution with df $=n-1$
$P\left[{\chi }_{\alpha /2,n-1}^2\le \frac{(n-1)S^2}{{\sigma }^2}\le {\chi }_{1-\alpha /2,n-1}^2\right]$

Theorem: Let $s^2$ denote the sample variance from $n$ observations drawn from a normal distribution. A $100(1-\alpha )\%$ confidence interval for ${\sigma }^2$ is $\left(\frac{(n-1)s^2}{{\chi }_{1-\alpha /2,n-1}^2},\frac{(n-1)s^2}{{\chi }_{\alpha /2,n-1}^2}\right)$

We can create a corresponding decision test

Theorem: Let $s^2$ be the sample variance from $n$ observations drawn from a normal distribution. $H_0:{\sigma }^2={\sigma }_0^2$. Let ${\chi }^2=(n-1)s^2/{\sigma }_0^2$.

• Accept $H_1:={\sigma }^2>{\sigma }_0^2$ if ${\chi }^2\ge {\chi }_{1-\alpha ,n-1}^2$
• Accept $H_1:={\sigma }^2<{\sigma }_0^2$ if ${\chi }^2\le {\chi }_{1-\alpha ,n-1}^2$
• Accept $H_1:{\sigma }^2\ne {\sigma }_0^2$ if ${\chi }^2\le -{\chi }_{\alpha /2,n-1}^2$ or ${\chi }^2\ge {\chi }_{\alpha /2,n-1}^2$

Working with Type II error under these new sampling distributions is quite more involved and involves working with noncentral distributions