Hypothesis Testing

Hypothesis testing aims to choose between a null hypothesis $H_0$ and an alternative hypothesis $H_1$

A function of the observed data who dictates the outcome of our hypothesis is called a test statistic. The set of values that result in our null hypothesis’ rejection is the critical region, denoted $C$. The points separating $C$ from the acceptance region is the critical value.

The probability a test statistic rejects $H_0$ when $H_0$ is true is the level of signifiance, denoted $\alpha$

$\alpha =0.01,0.05,0.1$ are common

Theorem: Let $y_1,y_2,\dots ,y_n$ be a random sample from a normal distribution with $\sigma$. $H_0:=\mu ={\mu }_0$. Let $z=\frac{\bar{y}-{\mu }_0}{\sigma /\sqrt{n}}$

• Accept $H_1:=\mu >{\mu }_0$ if $z\ge z_{\alpha }$
• Accept $H_1:=\mu <{\mu }_0$ if $z\le -z_{\alpha }$
• Accept $H_1:\mu \ne {\mu }_0$ if $z\le -z_{\alpha /2}$ or $z\ge z_{\alpha /2}$

An alternative way of formulating the same idea, the P-value of a test statistic is the probability of getting a value at least as extreme as a given statistic, assuming $H_0$ is true

We can run binomial hypothesis tests much like the test above. When we have a “large” number of samples, we can approximate a group of samples as a normal distribution.

We use the condition $0<n{p}_0-3\sqrt{n{p}_0(1-p_0)}<n{p}_0+3\sqrt{n{p}_0(1-p_0)}<n$ to identify large samples. This is true when the range of 3 standard deviations fall into the valid values of $X$.

$\sigma =\sqrt{n{p}_0(1-p_0)}$

Theorem: Let $k=k_1+k_2+\cdots +k_n$ be a random sample of $n$ Bernoulli random variables, for which $0<n{p}_0-3\sqrt{n{p}_0(1-p_0)}<n{p}_0+3\sqrt{n{p}_0(1-p_0)}<n$. We use $z=\frac{k-n{p}_0}{\sqrt{n{p}_0(1-p_0)}}$ and test as above.

If our inequality doesn’t hold, we use the exact binomial distribution
$P(X\ge k)={\sum }_{x=k}^{\infty }\left(\genfrac{}{}{0ex}{}{n}{x}\right)p_0^x(1-p_0{)}^{n-x}$
$P(X\le k)={\sum }_{x=0}^k\left(\genfrac{}{}{0ex}{}{n}{x}\right)p_0^x(1-p_0{)}^{n-x}$

• Accept $H_1:=p>p_0$ if $P(X\ge k)\le \alpha$
• Accept $H_1:=p<p_0$ if $P(X\le k)\le \alpha$
• Accept $H_1:=p\ne p_0$ if $P(X\le k)+P(X\ge k)\le \alpha$

This is equivalent to our previous theorems, but stated terms of the CDF and $\alpha$ instead of the critical value and $z_{\alpha }$ (which is the inverse-CDF of normal)

Type I and II Errors

We already defined $\alpha =$ the probability of incorrectly rejecting $H_0$. Call this the Type I error.

$\beta =$ the probability of incorrectly accepting $H_0$, is called the Type II error.

$\beta$ is a function of the presumed value $\mu$. If $\mu$ is really close to ${\mu }_0$ then $\beta$ will be high.

Definition: $1-\beta$ is the power of a decision test, as a function of the parameter being tested. A power curve graphs this relation.

The power of a test diminishes as $\mu \to {\mu }_0$. At $\mu ={\mu }_0$, $1-\beta =\alpha$.

A steeper power curve is a stronger test

The power of the $Z$ test is a function of $\alpha$, $\sigma$, and $n$. We can improve our power by either decreasing $\alpha$, decreasing $\sigma$, or increasing $n$.

Example: Say we’d like to test $H_0:\mu =100$ versus $H_1:\mu >100$ with $\alpha =0.05$ and $1-\beta =0.60$ when $\mu =103$. What is the smallest sample size that achieves this objective? Assume a normal distribution with $\sigma =14$

We solve this by writing a system of equations with the critical value ${\bar{y}}^{\ast }$
${\bar{y}}^{\ast }=100+z_{\alpha }\cdot \frac{14}{\sqrt{n}}$
${\bar{y}}^{\ast }=103+z_{\beta }\cdot \frac{14}{\sqrt{n}}$

Nonnormal Decision Rules

Decision rules on general pdfs $f_Y(y;\theta )$ work much the same

To test $H_0:\theta ={\theta }_o$, we define a decision rule in terms of a sufficient statistic $\hat{\theta }$. The critical region is the set of values of $\hat{\theta }$ least compatible with ${\theta }_o$ and admissible under $H_1$ whose total probability when $H_0$ is true is $\alpha$

Example: $f_Y(y;\theta )=1/\theta$ for $0\le y\le \theta$
$n=8$
$H_0:\theta =2.0$
$H_1:\theta <2.0$
$\alpha =0.10$

Suppose we base the decision rule on $Y_8^{\prime }$, the largest order statistic. What’s $\beta$ when $\theta =1.7$?

Some thought about the uniform distribution makes it clear we are looking for some value $P(Y_8^{\prime }\le c\mid H_0\text{ is true})=0.10$

$f_{Y_8^{\prime }}(y;\theta =2)=8{\left(\frac{y}{2}\right)}^7\cdot \frac{1}{2}$ for $0\le y\le 2$
${\int }_0^{c}8{\left(\frac{y}{2}\right)}^7\cdot \frac{1}{2}dy=0.10$
$c=1.50$

$\beta =P(Y_8^{\prime }>1.50\mid \theta =1.7)={\int }_{1.50}^{1.7}8{\left(\frac{y}{1.7}\right)}^7\cdot \frac{1}{1.7}dy=0.63$

The Generalized Likelihood Ratio

What is the best decision rule for choosing between $H_0$ and $H_1$, and how do we show it is optimal?

Define $\omega$ as the set of unknown parameter values admissible under $H_0$
Define $\Omega$ as the set of all possible values of all unknown parameters

Definition: Let $y_1,y_2,\dots ,y_n$ be a random sample from $f_Y(y;{\theta }_1,\dots ,{\theta }_k)$. The generalized likelihood ratio is $\lambda =\frac{{\max }_{\omega }L({\theta }_1,\dots ,{\theta }_k)}{{\max }_{\Omega }L({\theta }_1,\dots ,{\theta }_k)}=\frac{L({\omega }_e)}{L({\Omega }_e)}$

Maximizing $L(\theta )$ under $\Omega$ (no restrictions), is accomplished by substituting ${\theta }_e$ into $L(\theta )$

Values closer to $1$ show that the data is more compatible with $H_0$

Definition: A generalized likelihood ratio test (GLRT) rejections $H_0$ whenever $0<\lambda \le {\lambda }^{\ast }$ where $P(0<\Lambda \le {\lambda }^{\ast }\mid H_0\text{ is true})=\alpha$ ($\Lambda$ is $\lambda$ expressed as a random variable)

Expressed similarly, $\alpha ={\int }_0^{{\lambda }^{\ast }}{f}_{\Lambda }(\lambda \mid H_0)d\lambda$
In many situations, $f_{\Lambda }(\lambda \mid H_0)$ is not known, so we must show $\Lambda$ is a monotonic function of a quantity $W$, where the distribution of $W$ is known.

Example:
So say we work with a uniform distribution, $H_0:\theta ={\theta }_o$ and $H_1:\theta <{\theta }_o$. $\omega =\{{\theta }_o\}$ and $\Omega =\{\theta :0<\theta \le {\theta }_o\}$

We have $\lambda =\frac{(1/{\theta }_0{)}^n}{(1/y_{\text{max}}{)}^n}={\left(\frac{y_{\text{max}}}{{\theta }_0}\right)}^n$

$\alpha =P\left[{\left(\frac{Y_{\text{max}}}{{\theta }_0}\right)}^n\le {\lambda }^{\ast }\mid H_0\text{ is true}\right]$
Instead of trying to solve for ${\lambda }^{\ast }$ directly, we take $W=Y_{\text{max}}/{\theta }_0$ and $w^{\ast }=\sqrt[n]{{\lambda }^{\ast }}$

$f_W(w;{\theta }_0)={\theta }_0{f}_{Y_{\text{max}}}({\theta }_{0}w;{\theta }_0)=\frac{{\theta }_{0}n({\theta }_{0}w{)}^{n-1}}{{\theta }_0^n}=n{w}^{n-1}$ for $0\le w\le 1$
$P(W\le w^{\ast })\mid H_0\text{ is true}={\int }_0^{w^{\ast }}n{w}^{n_{0}1}dw=(w^{\ast }{)}^n=\alpha$

We conclude that the GLRT calls to reject $H_0$ if $\frac{y_{\text{max}}}{{\theta }_0}\le \sqrt[n]{\alpha }$
Fancy!