Two-Sample Tests

The one-sample model is nice and simple but can be limited in utility. More useful are methods that compare responses to different treatment levels.

Two-sample inferences fall into two categories,

• Choosing between two sets
• Measuring similarity between two sets

This usually involves testing the “location” of distributions, but we can compare variability

Testing $H_0$: ${\mu }_X={\mu }_Y$

Assume that two random samples $X_1,\dots ,X_n$ and $Y_1,\dots ,Y_m$ are drawn from independent normal distributions

Theorem: Let $S_X^2$ and $S_Y^2$ be corresponding sample variances and $S_p^2$ the pooled variance. We have $S_p^2=\frac{(n-1)S_X^2+(m-1)S_Y^2}{n+m-2}=\frac{\Sigma (X_i-\bar{X}{)}^2+\Sigma (Y_i-\bar{Y}{)}^2}{n+m-2}$. So $T_{n+m-2}=\frac{\bar{X}-\bar{Y}-({\mu }_X-{\mu }_Y)}{S_p\sqrt{\frac{1}{n}+\frac{1}{m}}}$ is a Student $t$ distribution with $n+m-2$ df.

Theorem: Let $x_1,\dots ,x_n$ and $y_1,\dots ,y_m$ be independent random samples from normal distributions with means ${\mu }_X$ and ${\mu }_Y$ and equal standard deviation $\sigma$. Let $t=\frac{\bar{x}-\bar{y}}{s_p\sqrt{\frac{1}{n}+\frac{1}{m}}}$

• Accept $H_1:={\mu }_X>{\mu }_Y$ if $t\ge t_{\alpha ,n+m-2}$
• Accept $H_1:={\mu }_X<{\mu }_Y$ if $t\le -t_{\alpha ,n+m-2}$
• Accept $H_1:={\mu }_X\ne {\mu }_Y$ if either $t\le -t_{\alpha /2,n+m-2}$ or $t\ge t_{\alpha /2,n+m-2}$

An equivalent $100(1-\alpha )\%$ sample interval for ${\mu }_X-{\mu }_Y$ is $\left(\bar{x}-\bar{y}-t_{\alpha /2,n+m-2}\cdot s_p\sqrt{\frac{1}{n}+\frac{1}{m}},\bar{x}-\bar{y}+t_{\alpha /2,n+m-2}\cdot s_p\sqrt{\frac{1}{n}+\frac{1}{m}}\right)$

When the standard deviations of the samples are not equal, things are more complicated. This is the Behrens-Fisher problem, and no exact solution is known

Theorem: Let $X_1,\dots ,X_n$ and $Y_1,\dots ,Y_m$ be independent random samples from normal distributions (with separate standard deviations). Let $W=\frac{\bar{X}-\bar{Y}-({\mu }_X-{\mu }_Y)}{\sqrt{\frac{S_X^2}{n}+\frac{S_Y^2}{m}}}$. $W$ has approximately a Student $t$ distribution with $\nu$ degrees of freedom where $\nu =\frac{{\left(\hat{\theta }+\frac{n}{m}\right)}^2}{\frac{1}{(n-1)}{\hat{\theta }}^2+\frac{1}{(m-1)}{\left(\frac{n}{m}\right)}^2}$ and $\hat{\theta }=\frac{s_X^2}{s_Y^2}$

The justification is a bit complicated, but essentially the weighted average of variances works as a valid approximation

Testing $H_0$: ${\sigma }_X^2={\sigma }_Y^2$

Testing equality of variances can be useful. Sometimes variance is the direct measure of interest. Sometimes it is used to justify the exact two-sample test used above.

We use the $F$ distribution to test this. The ratio of sample variances will be equal to an $F$ distribution assuming the true variances are equal.

Theorem: Let $f=\frac{s_Y^2}{s_X^2}$,

• Accept $H_1:{\sigma }_X^2>{\sigma }_Y^2$ if $f\le F_{\alpha ,m-1,n-1}$
• Accept $H_1:{\sigma }_X^2<{\sigma }_Y^2$ if $f\ge F_{1-\alpha ,m-1,n-1}$
• Accept $H_1:{\sigma }_X^2\ne {\sigma }_Y^2$ if $f\notin [F_{\alpha /2,m-1,n-1},F_{1-\alpha /2,m-1,n-1}]$

A $100(1-\alpha )\%$ interval for ${\sigma }_X^2/{\sigma }_Y^2$ is $(f\cdot F_{\alpha /2,m-1,n-2},f\cdot F_{1-\alpha /2,m-1,n-1})$

Non-Normal Data

It’s also important to consider non-normal distributions, which can be either continuous or discrete

Suppose $n$ Bernoulli trials in treatment $X$ yield $x$ successes and $m$ in treatment $Y$ yield $y$ successes. We’d like to check $H_0$: $p_X=p_Y$ versus $H_1$: $p_X\ne p_Y$

We use the GLRT to come up with a test

$\omega =\{(p_X,p_Y):0\le p_X=p_Y\le 1\}$
$\Omega =\{(p_X,p_Y):0\le p_X\le 1,0\le p_Y\le 1\}$

$L=p_X^x(1-p_X{)}^{n-x}\cdot p_Y^y(1-p_Y{)}^{m-y}$

MLE with $p_X=p_Y$ gives,
$p_e=\frac{x+y}{n+m}$

MLE with individual $p_X,p_Y$ gives,
$p_{X_e}=\frac{x}{n}$
$p_{Y_e}=\frac{y}{m}$

Substituting the sample spaces back into $L$ gives,
$\lambda =\frac{L({\omega }_e)}{L({\Omega }_e)}=\frac{[(x+y)/(n+m){]}^{x+y}[1-(x+y)/(n+m){]}^{n+m-x-y}}{(x/n{)}^x[1-(x/n){]}^{n-x}(y/m{)}^y[1-(y/m){]}^{m-y}}$

This is hard to work with. It can be shown that $-2\ln \lambda$ has an asymptotic ${\chi }^2$ distribution with $\text{df}=1$ (so reject $H_0$ if $-2\ln \lambda \ge 3.84$).

Another more common approach is to appeal to CLT, noting that $\frac{\frac{X}{n}-\frac{Y}{m}-E\left(\frac{X}{n}-\frac{Y}{m}\right)}{\sqrt{\text{Var}(\frac{X}{n}-\frac{Y}{m})}}$ is approximately normal

Under $H_0$,
$E(\frac{X}{n}-\frac{Y}{m})=0$
$\text{Var}\left(\frac{X}{n}-\frac{Y}{m}\right)=\frac{(n+m)p(1-p)}{nm}$ (maximize with $p=p_e$)

Theorem: Let $x$ and $y$ denote the numbers of successes observed in two independent sets of $n$ and $m$ Bernoulli trials, where $p_X$ and $p_Y$ are the true success probabilities,

Let $z=\frac{\frac{x}{n}-\frac{y}{m}}{\sqrt{\frac{p_e(1-p_e)}{n}+\frac{p_e(1-p_e)}{m}}}$ where $p_e=\frac{x+y}{n+m}$,

• Accept $H_1:=p_X>p_Y$ if $z\ge z_{\alpha }$
• Accept $H_1:=p_X<p_Y$ if $z\le -z_{\alpha }$
• Accept $H_1:=p_X\ne p_Y$ if $z\notin [-z_{\alpha /2},z_{\alpha /2}]$

A $100(1-\alpha )\%$ confidence interval for $p_X-p_Y$ is $\left(\frac{x}{n}-\frac{y}{m}-Z,\frac{x}{n}-\frac{y}{m}+Z\right)$ where $Z=z_{\alpha /2}\sqrt{\frac{\left(\frac{x}{n}\right)\left(1-\frac{x}{n}\right)}{n}+\frac{\left(\frac{y}{m}\right)\left(1-\frac{y}{m}\right)}{m}}$