Simple Linear Regression

Given a set of $n$ points $(x_i,y_i)$ and a positive integer $m$, which polynomial of degree $m$ is “closest” to the given points?

$p(x)=a+{\sum }_{i=1}^m{b}_i{x}^i$

Definition: The method of least squares finds the coefficients of $p$ minimizing $L={\sum }_{i=1}^n[y_i-p(x_i){]}^2$

Theorem: The line $y=a+bx$ minimizing $L$ has slope $b=\frac{n{\sum }_{i=1}^n{x}_i{y}_i-\left({\sum }_{i=1}^n{x}_i\right)\left({\sum }_{i=1}^n{y}_i\right)}{n\left({\sum }_{i=1}^n{x}_i^2\right)-{\left({\sum }_{i=1}^n{x}_i\right)}^2}$ and y-intercept $a=\frac{{\sum }_{i=1}^n{y}_i-b{\sum }_{i=1}^n{x}_i}{n}=\bar{y}-b\bar{x}$

This is solved by taking partial derivatives on $L$

Definition: $\hat{y}=a+bx$ is known as the predicted value of $y$
Definition: $y_i-\bar{y}$ is the ith residual

Residual plots graph residuals over $x$

If a residual plot shows no patterns or trends, the relationship is taken to be linear

Our theorem can also work on more complicated relationships, as long as they can be linearized

• Exponential regression: $y=a{e}^{bx}$ $⟹$ $\ln y=\ln a+bx$
• Logarithmic regression: $y=a{x}^b$ $⟹$ $\log y=\log a+b\log x$
• Logistic regression: $y=\frac{L}{1+e^{a+bx}}⟹\ln \left(\frac{L-y}{y}\right)=a+bx$

There are many other curvilinear models

The Linear Model

It’s more realistic to think of values as sampling from a random variable $Y$

Definition: Let $f_{Y|x}(y)$ denote the pdf of the random variable $Y$ for a given value $x$. $y=E(Y\mid x)$ is the regression curve of $Y$ on $x$

Definition: The simple linear model makes four assumptions:

1. $f_{Y\mid x}(y)$ is a normal pdf for all $x$
2. $\sigma$ is the same for all $x$
3. $y=E(Y\mid x)={\beta }_0+{\beta }_{1}x$
4. Each conditional distribution is independent

Theorem: Let $(x_1,Y_1),(x_2,Y_2),\dots ,$ and $(x_n,Y_n)$ be a set of points satisfying the simple linear model, $E(Y\mid x)={\beta }_0+{\beta }_{1}x$. The MLE estimators for ${\beta }_0,{\beta }_1,$ and ${\sigma }^2$ are given by ${\hat{\beta }}_1=\frac{n{\sum }_{i=1}^n{x}_i{Y}_i-\left({\sum }_{i=1}^n{x}_i\right)\left({\sum }_{i=1}^n{Y}_i\right)}{n\left({\sum }_{i=1}^n{x}_i^2\right)-{\left({\sum }_{i=1}^n{x}_i\right)}^2}$, ${\hat{\beta }}_0=\bar{Y}-{\hat{\beta }}_1\bar{x}$, and ${\hat{\sigma }}^2=\frac{1}{n}{\sum }_{i=1}^n(Y_i-{\hat{Y}}_i{)}^2$ where ${\hat{Y}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{x}_i$.

Theorem: Let ${\hat{\beta }}_0,{\hat{\beta }}_1,$ and ${\hat{\sigma }}^2$ be the MLE estimators for ${\beta }_0,{\beta }_1,$ and ${\sigma }^2$, respectively.

• ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$ are normally distributed
• ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$ are unbiased, $E({\hat{\beta }}_0)={\beta }_0$ and $E({\hat{\beta }}_1)={\beta }_1$
• $\text{Var}({\hat{\beta }}_1)=\frac{{\sigma }^2}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}$
• $\text{Var}({\hat{\beta }}_0)=\frac{{\sigma }^2{\sum }_{i=1}^n{x}_i^2}{n{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}$

Theorem:

• ${\hat{\beta }}_1$, $\bar{Y}$, and ${\hat{\sigma }}^2$ are mutually independent
• $\frac{n{\hat{\sigma }}^2}{{\sigma }^2}$ has a chi square distribution with $n-2$ degrees of freedom

The unbiased estimator for ${\sigma }^2$ is $S^2=\frac{1}{n-2}{\sum }_{i=1}^n(Y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i{)}^2$

Theorem: $T_{n-2}=\frac{{\hat{\beta }}_1-{\beta }_1}{S/\sqrt{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}}$ has Student $t$ distribution with $n-2$ degrees of freedom. Let $t=\frac{{\hat{\beta }}_1-{\beta }_1^{\prime }}{s/\sqrt{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}}$,

1. $H_1$: ${\beta }_1>{\beta }_1^{\prime }$ if $t\ge t_{\alpha ,n-2}$
2. $H_1:{\beta }_1<{\beta }_1^{\prime }$ if $t\le -t_{\alpha ,n-2}$
3. $H_1:{\beta }_1\ne {\beta }_1^{\prime }$ if $t\notin (-t_{\alpha /2,n-2},t_{\alpha /2,n-2})$

Let $w=t_{\alpha /2,n-2}\cdot \frac{s}{\sqrt{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}}$,
$[{\hat{\beta }}_1-x,{\hat{\beta }}_1+x]$ is a $100(1-\alpha )\%$ confidence interval for ${\beta }_1$

One useful test is checking for $H_0:{\beta }_1=0$, which says whether $E(Y)$ changes with $x$ or not

${\beta }_0$ is generally less interesting, but also has similar well-defined tests. Let $w=t_{\alpha /2,n-2}\cdot \frac{s\sqrt{{\sum }_{i=1}^n{x}_i^2}}{\sqrt{n}\sqrt{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}}$,
$[{\hat{\beta }}_0-x,{\hat{\beta }}_0+x]$ is a $100(1-\alpha )\%$ confidence interval for ${\beta }_0$

Likewise, $\left[\frac{(n-2)s^2}{{\chi }_{1-\alpha /2,n-2}^2},\frac{(n-2{)}^2}{{\chi }_{\alpha /2,n-2}^2}\right]$ is a $100(1-\alpha )\%$ confidence interval for ${\sigma }^2$
This is almost exactly the same as the regular test for variance, but with one less degree of freedom, since each estimated parameter essentially consumes a degree of freedom.

Theorem: A $100(1-\alpha )\%$ confidence interval for $E(Y\mid x)={\beta }_0+{\beta }_{1}x$ is given by $(\hat{y}-w,\hat{y}+w)$, where $w=t_{\alpha /2,n-2}\cdot s\sqrt{\frac{1}{n}+\frac{(x-\bar{x}{)}^2}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}}$ and $\hat{y}={\hat{\beta }}_0+{\hat{\beta }}_{1}x$.

This theorem is nifty but we might want a confidence interval for the actual range $Y$, not the expected range $E(Y\mid x)$. We can derive this with the random variable $\hat{Y}-Y$.

Theorem: A $100(1-\alpha )\%$ prediction interval for $Y$ at the fixed value $x$ is given by $(\hat{y}-w,\hat{y}+w)$, where $w=t_{\alpha /2,n-2}\cdot s\sqrt{1+\frac{1}{n}+\frac{(x-\bar{x}{)}^2}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}}$ and $\hat{y}={\hat{\beta }}_0+{\hat{\beta }}_{1}x$.

We can also devise a test to check the relationship between two independent regressions, where $H_0:{\beta }_1={\beta }_1^{\ast }$

Theorem: $T=\frac{{\hat{\beta }}_1-{\hat{\beta }}_1^{\ast }-({\beta }_1-{\beta }_1^{\ast })}{S\sqrt{\frac{1}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}+\frac{1}{{\sum }_{i=1}^m(x_i^{\ast }-{\bar{x}}^{\ast }{)}^2}}}$ where $S=\sqrt{\frac{{\sum }_{i=1}^n[Y_i-({\hat{\beta }}_0+{\hat{\beta }}_1{x}_i){]}^2+{\sum }_{i=1}^m[Y_i^{\ast }-({\hat{\beta }}_0^{\ast }+{\hat{\beta }}_1^{\ast }{x}_i^{\ast }){]}^2}{n+m-4}}$ is a Student $t$ distribution with $n+m-4$ degrees of freedom. Let $t=\frac{{\hat{\beta }}_1-{\hat{\beta }}_1^{\ast }}{s\sqrt{\frac{1}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}+\frac{1}{{\sum }_{i=1}^m(x_i^{\ast }-{\bar{x}}^{\ast }{)}^2}}}$, define decision tests in the usual way ($H_1:{\beta }_1\ne {\beta }_1^{\ast }$ if $t\notin (-t_{\alpha /2,n+m-4},t_{\alpha /2,n+m-4})$)

Covariance and Correlation

The current discussion was concerned with regression data. A more complicated situation is when measurements are of the form $(X_i,Y_i)$, where both $X$ and $Y$ are random but correlated variables.

Covariance is an important tool in measuring this kind of relation, but reflects the units of the individual variables

Definition: The correlation coefficient of $X$ and $Y$ is denoted $\rho (X,Y)=\frac{\text{Cov}(X,Y)}{{\sigma }_X{\sigma }_Y}=\text{Cov}(X^{\ast },Y^{\ast })$ where $X^{\ast }=(X-{\mu }_X)/{\sigma }_X$ and $Y^{\ast }=(Y-{\mu }_Y)/{\sigma }_Y$

Theorem: For any $X$ and $Y$,

• $|\rho (X,Y)|\le 1$
• $|\rho (X,Y)|=1$ $⟺$ $Y=aX+b$ for some constants $a$ and $b$

Can we estimate $\rho (X,Y)$? One useful identity is $\rho (X,Y)=\frac{E(XY)-E(X)E(Y)}{\sqrt{\text{Var}(X)}\sqrt{\text{Var}(Y)}}$

We can use this to define the sample correlation coefficient, $R=\frac{\frac{1}{n}{\sum }_{i=1}^n{X}_i{Y}_i-\bar{X}\bar{Y}}{\sqrt{\frac{1}{n}{\sum }_{i=1}^n(X_i-\bar{X}{)}^2}\sqrt{\frac{1}{n}{\sum }_{i=1}^n(Y_i-\bar{Y}{)}^2}}$

This is also known as the Pearson product-moment correlation coefficient (in honor of Karl Pearson)

We can use the square of the sample correlation coefficient and derive $r^2=\frac{{\sum }_{i=1}^n(y_i-\bar{y}{)}^2-{\sum }_{i=1}^n(y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i{)}^2}{{\sum }_{i=1}^n(y_i-\bar{y}{)}^2}=\frac{1-2}{1}$

1. Represents the total variability in the dependent variable
2. Represents the variation in the $y_i$‘s not accounted for by the linear regression with $x$

Therefore, $r^2$ is the proportion of the total variation in the $y_i$‘s that can be attributed to the linear relationship with $x$, sometimes called the coefficient of determination

The Bivariate Normal Distribution

After looking at relationships between two random variables, we start to wonder, can we generalize the normal distribution to higher dimensions?

Jumping to the definition…

Definition: $f_{X,Y}(x,y)=\frac{1}{2\pi {\sigma }_X{\sigma }_Y\sqrt{1-{\rho }^2}}\cdot \exp \left\{-\frac{1}{2}\left(\frac{1}{1-{\rho }^2}\right)\left[\frac{(x-{\mu }_X{)}^2}{{\sigma }_X^2}-2\rho \frac{x-{\mu }_X}{{\sigma }_X}\cdot \frac{y-{\mu }_Y}{{\sigma }_Y}+\frac{(y-{\mu }_Y{)}^2}{{\sigma }_Y^2}\right]\right\}$ is a bivariate normal distribution with parameters ${\mu }_X,{\sigma }_X^2,{\mu }_Y,{\sigma }_Y^2$, and $\rho$

Theorem: Suppose $X$ and $Y$ have a bivariate normal distribution,

1. $f_X(x)$ is normal with $({\mu }_X,{\sigma }_X^2)$ and $f_Y(y)$ is normal with $({\mu }_Y,{\sigma }_Y^2)$
2. $\rho$ is the correlation coefficient between $X$ and $Y$
3. $E(Y\mid x)={\mu }_Y+\frac{\rho {\sigma }_Y}{{\sigma }_X}(x-{\mu }_X)$
4. $\text{Var}(Y\mid x)=(1-{\rho }^2){\sigma }_Y^2$

Theorem: The maximum likelihood estimators for ${\mu }_X,{\mu }_Y,{\sigma }_X^2,{\sigma }_Y^2$, and $\rho$ are respectively $\bar{X},\bar{Y},\left(\frac{1}{n}\right){\sum }_{i=1}^n(X_i-\bar{X}{)}^2,\left(\frac{1}{n}\right){\sum }_{i=1}^n(Y_i-\bar{Y}{)}^2$, and $R$

How do we test the two variables are independent? We test $H_0:\rho =0$

Theorem: Under the null hypothesis $\rho =0$, $T_{n-2}=\frac{\sqrt{n-2}R}{\sqrt{1-R^2}}$ has a Student $t$ distribution with $n-2$ degrees of freedom