Special Distributions

Some probability distributions are extremely versatile and can be used to model a wide variety of natural phenomena

The binomial, hypergeometric, and uniform models were already explored in Probability. Some other key distributions are explored more in Statistics for their wide applicability.

Poisson Distribution

The binomial distribution is awesome, until you want to use it on scales. Then you’re stuck wondering, how do I calculate $p_X(50000)$?

The Poisson limit is an approximation used when $np=\lambda$ as $n\to \infty$, i.e. when $n$ is large, $p$ is small:

$$\underset{\stackrel{\stackrel{n\to \infty }{p{\to }_0}}{np=\lambda }}{\lim }\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k}=\frac{e^{-\lambda }{\lambda }^k}{k!}$$

How large does $n$ have to be and how small does $p$ have to be before the Poisson limit becomes a good approximation? Without trying to define “good approximation”, empirical calculations show that this can become remarkably good by $n=100$ ($\lambda =1$).

If you were clever and/or had previously heard about Poisson in your Computational Neuroscience class, you might realize that the binomial distribution taken asymptotically can be viewed as observing the occurrence of events over continuous time. The Poisson distribution was quickly (well it took 50 years), found to be very good at modeling situations that had neither an explicit binomial random variable nor defined values for $n$ and $p$.

Definition: The Poisson distribution is defined as $p_X(k)=P(X=k)=\frac{e^{-\lambda }{\lambda }^k}{k!}$ where $\lambda >0$. We find $E(X)=\text{Var}(X)=\lambda$

Let the random variable $Y$ denote the interval between consecutive events: $f_Y(y)=\lambda e^{-\lambda y}$ is an exponential distribution

Normal Distribution

De Moivre first derived another approximation for the binomial distribution,

$$P\left[a\le \frac{X-n\left(\frac{1}{2}\right)}{\sqrt{n\left(\frac{1}{2}\right)\left(\frac{1}{2}\right)}}\le b\right]=\frac{1}{\sqrt{2\pi }}{\int }_a^b{e}^{-z^2/2}dz$$

where $X$ is a binomial random variable, $n$ is large, and $p=1/2$

Laplace extended this to the De Moivre-Laplace theorem,

$$\underset{n\to \infty }{\lim }P\left[a\le \frac{X-np}{\sqrt{np(1-p)}}\le b\right]=\frac{1}{\sqrt{2\pi }}{\int }_a^b{e}^{-z^2/2}dz$$

where $X$ is a binomial random variable

Definition: $f_Z(z)=\frac{1}{\sqrt{2\pi }}{e}^{-z^2/2}$ is referred to as the standard normal (Gaussian) curve, and $Z$ is the convention for a random variable. We see that $E(Z)=0$ and $\text{Var}(Z)=1$

One way of proving this result is showing that the LHS’s moment-generating function is $M_Z(t)=e^{t^2/2}$ and that the RHS evaluates to the same quantity

There is no closed form for the CDF $F_Z(z)=P(Z\le z)$. Instead, we either find values with computers or consult a normal table

$P(b<Z<+\infty )=1-F_Z(b)$
$P(a\le Z\le b)=F_Z(b)-F_Z(a)$

Continuity correction deals with the fact that binomial random variables are discrete by improving our estimation with $P(a\le X\le b)={\int }_{a-0.5}^{b+0.5}f(y)dy=F_Z\left[\frac{b+0.5-np}{\sqrt{np(1-p)}}\right]-F_Z\left[\frac{a-0.5-np}{\sqrt{np(1-p)}}\right]$

As a general rule of thumb, this limit is best used when $n>9\frac{p}{1-p}$ and $n>9\frac{1-p}{p}$

Central Limit Theorem: Let $W_1,W_2,\dots$ be an infinite sequence of i.i.d. random variables, with finite $\mu$ and ${\sigma }^2$. For any $a$ and $b$,

$$\underset{n\to \infty }{\lim }P\left(a\le \frac{\sum W_i-n\mu }{\sqrt{n}\sigma }\le b\right)=\frac{1}{\sqrt{2\pi }}{\int }_a^b{e}^{-z^2/2}dz$$

This is also often written as ${\lim }_{n\to \infty }P\left(a\le \frac{\bar{W}-\mu }{\sigma /\sqrt{n}}\right)=\frac{1}{\sqrt{2\pi }}{\int }_a^b{e}^{-z^2/2}dz$ (where $\bar{W}$ is the average of $W_1,\dots ,W_n$).

The amazing thing is this theorem works for any sequence of i.i.d. random variables

Many individual random variables can also be approximated with a standard curve. One could view this as a sum over error factors. A random variable $Y$ is normally distributed if $f_Y(y)=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{1}{2}(\frac{y-\mu }{\sigma }{)}^2}$. One could also write $Y\sim N(\mu ,{\sigma }^2)$.

$M_Y(t)=e^{\mu t+({\sigma }^2{t}^2)/2}$

Using $M_Y(t)$, we find that the sum of two normal variables $N({\mu }_1,{\sigma }_1^2)$ and $N({\mu }_2,{\sigma }_2^2)$ is another normal variable $N({\mu }_1+{\mu }_2,{\sigma }_1^2+{\sigma }_2^2)$

Proof of the Central Limit Theorem

Proving the Central Limit Theorem in full generality is difficult, but a slightly weaker proof assuming moment-generating functions exist for $W_i$ is succinct

Lemma: If ${\lim }_{n\to \infty }{M}_{W_n}(t)=M_W(t)$ then ${\lim }_{n\to \infty }{F}_{W_n}(w)=F_W(w)$. I.e. if the moment generating functions match, then the CDFs match.

We must show ${\lim }_{n\to \infty }{M}_{(\Sigma W-n\mu )/(\sqrt{n}\sigma )}(t)=M_Z(t)=e^{t^2/2}$
We define $S_i=(W_i-\mu )/\sigma$
$M_{\Sigma S/\sqrt{n}}(t)={\left[M\left(\frac{t}{\sqrt{n}}\right)\right]}^n$

Taylor’s theorem gives us $M(t)=1+M^{\prime }(0)t+\frac{1}{2}{M}^{\prime \prime }(r)t^2=1+\frac{1}{2}{t}^2{M}^{\prime \prime }(r)$ where $|r|<|t|$ is some remainder

${\lim }_{n\to \infty }{\left[M\left(\frac{t}{\sqrt{n}}\right)\right]}^n={\lim }_{n\to \infty }{\left[1+\frac{t^2}{2n}{M}^{\prime \prime }(s)\right]}^n$, $|s|<\frac{|t|}{\sqrt{n}}$
$=\exp {\lim }_{n\to \infty }n\ln \left[1+\frac{t^2}{2n}{M}^{\prime \prime }(s)\right]$
$=\exp {\lim }_{n\to \infty }\frac{t^2}{2}{M}^{\prime \prime }(s)\frac{\ln \left[1+\frac{t^2}{2n}{M}^{\prime \prime }(s)\right]-\ln (1)}{\frac{t^2}{2n}{M}^{\prime \prime }(s)}$

The existence of $M(t)$ implies the existence of all derivatives, and therefore $M^{(i)}(t)$ is continuous for all $i$

Since $|s|<|t|/\sqrt{n}$, $s\to 0$ as $n\to \infty$, and ${\lim }_{n\to \infty }{M}^{\prime \prime }(s)=1$
So ${\lim }_{n\to \infty }{\left[M\left(\frac{t}{\sqrt{n}}\right)\right]}^n=\exp \frac{t^2}{2}\cdot 1\cdot {\ln }^{(1)}(1)=e^{t^2/2}$

Geometric Distribution

Consider a series of independent trials, each succeeding or failing. If $X$ is the trial at which the first success occurs, $p_X(k)=(1-p{)}^{k-1}p$.

This is called a geometric distribution,

• $M_X(t)=\frac{p{e}^t}{1-(1-p)e^t}$
• $E(X)=\frac{1}{p}$
• $\text{Var}(X)=\frac{1-p}{p^2}$

The negative binomial distribution generalizes this to waiting for the $r$th success. This is equivalent to saying there are $r-1$ successes in the first $k-1$ trials and a success on the $k$th trial,

$$p_X(k)=\left(\genfrac{}{}{0ex}{}{k-1}{r-1}\right)p^{r-1}(1-p^{k-1-(r-1)})\cdot p$$

$M_X(t)={\left[\frac{p{e}^t}{1-(1-p)e^t}\right]}^r$
$E(X)=\frac{1}{p}$
$\text{Var}(X)=\frac{r(1-p)}{p^2}$

$M_X(t)$, $E(X)$, $\text{Var}(X)$ all result immediately from linearity of expectation

Gamma Distribution

Theorem: Suppose that Poisson events occur with a rate of $\lambda$ per unit time. Let $Y$ denote the waiting time for the $r$th event,

$$f_Y(y)=\frac{{\lambda }^r}{(r-1)!}{y}^{r-1}{e}^{-\lambda y}$$

This implies that ${\int }_0^{\infty }{y}^{r-1}{e}^{-\lambda y}dy$ converges for any real number, which justifies considering the gamma function $\Gamma (r)={\int }_0^{\infty }{y}^{r-1}{e}^{-y}dy$

This is a funky function! $\Gamma (1)=1$, $\Gamma (r)=(r-1)\Gamma (r-1)$, so if $r$ is an integer then $\Gamma (r)=(r-1)!$

We extend our equation above and define the gamma pdf as

$$f_Y(y)=\frac{{\lambda }^r}{\Gamma (r)}{y}^{r-1}{e}^{-\lambda y}$$

We get $E(Y)=r/\lambda$ and $\text{Var}(Y)=r/{\lambda }^2$
$M_Y(t)=(1-t/\lambda {)}^{-r}$

Gamma random variables are additive, just like Poisson and Normal

The Gamma distribution is used in ${\chi }^2$ tests, based on the ${\chi }^2$ distribution, which is a special case of the gamma pdf where $\lambda =\frac{1}{2}$ and $r=\frac{m}{2}$