Joint Probability

Review the CDF of $X$ is $F_X(x)=P(X\le x)={\int }_{-\infty }^x{f}_X(t)dt$
$\frac{d}{dx}{F}_X(x)=f_x(x)$

Say we have two random variables $X$ and $Y$ which are dependent on each other. The joint PDF is a two variable function $f_{X,Y}(x,y)$.

In this analog, our CDF is $F_{X,Y}(x,y)=P(X\le x,Y\le y)={\iint }_C{f}_{X,Y}(x,y)dA$
$={\int }_{-\infty }^x{\int }_{-\infty }^y{f}_{X,Y}(t,w)dwdt$
And likewise, $f_{X,Y}(x,y)=\frac{{\partial }^2}{\partial x\partial y}{F}_{X,Y}(x,y)$

Voila!

If $X$ and $Y$ are independent, $f_{X,Y}(x,y)=f_X(x)f_Y(y)$

When $X,Y$ are discrete, we define $p_{X,Y}(a,b)=P(X=a,Y=b)$
It’s easy to see that $p_{X,Y}(a,b)=p_X(a)\cdot p_Y(b)$ if $X,Y$ are independent

We can see that $F_{X+Y}(a)=P\{X+Y\le a\}={\iint }_{x+y\le a}{f}_X(x)f_Y(y)dxdy$
$={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{a-y}{f}_X(x)f_Y(y)dxdy={\int }_{-\infty }^{\infty }{F}_X(a-y)f_Y(y)dy$ which is often called the convolution of $F_X$ and $F_Y$ (which are independent)

The probability density function in this case is $f_{X+Y}(a)={\int }_{-\infty }^{\infty }{f}_X(a-y)f_Y(y)dy$

This equation is a bit confusing but has wide applications.

Conditional Joint Probabilities

Recall that we have $P(E|F)=\frac{P(EF)}{P(F)}$. If $X$ and $Y$ are discrete random variables, it is natural to define $p_{X|Y}(x|y)=P\{X=x|Y=y\}=\frac{P\{X=x,Y=y\}}{P\{Y=y\}}=\frac{p(x,y)}{p_Y(y)}$

We might also define the distribution function as $F_{X|Y}(x|y)=P\{X\le x|Y=y\}={\sum }_{a\le x}^{}{p}_{X|Y}(a|y)$. This works well; if $X$ and $Y$ are independent then it simplifies to the unconditional cases.

If $X$ and $Y$ are continuous, we define things similarly, $f_{X|Y}(x|y)=\frac{f(x,y)}{f_Y(y)}$

$F_{X|Y}(a|y)=P\{X\le a|Y=y\}={\int }_{-\infty }^a{f}_{X|Y}(x|y)dx$

The bivariate normal distribution is defined with constants ${\mu }_x,{\mu }_y,{\sigma }_x>0,{\sigma }_y>0,-1<\rho <1$ and density function $f(x,y)=\frac{1}{2\pi {\sigma }_x{\sigma }_y\sqrt{1-{\rho }^2}}\exp \left\{-\frac{1}{2(1-{\rho }^2)}\left[{\left(\frac{x-{\mu }_x}{{\sigma }_x}\right)}^2+{\left(\frac{y-{\mu }_y}{{\sigma }_y}\right)}^2-2\rho \frac{(x-{\mu }_x)(y-{\mu }_y)}{{\sigma }_x{\sigma }_y}\right]\right\}$
Now to determine the density of $X$ given $Y=y$, we collect factors not depending on $x$ into constants.

$f_{X|Y}(x|y)=\frac{f(x,y)}{f_Y(y)}$
$=C_{1}f(x,y)$
$=C_2\exp \left\{-\frac{1}{2(1-{\rho }^2)}\left[{\left(\frac{x-{\mu }_x}{{\sigma }_x}\right)}^2-2\rho \frac{x(y-{\mu }_y)}{{\sigma }_x{\sigma }_y}\right]\right\}$
$=C_3\exp \left\{-\frac{1}{2{\sigma }_x^2(1-{\rho }^2)}\left[x^2-2x\left({\mu }_x+\rho \frac{{\sigma }_x}{{\sigma }_y}(y-{\mu }_y)\right)\right]\right\}$
$=C_4\exp \left\{-\frac{1}{2{\sigma }_x^2(1-{\rho }^2)}{\left[x-\left({\mu }_x+\rho \frac{{\sigma }_x}{{\sigma }_y}(y-{\mu }_y)\right)\right]}^2\right\}$

This is now a normal density with mean $u_x=\rho \frac{{\sigma }_x}{{\sigma }_y}(y-{\mu }_y)$ and variance ${\sigma }_x^2(1-{\rho }^2)$. We see that $X$ and $Y$ are independent when $\rho =0$, as expected.

Order Statistics

Let $X_1,X_2,\dots ,X_n$ be $n$ independent and identically distributed continuous random variables. Define $X_{(1)}$ as the smallest of them, $X_{(i)}$ as the ith smallest of them.

$X_{(1)}\le X_{(2)}\le \cdots \le X_{(n)}$ are known as the order statistics of the random variables.

We can see that $f_{X_{(1)},\dots ,X_{(n)}}(x_1,x_2,\dots ,x_n)=n!f(x_1)\dots f(x_n)$ for $x_1<x_2<\cdots <x_n$, essentially since it is sufficient for $X_1,\dots ,X_n$ to equal any permutation of $x_1,\dots ,x_n$.