Random Variables and Vector Spaces

This document follows an explanation from LibreTexts

Many concepts have elegant interpretations if real-valued random variables are viewed as vectors in a vector space. Variance, covariance, and moments take on nice definitions.

A random experiment is modeled by a probability space $(\Omega ,F,P)$ where $\Omega$ is a set of outcomes, $F$ is the $\sigma$-algebra of events which defines the sample space in conjunction with $\Omega$ (we’ve looked at operations of sets), and $P$ is the probability measure on the sample space $(\Omega ,F)$

Our basic vector space $V$ consists of all real-valued random variables defined on $(\Omega ,F,P)$. Random variables $X_1$ and $X_2$ are equal if $P(X_1=X_2)=1$, so technically $V$ consists of equivalence classes under this relation (think how we can state the same random variable in many different ways).

We can define addition and scalar multiplication exactly how we’d expect, meaning $V$ is a well-defined vector space.

For brevity, I’m not going to be 100% precise with my notation here…

Definition: The k-norm of $X$ is defined as $\Vert X{\Vert }_k=[E[|X{|}^k]{]}^{1/k}$. This value measures the size of $X$ in a certain sense. We get a few properties very naturally,

1. $\Vert X{\Vert }_k\ge 0$
2. $\Vert X{\Vert }_k=0⟺P(X=0)=1$
3. $\Vert cX{\Vert }_k=|c|\Vert X{\Vert }_k$
4. $\Vert X+Y{\Vert }_k\le \Vert X{\Vert }_k+\Vert Y{\Vert }_k$

This is a nice well-behaved norm, so we denote ${\mathcal{L}}_k$ as the normed vector space of $X\in V$ with $\Vert X{\Vert }_k<\infty$ and with norm $\Vert \cdot {\Vert }_k$. In conventional notation, $p$ is used instead of $k$.

If $j\le k$ then $\Vert X{\Vert }_j\le \Vert X{\Vert }_k$, so we can say that ${\mathcal{L}}_k\subseteq {\mathcal{L}}_j$

Now that we have a well defined norm, we can make a nice distance function.

Definition: The k-metric between $X$ and $Y$ is defined as $d_k(X,Y)=\Vert X-Y{\Vert }_k$. It shares the norm’s properties, since it’s essentially the same thing.

This lets us define standard deviation pretty easily as $\Vert X-\mu {\Vert }_2$ and the variance as $\Vert X-\mu {\Vert }_2^2$

The root mean square error function is $\Vert X-t{\Vert }_2$. This function is minimized when $t=E[X]$

The mean absolute error is $\Vert X-t{\Vert }_1=E[|X-t|]$. This function is minimized when $t$ is any median of $X$. This point is confusing, and the text doesn’t explain it so well.

Once we have a measure of distance, we also have a measure of criterion for convergence. $X_n\to X$ as $n\to \infty$ if $d_k(X_n,X)\to 0$

When $k=1$, we say that $X_n$ approaches $X$ in mean and when $k=2$ we say in mean square.

Convergence in kth mean implies that the $k$ norms converge, but not vice versa. Meaning $\Vert X_n,X{\Vert }_k\to 0$ implies $|\Vert X_n{\Vert }_k-\Vert X{\Vert }_k|\to 0$

The text also states that convergence in mean is stronger than convergence in probability, however this term is not defined

${\mathcal{L}}_2$ is special because it’s the space where norm corresponds to an inner product,
Definition: The inner product of $X$ and $Y$ is defined as $⟨X,Y⟩=E(XY)$

We get our properties for inner product fairly easily,

1. $⟨X,Y⟩=⟨Y,X⟩$
2. $⟨X,X⟩\ge 0$ and $⟨X,X⟩=0⟺P(X=0)=1$
3. $⟨aX,Y⟩=a⟨X,Y⟩$
4. $⟨X+Y,Z⟩=⟨X,Z⟩+⟨Y,Z⟩$

We can define covariance correlation with this inner product,
Definition: $\text{Cov}(X,Y)=⟨X-E[X],Y-E[Y]⟩$
Definition: $\rho (X,Y)=⟨\frac{X-E[X]}{\Vert X{\Vert }_2},\frac{Y-E[Y]}{\Vert Y{\Vert }_2}⟩$

So $X$ and $Y$ are uncorrelated if the centered variables $X-E[X]$ and $Y-E[Y]$ are orthogonal in ${\mathcal{L}}_2$

We can see that the inner product corresponds to the 2-norm, since $⟨X,X⟩=E(X^2)=\Vert X{\Vert }_2^2$, which also corresponds to the root mean square

We call ${\mathcal{L}}_2$ a Hilbert space

Theorem: The Cauchy-Schwarz inequality states that $E[|X||Y|]\le \sqrt{E[X^2]}\sqrt{E[Y^2]}$, or in more familiar terms, $⟨X,Y⟩\le \Vert X{\Vert }_2\Vert Y{\Vert }_2$

We can also write this like $|\text{Cov}(X,Y)|\le \text{sd}(X)\text{sd}(Y)$

Discussions of best linear predictors work nicely when described as projections onto subspaces. Say $U\subseteq V$, then the projection of $X$ onto $U$ is the vector $v$ $⟨X-v,U⟩=0$

This projection (if it exists) is unique. Furthermore, $\Vert X-v{\Vert }_2^2\le \Vert X-u{\Vert }_2^2$ for all $u\in U$. This implies $v$ is minimizing a distance.

We say that $W_X=\{a+bX:a\in \mathbb{R},b\in \mathbb{R}\}$ is a subspace of ${\mathcal{L}}_2$, the subspace generated by $X$ and $1$ (linear combinations of these two vectors).

We write $X_0=X-E[X]=X-⟨X,1⟩$ which is perpendicular to $1$. Then we can say that the best linear predictor of $Y$ given $X$ is the projection of $Y$ onto $W_X$, which is spanned by $1$ and $X_0$,
So $L(Y|X)=⟨Y,1⟩1+\frac{⟨Y,X_0⟩}{⟨X_0,X_0⟩}{X}_0$

Or in the language of random variables,
$L(Y|X)=E[Y]+\frac{\text{Cov}(X,Y)}{\text{Var}(X)}(X-E[X])$