Time Series

Definition: A time series is a sequence of random variables $X_t$ ordered by time, denoted as $\{X_t\}$

Definition: A sample of size $T$ from a time series is a realized path of the time series $x_1,x_2,x_3,\dots ,x_T$

Definition: A time series model for a sample $\{x_t{\}}_{t=1}^T$ is a set of joint distributions for the corresponding time series $\{X_t{\}}_{t=1}^T$

We often omit the interval in our notation, sometimes assuming the series is infinite
We often focus only on the first two moments of the data (means, variances, covariances)

The simplest model is that of independent and identically distributed points, denoted as $\{X_t\}\sim IID({\mu }_X,{\sigma }_X^2)$

We can be more specific and specify the distribution $X_t\sim N({\mu }_X,{\sigma }_X^2)$, in which case we denote the series as $\{X_t\}\sim NID({\mu }_X,{\sigma }_X^2)$

In real life, we expect dependence in time series data between nearby points, however IID sequences are still important components of more realistic data

Definition: A white noise process, denoted $\{X_t\}\sim WN(0,{\sigma }_X^2)$, is a sequence of uncorrelated random variables with zero mean and constant variance ${\sigma }_X^2<\infty$

This is more general than the IID conditions (with mean 0)

Example:
Consider $X_t=Z_t\sim N(0,1)$ if $t$ is odd and $X_t=\frac{1}{c}(X_{t-1}^2-1)$ where $c=\sqrt{\text{Var}(X_{t-1}^2-1)}$ if $t$ is even

$\mathbb{E}[X_2]=\frac{1}{c}\mathbb{E}[X_1^2-1]=\frac{1}{c}\mathbb{E}[{\chi }_1-1]=0$
$\text{Cov}[X_2,X_1]=\mathbb{E}[(X_2-\mathbb{E}[X_2])(X_1-\mathbb{E}[X_1])]=\mathbb{E}[X_2{X}_1]=\frac{1}{c}(\mathbb{E}[X_1^3]-\mathbb{E}[X_1])=0$

$E[X_1^3]$ is the skewness, which is 0 for a normal variable (it’s symmetric)

We can also see that variance is constant, $\text{Var}(X_t)=1$ ($c$ accomplishes this on the even terms)

For a general time series, we define functions for the first two moments,

• ${\mu }_X(t)=\mathbb{E}(X_t)$
• ${\sigma }_X^2(t)=\text{Var}(X_t)$
• ${\gamma }_X(t,t+h)=\text{Cov}(X_t,X_{t+h})$

An important classification is whether these first moments are time-invariant (dependent on $t$)

Definition: A time series $\{X_t\}$ is weakly stationary (or covariance stationary) if the mean and autocovariance are constant in $t$, and the variance is finite.

1. ${\mu }_X(t)={\mu }_X$
2. ${\gamma }_X(t,t+h)={\gamma }_X(h)$, which implies ${\sigma }_X^2(t)=\gamma (0)$

$\{X_t\}\sim WN(0,{\sigma }_X^2)$ is weakly stationary

This is a powerful condition
$\text{Cov}(X_t,X_{t+h})=\text{Cov}(X_{t-h},X_t)=\text{Cov}(X_t,X_{t-h})$

We can easily define estimation functions for these moments for a given sample

Definition: The autocorrelation function (ACF) for a weakly stationary function is ${\rho }_X(h)=\frac{{\gamma }_X(h)}{{\gamma }_0(h)}$
This is analogous to how we scale covariance to obtain the correlation coefficient

Of course, plenty of time series are not weakly stationary! Consider $X_t=t+Y_t$
$\mathbb{E}[X_t]=t+{\mu }_Y$ is not constant in $t$

Definition: Trend-stationary processes combine a stationary process with a deterministic trend (such as $t$)
These are non-stationary

Definition: A random walk is a time series $\{X_t\}$ defined as $X_t={ϵ}_1+{ϵ}_2+\cdots +{ϵ}_t$ where $\{{ϵ}_t\}$ is a white noise process
This is often written as $X_t=X_{t-1}+{ϵ}_t$ with $X_1={ϵ}_1$
This is not weakly stationary because the variance increases with $t$

This is said to have a stochastic trend which will be discussed more later

Definition: The autoregressive model of order one, AR(1), is defined by the process $X_t=\varphi X_{t-1}+{ϵ}_t$, where $\{{ϵ}_t\}\sim WN(0,{\sigma }_{ϵ}^2)$

This is probably the most widely used time series in practice

We can also write this in closed form, $X_t={\sum }_{i=0}^{t-1}{\varphi }^i{ϵ}_{t-i}$
$X_t={\sum }_{i=0}^{\infty }{\varphi }^i{ϵ}_{t-i}$ considers all indices over $t\in \mathbb{Z}$

If $\varphi =1$ (random walk), the process is non-stationary

An AR(1) process with $|\varphi |<1$ is called stable
Assuming a stable process is stationary, we have ${\mu }_X=\varphi {\mu }_X$, which means ${\mu }_X=0$

Using $\text{Cov}(X_{t-1},{ϵ}_t)=0$, we can derive ${\sigma }_X^2=\frac{{\sigma }_{ϵ}^2}{1-{\varphi }^2}$

Exercise: Show ${\gamma }_X(h)={\varphi }^h{\gamma }_X(0)$
${\gamma }_X(1)=\text{Cov}(X_t,X_{t-1})=\text{Cov}(\varphi X_{t-1}+{ϵ}_t,X_{t-1})=\varphi \text{Cov}(X_{t-1},X_{t-1})+\text{Cov}({ϵ}_t,X_{t-1})=\varphi {\sigma }_X^2=\varphi {\gamma }_X(0)$
${\gamma }_X(2)=\text{Cov}(X_t,X_{t-2})=\text{Cov}({\varphi }^2{X}_{t-2}+\varphi {ϵ}_{t-1}+{ϵ}_t,X_{t-2})=\cdots ={\varphi }^2\text{Cov}(X_{t-2},X_{t-2})={\varphi }^2{\gamma }_X(0)$
We can also solve this more precisely, or write out a recursive formula ${\gamma }_X(h)=\varphi {\gamma }_X(h-1)$

It follows that ${\rho }_X(h)={\varphi }^h$
This reveals that $\varphi$ directly controls the decay of correlation as $h$ increases

Definition: The lag operator $L$ maps $X_t$ to $X_{t-1}$, $L{X}_t=X_{t-1}$

• $L(X_t+Y_t)=L{X}_t+L{Y}_t$
• $L^2{X}_t\equiv LL{X}_t=X_{t-2}$
• $L^h{X}_t\equiv X_{t-h}$

Definition: The difference operator is defined as $\Delta =1-L$, $\Delta X_t=X_t-X_{t-1}$

• ${\Delta }^2\equiv \Delta \Delta =(1-L)(1-L)=1-2L+L^2$
• ${\Delta }^2{X}_t=X_t-2X_{t-1}+X_{t-2}$

The difference operator plays an important role in detrending data

Example: Let $X_t=t+{ϵ}_t$ with $ϵ\sim WN(0,{\sigma }_{ϵ}^2)$
$\Delta X_t=X_t-X_{t-1}=(t+{ϵ}_t)-[(t-1)+{ϵ}_{t-1}]=1+\Delta {ϵ}_t$

Is this weakly stationary?

$\mathbb{E}[\Delta X_t]=1+\mathbb{E}[{ϵ}_t-{ϵ}_{t-1}]=1+\mathbb{E}[{ϵ}_t]-\mathbb{E}[{ϵ}_{t-1}]=1$ Yes

$\text{Cov}(\Delta X_t,\Delta X_{t-h})=\text{Cov}(1+\Delta {ϵ}_t,1+\Delta {ϵ}_{t-h})=\text{Cov}({ϵ}_t-{ϵ}_{t-1},{ϵ}_{t-h}-{ϵ}_{t-h-1})$
$=\text{Cov}({ϵ}_t,{ϵ}_{t-h})-\text{Cov}({ϵ}_t,{ϵ}_{t-h-1})-\text{Cov}({ϵ}_{t-1},{ϵ}_{t-h})+\text{Cov}({ϵ}_{t-1},{ϵ}_{t-h-1})$
If $h=0$, this is $2{\sigma }_{ϵ}^2$
If $h=1$, this is $-{\sigma }_{ϵ}^2$
If $h>1$, this is 0
So yes

In general, we can take ${\Delta }^k$ to detrend a polynomial trend of degree $k$
The approach of differencing until a time series is stationary is called the Box-Jenkins approach

Differencing is also useful for stochastic trends, like the random walk $X_t=X_{t-1}+{ϵ}_t$
$\Delta X_t={ϵ}_t$ is weakly stationary

Definition: The first-order lag polynomial is defined as $1-\varphi L$, $\varphi \in \mathbb{R}$
This lets us write AR(1) as $(1-\varphi L)X_t={ϵ}_t$
Why would we do this?

If the inverse of this operator exists, we can now say $X_t=(1-\varphi L{)}^{-1}{ϵ}_t$
This is more abstract but looks interesting

Lemma: Let $|\varphi |<1$ and suppose $\{Z_t\}$ is weakly stationary. Then $(1-\varphi L{)}^{-1}{Z}_t={\sum }_{i=0}^{\infty }{\varphi }^i{Z}_{t-i}$ where the equality is in mean square.

Definition: Two random variables $X$ and $Y$ are equal in mean square if $\mathbb{E}[(X-Y{)}^2]=0$

This lemma allows us to easily derive our earlier expression $X_t={\sum }_{i=0}^{\infty }{\varphi }^i{ϵ}_{t-i}$