ARMA Models

We start with some important results on stationarity

Weak stationarity is important because weakly stationary processes are much easier to estimate and make calculations on

Theorem: Let $\{Y_t\}$ be weaky stationary with ${\mu }_Y$ and ${\gamma }_Y(h)$. If ${\sum }_{j=0}^{\infty }|{\psi }_j|<\infty$ then $X_t={\sum }_{j=0}^{\infty }{\psi }_j{Y}_{t-j}$ is also weakly stationary

• $\mathbb{E}[X_t]={\mu }_Y{\sum }_{j=0}^{\infty }{\psi }_j$ and
• ${\gamma }_X(h)={\sum }_{i=0}^{\infty }{\sum }_{j=0}^{\infty }{\psi }_i{\psi }_j{\gamma }_Y(h+j-i)$.

This is a powerful tool that we will use

MA(q)

Definition: The MA(q) model is defined as $X_t=\alpha +{ϵ}_t+{\theta }_1{ϵ}_{t-1}+\cdots +{\theta }_q{ϵ}_{t-q}$

In lag polynomial notation, we write $X_t=\alpha +\theta (L){ϵ}_t$ where $\theta (L)=1+{\theta }_{1}L+\cdots +{\theta }_q{L}^q$

We can see that $MA(q)$ is weakly stationary

$\gamma (h)={\sigma }_{ϵ}^2({\theta }_0{\theta }_h+{\theta }_1{\theta }_{h+1}+\cdots +{\theta }_{q-h}{\theta }_q)$ if $h\le q$ and $0$ otherwise
$\gamma (0)={\sigma }_{ϵ}^2(1+{\theta }_1^2+\cdots +{\theta }_q^2)$

$\rho (h)=\frac{\gamma (h)}{\gamma (0)}$, i.e. the ACF is zero after $q$ lags
We can use this to determine the order $q$ from a graph of the ACF

AR(p)

Definition: The AR(p) model is defined as $X_t=\alpha +{\varphi }_1{X}_{t-1}+\cdots +{\varphi }_p{X}_{t-p}+{ϵ}_t$

Using lag polynomial notation $\varphi (L)X_t=Y_t$ where $Y_t\equiv \alpha +{ϵ}_t$

Recall that the stability condition for $AR(1)$ is $|{\varphi }_1|<1$
Our condition is a bit more complicated for $AR(p)$

Definition: Let $\{X_t\}$ follow an $AR(p)$ process, $\{X_t\}$ is called stable if all roots $z_i^{\ast }:\varphi (z_i^{\ast })=0$ satisfy $|z_i^{\ast }|>1$

$\varphi (z)=1-{\varphi }_{1}z-\cdots -{\varphi }_p{z}^p$ has $p$ (possibly complex) roots by the Fundamental Theorem of Algebra

Theorem: A stable $AR(p)$ process is weakly stationary

We can factorize $\varphi (z)=1-{\sum }_{i=1}^p{\varphi }_i{z}^i$ into ${\prod }_{i=1}^p(1-{\lambda }_{i}z)$ for suitable values ${\lambda }_i$ which implies the stability condition holds when $|{\lambda }_i|<1$

We get the inverse lag polynomial is ${\varphi }^{-1}(L)={\prod }_{i=1}^p(1-{\lambda }_{i}L{)}^{-1}$

We use our lemma from earlier to show $(1-{\lambda }_{i}L{)}^{-1}={\sum }_{j=0}^{\infty }({\lambda }_{i}L{)}^j$ if $|{\lambda }_i|<1$

We apply ${\varphi }^{-1}(L)$ to $AR(p)$ to get $X_t={\prod }_{i=1}^p(1-{\lambda }_{i}L{)}^{-1}\cdot Y_t$

Each application of $(1-{\lambda }_{i}L{)}^{-1}$ on a weakly stationary process results in a weakly stationary process, so we conclude with our theorem

It can be shown that we can decompose $AR(p)$ into $X_t={\sum }_{j=0}^{\infty }{\psi }_j{ϵ}_{t-j}+{\mu }_X$

This implies $\mathbb{C}\text{ov}[X_{t-h},{ϵ}_t]=0$ for $h>0$ and $\text{Cov}[X_t,{ϵ}_t]={\sigma }_{ϵ}^2$ for $h=0$

We can derive recursive definitions for auto-covariances,
$\gamma (h)={\varphi }_1\gamma (h-1)+\cdots +{\varphi }_p\gamma (h-p)$ for $h\ge 1$
$\gamma (0)={\sigma }_{ϵ}^2+{\varphi }_1\gamma (1)+\cdots +{\varphi }_p\gamma (p)$

However this still requires solving the initial system of the first $p$ auto-covariances, which we call the Yule-Walker equations

$\left(\begin{array}{c}\rho (1)\\ \rho (2)\\ ⋮\\ \rho (p)\end{array}\right)=\left(\begin{array}{cccc}\rho (0)& \rho (1)& \cdots & \rho (p-1)\\ \rho (1)& \rho (0)& \cdots & \rho (p-2)\\ ⋮& ⋮& \ddots & ⋮\\ \rho (p-1)& \rho (p-2)& \cdots & \rho (0)\end{array}\right)\left(\begin{array}{c}{\varphi }_1\\ {\varphi }_2\\ ⋮\\ {\varphi }_p\end{array}\right)$

We can calculate estimates for ${\varphi }_1,\dots ,{\varphi }_p$ if we have sample autocorrelations for $\rho (1),\dots ,\rho (p)$

Or we can solve for the auto-correlations given the process parameters

ARMA(p, q)

Theorem: Let $\{X_t\}$ be weakly stationary, then we can write $X_t={\sum }_{j=0}^{\infty }{\varphi }_j{ϵ}_{t-j}+V_t$ where ${\varphi }_0=1$ and ${\sum }_{j=0}^{\infty }{\varphi }_j^2<\infty$, ${ϵ}_t$ is white-noise, $\text{Cov}({ϵ}_s,V_t)=0$ for $\forall s,t$, and $V_t$ is perfectly predictable from its past values (for many models $V_t=\mathbb{E}[X_t]$. This is called the Wold decomposition.

We already did this for the $AR(1)$ process, $X_t={\sum }_{j=0}^{\infty }{\varphi }_1^j{ϵ}_{t-j}$

Notice that the Wold representation is essentially an MA($\infty$) model ($V_t=\alpha$)
The $<\infty$ condition means ${\lim }_{j\to \infty }{\varphi }_j=0$, meaning a weakly stationary process can be approximated by an MA(q) process!

The ARMA model is motivated as a way to reduce $j$ necessary for a good approximation

Definition: The autoregressive moving average model of order $(p,q)$ is defined as $X_t=\alpha +{\sum }_{i=1}^p{\varphi }_i{X}_{t-i}+{ϵ}_t+{\sum }_{j=1}^q{\theta }_j{ϵ}_{t-j}$ with $\{{ϵ}_t\}\sim WN(0,{\sigma }_{ϵ}^2)$

Using lag polynomials (with $\alpha =0$), $\varphi (L)=1-{\sum }_{i=1}^p{\varphi }_i{L}^i$, and $\theta (L)=1+{\sum }_{j=1}^q{\theta }_j{L}^j$
$\varphi (L)X_t=\theta (L){ϵ}_t$

This model is weakly stationary if the $AR(p)$ portion is stable, in which case $X_t=\frac{\theta (L)}{\varphi (L)}{ϵ}_t$

The Wold Decomposition is $X_t={\sum }_{j=0}^{\infty }{\psi }_j{ϵ}_{t-j}$ with ${\psi }_0=1$ and ${\sum }_{j=0}^{\infty }|{\psi }_j|<\infty$

Getting the actual $\psi$ coefficients is a bit more involved (but we’d start by looking at $\psi (L)=\theta (L)/\varphi (L)$)

In other words, a stable $ARMA(p,q)$ model (also known as causal) can be approximated by an $MA$ model

Can we also approximate it with an $AR$ model? Let’s examine $MA(1)$

$X_t={ϵ}_t+{\theta }_1{ϵ}_{t-1}=\theta (L){ϵ}_t$
If $|{\theta }_1|<1$, then we call this process invertible (by the same logic we’ve used previously),
$(1+{\theta }_{1}L{)}^{-1}{X}_t={\sum }_{j=0}^{\infty }(-{\theta }_1{)}^j{X}_{t-j}$
${ϵ}_t=X_t+{\sum }_{j=1}^{\infty }(-{\theta }_1{)}^j{X}_{t-j}$
$X_t=ϵ-{\sum }_{j=1}^{\infty }(-{\theta }_1{)}^j{X}_{t-j}$ Yes!

Extending to the general case, the $MA$ polynomial $\theta (L)$ is invertible if all roots $z^{\ast }:\theta (z^{\ast })=0$ satisfy $|z^{\ast }|>1$

So invertibility implies that the $ARMA$ can be approximated by an $AR$ model

Invertibility also implies we can determine the coefficients that made up the time series at time $t$