Forecasting

Forecasting is predicting $X_{T+h}$ given available information $D_t\equiv \{X_1,\dots ,X_T\}$ for $h=1,2,\dots$

We have several sources of error, like parameter uncertainty, model uncertainty, and ignorance about future errors

Our 1-step ahead forecast is defined as ${\hat{X}}_{T+1}\equiv g(D_T)$
How do we get an optimal forecast? In practice, we minimize the mean squared error $\mathbb{E}[X_{T+1}-{\hat{X}}_{T+1}{]}^2$

We propose this will look like ${\hat{X}}_{T+1}=\mathbb{E}[X_{T+1}\mid D_T]$
We can do some algebra with expectations to show that $g(D_T)=\mathbb{E}[X_{T+1}\mid D_T]$ is a lower bound for any forecaster (with regards to MSE)

The same result holds for the h-step ahead forecast for general $h\ge 1$, which is ${\hat{X}}_{T+h}=\mathbb{E}[X_{T+h}\mid D_T]$

Example:
For an $AR(1)$ process, we get
${\hat{X}}_{T+1}=\varphi X_T$

$e_{T+1}\equiv X_{T+1}-{\hat{X}}_{T+1}=X_{T+1}-\varphi X_T={ϵ}_{T+1}$
$\mathbb{E}[e_{T+1}]=0$
$\text{Var}[e_{T+1}]={\sigma }_{ϵ}^2$

${\hat{X}}_{T+h}={\varphi }^h{X}_T+{\sum }_{j=0}^{h-1}{\varphi }^j{ϵ}_{T+h-j}$
$e_{T+h}\equiv X_{T+h}-{\hat{X}}_{T+h}={\sum }_{j=0}^{h-1}{\varphi }^j{ϵ}_{T+h-j}$
$\mathbb{E}[e_{T+h}]=0$
$\text{Var}[{ϵ}_{T+h}]=h{\sigma }_{ϵ}^2$

If $ϵ$ is normal then our full error term $e_{T+h}$ is also normally distributed, which means we can build confidence intervals

With an intercept, our $AR(1)$ forecasts look like,
${\hat{X}}_{T+1}=\alpha +\varphi X_T+{ϵ}_T$
${\hat{X}}_{T+2}=\alpha +\varphi \alpha +{\varphi }^2{X}_T$
${\hat{X}}_{T+h}={\varphi }^h{X}_T+\alpha {\sum }_{j=0}^{h-1}{\varphi }^j$

For $AR(p)$, we can derive
${\hat{X}}_{T+h}={\varphi }_1{\hat{X}}_{T+h-1}+\cdots +{\varphi }_p{\hat{X}}_{T+h-p}$

Example:
Let $\{X_t\}$ follow $X_t={ϵ}_t+\theta {ϵ}_{t-1}$
${\hat{X}}_{T+1}=\theta \cdot \mathbb{E}({ϵ}_t\mid D_T)=\theta {ϵ}_T$ (assuming ${ϵ}_0=0$)
$e_{T+1}={ϵ}_{T+1}$

${\hat{X}}_{T+2}=0$
$e_{T+2}={ϵ}_{T+2}+\theta {ϵ}_{T+1}$

${\hat{X}}_{T+h}=0$
$e_{T+h}={ϵ}_{T+h}+\theta {ϵ}_{T+h-1}$

Definition: The least squares estimator of a parameter minimizes the sum of squared residuals

Suppose we have a sample of size $T$ from an $AR(p)$ process,
$y_t=\alpha +{\sum }_{i=1}^p{\varphi }_i{y}_{t-i}+{ϵ}_t$
$y_t=x_t^{\prime }\beta +{ϵ}_t$
with $x_t=(1,y_{t-1},\dots ,y_{t-p}{)}^{\prime }$ and $\beta =(\alpha ,{\varphi }_1,\dots ,{\varphi }_p{)}^{\prime }$

In matrix form, we can write $y=X\beta +ϵ$

$\hat{\beta }={\min }_{\beta }{\sum }_{t=p+1}^T(y_t-x_t^{\prime }\beta {)}^2={\min }_{\beta }(y-X\beta {)}^{\prime }(y-X\beta )$
$\frac{\partial (y-X\beta {)}^{\prime }(y-X\beta )}{\partial \beta }=-2X^{\prime }y+2X^{\prime }X\beta =0$
$\hat{\beta }=(X^{\prime }X{)}^{-1}{X}^{\prime }y$

For $AR(1)$, this reduces to $\hat{\beta }=\hat{\varphi }=\frac{{\sum }_{t=2}^T{y}_t{y}_{t-1}}{{\sum }_{t=2}^T{y}_{t-1}^2}$

With some reasonable assumptions, it can be shown that $\sqrt{T}(\hat{\beta }-\beta )\to N(0,V)$ as $T\to \infty$, with asymptotic variance $V={\sigma }_{ϵ}^2(\mathbb{E}[x_t{x}_t^{\prime }]{)}^{-1}$

• $\{{ϵ}_t\}\sim NID(0,{\sigma }_{ϵ}^2)$ with ${\sigma }_{ϵ}^2>0$
• $\{y_t\}$ is stationary

For $AR(1)$, this reduces to $V=1-{\varphi }^2$

This means this is a robust estimator!

For large $T$, we have $\hat{\beta }\sim N(\beta ,V/T)$

$\hat{V}\equiv s^2{S}_{xx}^{-1}$ where $s^2\equiv \frac{1}{T-p}{\sum }_{t=p+1}^T{e}_t^2$ and $S_{xx}\equiv \frac{1}{T-p}{\sum }_{t=p+1}^T{x}_t{x}_t^{\prime }$

Definition: For a sample ${\mathbf{x}}_t\equiv (x_1,\dots ,x_T)$ with joint density $f({\mathbf{x}}_T,\psi )$, the maximum likelihood estimator is ${\hat{\psi }}_T=\arg {\max }_{\psi }f({\mathbf{x}}_T;\psi )$

In practice, we maximize the log likelihood
$f(x_1,\dots ,x_T;\psi )=f(x_1;\psi )\times {\prod }_{t=2}^{T}f(x_t\mid x_{t-1},\dots ,x_1;\psi )$
$\log f(x_1,\dots ,x_T;\psi )=\log f(x_1;\psi )+{\sum }_{t=2}^T\log f(x_t\mid x_{t-1},\dots ,x_1;\psi )$

This writes a $T$-dimensional joint density as a sum of $T$ univariate densities

Example:
Now take $X_t=\varphi X_{t-1}+{ϵ}_t$

${\hat{\psi }}_T=\arg {\max }_{\psi }f({\mathbf{x}}_T;\psi )={\max }_{\psi }{\prod }_{t=2}^{T}f(x_t\mid x_{t-1},\dots ,x_1;\psi )$
$=\arg {\max }_{\psi }{\prod }_{t=2}^T\frac{1}{\sqrt{2\pi {\sigma }_{ϵ}^2}}\exp \left[-\frac{(x_t-\varphi x_{t-1}{)}^2}{2{\sigma }_{ϵ}^2}\right]$
$=\arg {\max }_{\psi }{\sum }_{t=2}^T\left(-\log \sqrt{2\pi {\sigma }_{ϵ}^2}-\frac{(x_t-\varphi x_{t-1}{)}^2}{2{\sigma }_{ϵ}^2}\right)$

We then derive this by setting the derivative to 0

For any ${\sigma }_{ϵ}^2>0$, the value of $\varphi$ that maximizes the above function is the one that minimizes ${\sum }_{t=2}^T(x_t-\varphi x_{t-1}{)}^2$. So effectively, we’ve reached the LS estimator!

For ARMA models, we use a similar approach to derive a ML estimator

Example:
Consider $X_t=\alpha +{\varphi }_1{X}_{t-1}+\cdots +{\varphi }_p{X}_{t-p}+{ϵ}_t$
We get $X_t\mid X_{t-1},\dots ,X_{t-p}\sim N(\alpha +{\varphi }_1{X}_{t-1}+\cdots +{\varphi }_p{X}_{t-p},{\sigma }_{ϵ}^2)$
${\hat{\psi }}_t=\arg {\max }_{\psi }{\sum }_{t=p+1}^T\left(-\log \sqrt{2\pi {\sigma }_{ϵ}^2}-\frac{(x_t-\alpha -{\varphi }_1{x}_{t-1}-\cdots -{\varphi }_p{x}_{t-p}{)}^2}{2{\sigma }_{ϵ}^2}\right)$
Again, the ML estimate of ${\varphi }_1,\dots ,{\varphi }_p$ equals the LS estimate, essentially because ${ϵ}_t$ are assumed to be normal

Example:
Consider $X_t=\alpha +{ϵ}_t+{\theta }_1{ϵ}_{t-1}+\cdots +{\theta }_q{ϵ}_{t-q}$ with $\{{ϵ}_t\}\sim \text{NID}(0,{\sigma }_{ϵ}^2)$

Assume ${ϵ}_{t\le 0}=0$
$\mathbb{E}[{ϵ}_k\mid X_1,\dots ,X_{t-1}]={ϵ}_k$ for $k\le t-1$ and $0$ for $k>t-1$

So we get $X_t\mid X_{t-1},\dots ,X_1\sim N(\alpha +{\theta }_1{ϵ}_{t-1}+\cdots +{\theta }_q{ϵ}_{t-q},{\sigma }_{ϵ}^2)$

Model Selection

How do we select parameters?

If we select $p$ too large then we risk higher variance for our model

If we select $p$ too small then our model cannot possibly capture all the data’s nuances

We can strike a balance by minimizing an information criterion,
$T\cdot \log (SS{R}_k/T)+kC(T)$

• $SS{R}_k$ is the sum of squared residuals based on $k$ parameters
• For $AR(p)$ with intercept, $k=p+1$
• $C(T)=2$ is the Akaike information criterion and $C(T)=\log (T)$ is the Bayesian information criterion

We can use this by choosing $p_{\text{max}}$ and testing each model

These can also be computed with the log likelihood $kC(T)-2\log f({\mathbf{x}}_T)$

As the sample size goes to infinity BIC correctly estimates the true order, but AIC can outperform in finite samples