%% This document created by Scientific Word (R) Version 2.5


\documentclass{sebase}
\usepackage{amssymb}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{SESLIDEA2}

%TCIDATA{TCIstyle=misc/misc.lat,SESLIDEA2,SESLIDEA2}

%TCIDATA{Created=Sat Jan 11 12:06:00 2003}
%TCIDATA{LastRevised=Sat Feb 08 13:50:07 2003}

\input{tcilatex}
\begin{document}


\textit{EC821:\ Time Series Econometrics, Spring 2003}

\bigskip \textit{Notes Section 3 }

\textbf{ARMA models}

We have spoken of a univariate white noise process, $\epsilon _{t},$ a
zero--mean covariance stationary process with no serial correlation: 
\[
E\left[ \epsilon _{t}\right] =0,E\left[ \epsilon _{t}^{2}\right] =0<\sigma
^{2}<\infty ,~E\left[ \epsilon _{t}\epsilon _{t-j}\right] =0\curlyvee j\neq
0. 
\]

A very important class of covariance--stationary processes, called \textbf{%
linear processes}, can be created by taking a \textbf{moving average} of a
white noise process. Let us consider that the white noise process is defined
for all integers, $\{t=0,\frac{+} {}1,\frac{+} {}2,...\}$ so that we may
assume that the process started in the distant past, and its mean and
autocovariance function have stabilized to time--invariant constants.

The simplest linear process that exhibits serial correlation is a
finite--order moving average process. $\{y_{t}\}$ is said to be a \textbf{q}$%
^{th}$\textbf{-order moving average }process, $MA(q),$ if it can be written
as a weighted average of the current and most recent $q$ values of a white
noise process: 
\begin{equation}
y_{t}=\mu +\theta _{0}\epsilon _{t}+\theta _{1}\epsilon _{t-1}+....+\theta
_{q}\epsilon _{t-q},\theta _{0}=1.  \label{maq}
\end{equation}

A moving average process is covariance stationary with mean $\mu .$ It is
easy to show that the j$^{th}$-order autocovariance is $\gamma _{j}=\sigma
^{2}\sum_{k=0}^{q-j}\theta _{j+k}\theta _{k}$ for $j=0,1,...q,$ and zero for 
$j>q,$ with $\sigma ^{2}$ denoting the variance of the $\epsilon $ white
noise process. Thus, if $q=1,$ the zero-order autocovariance is $\left(
1+\theta _{1}\right) \sigma ^{2},$ the first-order autocovariance is $\theta
_{1}\sigma ^{2},$ and the second-order (and all higher-order)
autocovariances are zero. The entire autocovariance function is described by 
$(q+1)$ parameters: $\{\theta _{1},...\theta _{q},\sigma ^{2}$ \}. The
autocorrelation function depends only upon the $q$ parameters in the $\theta 
$ sequence.

We may also consider an $MA(\infty )$ process, in which the effects of past
shocks do not abruptly dissipate after $q$ periods. Thus, we might consider
that the finite set of terms in (\ref{maq}) could be replaced by an infinite
set of terms $\sum_{j=0}^{\infty }\psi _{j}\epsilon _{t-j}.$ For this to
make sense, it must be the case that this sum of an infinite set of random
variables is well defined---that is, that the partial sum $%
\sum_{j=0}^{n}\psi _{j}\epsilon _{t-j}$ converges to a finite random
variable as $n\rightarrow \infty .$ A condition under which this convergence
will occur is that of absolute summability: $\sum_{j=0}^{n}|\psi
_{j}|<\infty .$ Intuitively, this requires that the effects of the past
shocks represented by $\psi _{j}$ eventually die away. If the sequence $\psi
_{j}$ is absolutely summable, then an infinite--order MA\ process ($%
MA(\infty ))$ for $y_{t}$ converges in mean square, and the process is
covariance stationary, with mean $\mu $ and autocovariances $\gamma
_{j}=\sigma ^{2}\sum_{k=0}^{\infty }\psi _{j+k}\psi _{k}.$ The
autocovariances will themselves be absolutely summable. If the $\epsilon $
process is $iid,$ then the $y$ process is strictly stationary and ergodic.

We may now define a \textbf{linear filter}. Let $\{x_{t}\}$ be a covariance
stationary process and $\{h_{j}\}$ be a sequence of real numbers that is
absolutely summable. Then the infinite sum $y_{t}=\sum_{j=0}^{\infty
}h_{j}x_{t-j}$ converges in mean square, such that the $y$ process is
covariance stationary. If the autocovariances of the $x$ process are
absolutely summable, then so are the autocovariances of the $y$ process.

The operation of taking a weighted average of (possibly infinitely many)
successive values of the $x$ process is called \textbf{filtering}, and we
can write it using lag operator notation, with a filter being represented by
a lag polynomial $\alpha (L)=\alpha _{0}+\alpha _{1}L+\alpha _{2}L^{2}...$
The object $y_{t}=\alpha (L)x_{t}$ is a well-defined random variable forming
a covariance stationary process if the sequence $\{\alpha _{j}\}$ is
absolutely summable and if the input process $x_{t}$ is covariance
stationary.

\textbf{Properties of linear filters}

For a given sequence of real numbers $\{\alpha _{0},\alpha _{1},...\}$
define a \textbf{filter} as the lag polynomial $\alpha (L)=\alpha
_{0}+\alpha _{1}L+\alpha _{2}L^{2}+...,$ which may be applied to an input
process $\{x_{t}\}$ to yield $\alpha (L)x_{t}=\sum_{j=0}^{\infty }\alpha
_{j}^{{}}x_{t-j}.$ The filter could be finite, such that $\alpha _{j}=0$ for 
$j>p,$ which then defines a $p^{th}-\,$order lag polynomial. When applied to
an input process, this finite filter creates a weighted average of the
current and $p$ most recent values of the process. If the input process is
covariance stationary and the sequence $\{\alpha _{j}\}$ is absolutely
summable, the filtered series will be a covariance stationary process.

Let $\{\alpha _{j}\}$ and $\{\beta _{j}\}$ be two arbitrary sequences of
real numbers, and define the \textbf{convolution} of these sequences $%
\{\delta _{j}\}$ as 
\begin{eqnarray}
\alpha _{0}\beta _{0} &=&\delta _{0},  \label{convolute} \\
\alpha _{0}\beta _{1}+\alpha _{1}\beta _{0} &=&\delta _{1},  \nonumber \\
\alpha _{0}\beta _{2}+\alpha _{1}\beta _{1}+\alpha _{2}\beta _{0} &=&\delta
_{2},...  \nonumber \\
\alpha _{0}\beta _{j}+\alpha _{1}\beta _{j-1}+\alpha _{2}\beta
_{j-2}+...+\alpha _{j-1}\beta _{1}+\alpha _{j}\beta _{0} &=&\delta _{j} 
\nonumber
\end{eqnarray}
\textit{Product}: the convolution may be viewed as the product of two
filters: 
\[
\delta (L)=\alpha (L)\beta (L) 
\]

which may be computed with the same techniques as the product of two
ordinary polynomials, e.g.

\[
\left( 1+\alpha _{1}L\right) \left( 1+\beta _{1}L\right) =1+\left( \alpha
_{1}+\beta _{1}\right) L+\alpha _{1}\beta _{1}L^{2}. 
\]

If each of these sequences are absolutely summable, and the input series is
covariance stationary, then $\alpha (L)\beta (L)x_{t}=\delta (L)x_{t}$ is a
well--defined random variable, also covariance stationary, and the sequence $%
\{\delta _{j}\}$ will also be absolutely summable.

\textit{Inverse}: if $\delta (L)=1,$ so that $\alpha (L)\beta (L)=1,$ we may
say that $\beta (L)$ is the inverse of $\alpha (L),$ denoted as $\alpha
(L)^{-1}.$ As long as $\alpha _{0}\neq 0,$ the inverse of $\alpha (L)$ may
be defined for any arbitrary sequence $\{\alpha _{j}\}$ by successively
solving the equations (\ref{convolute}). For instance, $%
(1-L)^{-1}=1+L+L^{2}+L^{3}+...$ which may not be absolutely summable.

To work with ARMA\ processes, we must calculate the inverse of a
finite--order ($p^{th}-$ degree) lag polynomial $\Theta (L):$

\[
\Theta (L)=1-\Theta _{1}L-\Theta _{2}L^{2}-...-\Theta _{p}L^{p} 
\]

Since $\Theta _{0}\neq 0,$ the inverse polynomial $\Psi (L)=\Theta (L)^{-1}$
may be defined. We may determine the coefficients of the inverse polynomial
using the convolution formula in (\ref{convolute}); for $j>p,$ the
coefficients of $\Psi (L)$ may be solved recursively, knowing the first $p$
coefficients. The coefficients in the sequence $\{\Psi _{j}\}$ will
eventually decline if the stability condition on the $\Theta (L)$
polynomial---relating to its eigenvalues---holds, and those coefficients
will then be absolutely summable. As an example, consider the lag polynomial 
$1-\phi L.$ The root of the associated polynomial $1-\phi z=0$ is $\phi
^{-1},$ and the stability condition requires that $|\phi ^{-1}|>1$ or,
alternatively, that $|\phi |<1.$ The inverse of the filter is, given
stability, $1+\phi L+\phi ^{2}L^{2}+\phi ^{3}L^{3}+...,$ an infinite
sequence, but one which is bounded and finite.

\textbf{The} $AR(1)$\ \textbf{process}

A\ first-order autoregressive process ($AR(1))$ satisfies the stochastic
difference equation

\begin{eqnarray*}
y_{t} &=&c+\phi y_{t-1}+\epsilon _{t} \\
(1-\phi L)y_{t} &=&c+\epsilon _{t}
\end{eqnarray*}

where the $\epsilon $ process is white noise. For $\phi \neq 1,$ may rewrite
this equation as 
\begin{eqnarray*}
(1-\phi L)\left( y_{t}-\mu \right) &=&\epsilon _{t}, \\
\mu &=&c/(1-\phi )
\end{eqnarray*}

If $y_{t}$ is covariance stationary, $\mu $ is the mean of the process, and
the moving average is formed from the successive values of $y_{t}.$ We seek
a covariance--stationary solution to this stochastic difference equation. If 
$|\phi |<1,$ we may use the inverse given above, so that we may define 
\begin{eqnarray*}
(1-\phi L)^{-1}(1-\phi L)\left( y_{t}-\mu \right) &=&(1-\phi L)^{-1}\epsilon
_{t} \\
\left( y_{t}-\mu \right) &=&(1-\phi L)^{-1}\epsilon _{t} \\
y_{t} &=&\mu +\sum_{j=0}^{\infty }\phi ^{j}\epsilon _{t-j}.
\end{eqnarray*}

In this case, $y_{t}$ has the moving average representation shown here:an
infinite--order moving average process, or $MA(\infty ).$ The stability
condition that allows us to form the inverse is known as the stationarity
condition for the autoregressive lag polynomial, which states that shocks to
the system will be damped.

What about the case where that condition is violated, and $|\phi |>1?$ Then
there is a ``forward solution'' where the unique covariance--stationary
process $y_{t}$ is an infinite-order moving average of \textit{future}
values of $\epsilon :$%
\[
y_{t}=\mu -\sum_{j=0}^{\infty }\phi ^{-j}\epsilon _{t+j} 
\]

and this infinite sum is well defined, since the sequence $\{\phi ^{-j}\}$
is absolutely summable if $|\phi |>1.$

Last, let us consider the case where there is no covariance--stationary
solution, either in terms of past values of $\epsilon $ nor future values of 
$\epsilon $ : the case of a \textbf{unit root}, where $\phi =1.$ In this
case, the $j-$ period change in $y$ becomes

\[
y_{t}-y_{t-j}=c\cdot j+\epsilon _{t}+\epsilon _{t-1}+\epsilon
_{t-2}+...+\epsilon _{t-j+1}, 
\]
which we call a \textbf{random walk with drift}.

\textbf{The }$AR(p)$ \textbf{process}

All that we have derived here for the $AR(1)$ process may equally well be
expressed in terms of the $p^{th}-$ order autoregressive process, which
satisfies the $p^{th}-$ order stochastic difference equation $\Phi
(L)y_{t}=c+\epsilon _{t}$ with $\Phi (L)=1-\phi _{1}L-\phi
_{2}L^{2}-...-\phi _{p}L^{p}.$ If the process is CS, then $y_{t}$ has a mean
of $\mu =c\Phi (1)^{-1},$ and an $MA(\infty )$ representation $y_{t}=\mu
+\Psi (L)\epsilon _{t},$ with $\Psi (L)=\psi _{0}+\psi _{1}L+\psi
_{2}L^{2}...,$ where $\Psi (L)=\Phi (L)^{-1}.$

$\mathbf{The}$ $ARMA(p,q)$ \textbf{process}

We may combine finite--order $AR(p)$ and $MA(q)$ processes to yield an $%
ARMA(p,q)$ process:

\begin{eqnarray*}
y_{t} &=&c+\phi _{1}y_{t-1}+\phi _{2}y_{t-2}+...+\phi _{p}y_{t-p}+\theta
_{0}\epsilon _{t}+ \\
&&\theta _{1}\epsilon _{t-1}+\theta _{2}\epsilon _{t-2}+...+\theta
_{q}\epsilon _{t-q},
\end{eqnarray*}

or 
\[
\Phi (L)y_{t}=c+\Theta (L)\epsilon _{t}. 
\]

If $\Phi (1)\neq 1,$ we may set $\mu =c\Phi (1)^{-1},$ and write the model
in the deviation--from--mean form

\[
\Phi (L)\left( y_{t}-\mu \right) =\Theta (L)\epsilon _{t} 
\]
which is still a $p^{th}-$ order stochastic difference equation, but with a
serially correlated forcing process $\Theta (L)\epsilon _{t}$ rather than
the white noise process $\epsilon _{t}.$ If the equation satisfies the
stationarity conditions, then its unique covariance--stationary solution has
the $MA(\infty )$ representation

\begin{eqnarray*}
y_{t} &=&\mu +\Psi (L)\epsilon _{t}. \\
\Psi (L) &=&\Phi (L)^{-1}\Theta (L).
\end{eqnarray*}
The mean of the $ARMA(p,q)$ process is again $c\Phi (1)^{-1}.$

If there are common roots in the two polynomials, then they could be
factorized to represent the $ARMA(p,q)$ process with a lower order process $%
ARMA(p-1,q-1).$ We generally assume that there are no common roots in the $AR
$ and $MA$ polynomials. If the $\Theta (L)$ polynomial satisfies the
stability condition, then the $ARMA\;$process is said to be \textbf{%
invertible}, and that stability condition is often termed the invertibility
condition. If an $ARMA\;$process is invertible, then we may also express it
in an infinite--order $AR\;$ representation 
\[
\Theta (L)^{-1}\Phi (L)y_{t}=c\Theta (1)^{-1}+\epsilon _{t}.
\]

Note that a process can be invertible yet not be stationary, and vice versa;
both of its lag polynomials have to satisfy their respective stability
condition if the process is to be termed \textbf{stationary and invertible}.
If the process is both stationary and invertible, then the finite--order $%
ARMA(p,q)$ process also possesses both an $AR(\infty )$ and a $MA(\infty )$
representation.

\textbf{The autocovariance--generating function}

The entire set of autocovariances of a covariance stationary process $y_{t}$
may be summarized by the \textbf{autocovariance generating function}:

\begin{eqnarray*}
g_{y}\left( z\right)  &=&\sum_{j=-\infty }^{\infty }\gamma _{j}z^{j} \\
&=&\sum_{j=1}^{\infty }\gamma _{j}\left( z^{j}+z^{-j}\right) 
\end{eqnarray*}
with $z$ a complex scalar. Note that for $\left| z\right| =1,$ the unit
circle, this implies that 
\begin{eqnarray*}
g_{y}\left( 1\right)  &=&\sum_{j=-\infty }^{\infty }\gamma _{j} \\
&=&\gamma _{0}+2\sum_{j=1}^{\infty }\gamma _{j}
\end{eqnarray*}
which will represent a convergent sum given that the autocovariances are
absolutely summable. If we transform this function $g_{y}$ by dividing by $2%
\pi $ and setting \thinspace $z,$ a complex number on the unit circle, to $%
e^{-i\omega }=\cos (\omega )-i\sin (\omega ),$ we have defined the \textbf{%
spectral density function} or \textbf{power spectrum} of $y_{t}:$

\begin{eqnarray*}
s_{y}\left( \omega \right)  &=&\left( 2\pi \right) ^{-1}g_{y}\left(
e^{-i\omega }\right)  \\
&=&\left( 2\pi \right) ^{-1}g_{y}\left( \cos (\omega )-i\sin (\omega
)\right) 
\end{eqnarray*}
where $\omega $ represents the \textbf{frequency} (the inverse of the
period) of the cyclical component in $y.$ The time series may be represented
in the frequency domain, as the sum of an infinite number of sines and
cosines, or in the time domain, as a process with an infinite number of
autocovariances. The two representations are equivalent, and we can
represent $y$ in the frequency domain as the \textbf{Fourier transform} of
the timeseries data. We will discuss the frequency domain representation at
greater length when we discuss fractionally--integrated processes. (See
Hamilton, Chapter 6).

\textbf{Extension to vector processes}

A \textbf{vector white noise process} $\varepsilon _{t}$ is a jointly
covariance--stationary process satisfying 
\begin{eqnarray*}
E\left( \varepsilon _{t}\right) &=&0, \\
E\left( \varepsilon _{t}\varepsilon _{t}^{\prime }\right) &=&\Omega ,p.d., \\
E\left( \varepsilon _{t}\varepsilon _{t-j}^{\prime }\right) &=&0,~for~j\neq
0.
\end{eqnarray*}

Since $\Omega $ need not be diagonal, there may be contemporaneous
covariance among the elements of $\varepsilon _{t}.$ Perfect correlation is
ruled out, since $\Omega $ must be positive definite. A \textbf{vector} $MA(%
\infty )$ \textbf{process} may be expressed as: 
\begin{eqnarray*}
y_{t} &=&\mu +\sum_{j=0}^{\infty }\psi _{j}\varepsilon _{t-j}^{{}}. \\
\psi _{0} &=&I.
\end{eqnarray*}
This implies that if $\Gamma _{j}$ is the $j^{th}-$ order autocovariance
matrix $E\left[ \left( y_{t}-\mu \right) \left( y_{t-j}-\mu \right) ^{\prime
}\right] ,$ the autocovariance function may be written as

\[
\Gamma _{j}=\sum_{k=0}^{\infty }\psi _{j+k}\Omega \psi _{k}^{\prime } 
\]

Multivariate filters may also be written as $%
H(L)=H_{0}+H_{1}L+H_{2}L^{2}+...,$ with vector $y_{t}=H(L)x_{t}.$ Products
of filters may now be computed with linear algebra: for instance, the
product of two filters $A(L)$ and $B(L)$ can be written in terms of sums and
products of their coefficient matrices.

We may also consider the stability conditions in terms of lag matrix
polynomials. For instance, if we have 
\[
\Phi (L)=I_{r}-\Phi _{1}L-\Phi _{2}L^{2}-...-\Phi _{p}L^{p} 
\]

where $\Phi _{j}$ are $r\cdot r$ matrices, we may write the stability
condition for this polynomial as

\[
\left| I-\Phi _{1}z-...-\Phi _{p}z^{p}\right| =0
\]
all of the roots of which must lie outside the unit circle. For instance, if
we consider a two--variable form of a first--order lag polynomial $\Phi
(L)=I_{r}-\Phi _{1}L$ with coefficient matrix $
\begin{array}{cc}
\phi _{11} & \phi _{12} \\ 
\phi _{21} & \phi _{22}
\end{array}
,$ the determinantal equation becomes $1-\left( \phi _{11}+\phi _{22}\right)
z+\left( \phi _{11}\phi _{22}-\phi _{12}\phi _{21}\right) z^{2}=0.$

The $p^{th}-$ order\textbf{\ vector autoregressive process}, $VAR(p),$ will
be the solution to the vector stochastic difference equation

\begin{eqnarray*}
\Phi (L)\left( y_{t}-\mu \right) &=&\varepsilon _{t}, \\
\mu &=&\Phi (1)^{-1}c.
\end{eqnarray*}

The $VAR(p)$ is itself a special case of a multivariate $ARMA$ model, the $%
VARMA(p,q),$ where each of the variables in the system are considered to
follow an $ARMA\;$process of order $(p,q).$

The autocovariance generating function for a vector covariance--stationary
process $y_{t}$ may be written as

\begin{eqnarray*}
G_{Y}\left( z\right)  &=&\sum_{j=-\infty }^{\infty }\Gamma _{j}z^{j} \\
&=&\Gamma _{0}+\sum_{j=1}^{\infty }\left( \Gamma _{j}z_{j}+\Gamma
_{j}^{\prime }z^{-j}\right) 
\end{eqnarray*}
which leads to the definition of the spectrum of the vector process as 
\[
s_{Y}\left( \omega \right) =\left( 2\pi \right) ^{-1}G_{y}\left( e^{-i\omega
}\right) .
\]

Special cases of this for vector processes are: 
\begin{eqnarray*}
VWN &:&G_{Y}\left( z\right) =\Omega  \\
VMA(\infty ) &:&G_{Y}\left( z\right) =\Psi \left( z\right) \Omega \Psi
\left( z^{-1}\right) ^{\prime } \\
VAR(p) &:&G_{Y}\left( z\right) =\left[ \Phi (z)^{-1}\right] \Omega \left[
\Phi (z^{-1})^{-1}\right] ^{\prime }
\end{eqnarray*}
which thus allows us to define the autocovariances of these processes at all
lags.

\textbf{Estimating autoregressions, VARs and} $ARMA\;$\textbf{processes}

Pure $AR$ models may be consistently estimated with OLS, with the assumption
that the error process is $iid.$ How should the order of an $AR\;$process be
chosen?\ A sensible criterion is the general--to--specific sequential $t$
rule, which starts by estimating a model with $p_{\max }$ lags, where that
parameter is selected to a.s. overfit the process. If the high--order lag
term is significant at some prespecified level, then $p_{\max }$ should be
increased. Then, the autoregression is reestimated, dropping the high--order
lag, until the high--order lag is significant. Since the $t-$ test is
consistent, this rule will never underfit the model; it is biased toward
overfitting the model, and thus may be criticized on its lack of parsimony.
It should be noted that all of the models described above should be fit over
a common sample, to prevent the inclusion of additional data points from
modifying the judgement.

A second approach is to apply the Akaike information criterion ($AIC)$ or
the Schwartz information criterion ($SIC),$ also known as the Bayesian
information criterion $(BIC).$ In general terms, these criteria seek the
model which minimizes 
\[
\log \left( \frac{SSR_{j}}{n}\right) +\frac{\left( j+1\right) C(n)}{n} 
\]
where $n$ is the sample size, $SSR_{j}$ is the sum of squared residuals (the
least squares criterion) for the $j^{th}-$ order autoregression, and $C(n)$
is set to 2 for the $AIC\;$and $\log (n)$ for the $SIC$ ($BIC).$ Either of
the information criteria strike a balance between a better fit (in the first
term) and the parsimony of the model (in the second term). Ng and Perron
suggest, as above, the use of a fixed sample period to compare models with
these criteria. Just as with the general--to--specific rule, the $AIC\;$has
a positive probability of overfitting the model. In contrast, the $BIC$ is a
consistent estimator of $j.$

VARs may equally easily be estimated by single--equation OLS. Since the
right--hand side of each equation of the VAR contains the same set of
regressors, there is no gain in applying a system estimator, conditional on $%
iid$ errors. Likewise, there is a multivariate generalization of the $AIC\;$
and $BIC\;$that may be applied to search for the appropriate lag length of
the VAR\ (presuming that the same number of lags are to be used in each
equation).

The estimation of $ARMA(p,q)$ models is more challenging, in that the model
may be written as 
\begin{eqnarray*}
u_{t} &=&\epsilon _{t}+\theta _{1}\epsilon _{t-1}+...+\theta _{q}\epsilon
_{t-q}, \\
z_{t} &=&\left( 1,y_{t-1},y_{t-2},...,y_{t-p}\right) ^{\prime }, \\
\delta  &=&\left( c,\phi _{1},\phi _{2},...,\phi _{p}\right) ^{\prime }. \\
y_{t} &=&z_{t}^{\prime }\delta +u_{t}
\end{eqnarray*}
which poses several problems. The errors are serially correlated (since they
are $MA(q)$ ), and since the lagged dependent variables included in $z_{t}$
are correlated, by construction, with lags of $\epsilon $ included in the
error term, the regressors \thinspace $z_{t}$ are not orthogonal to the
error term $u_{t}.$ The second problem could be solved by using suitably
lagged values of the dependent variable (at lags higher than $q)$ as
instruments. But the efficient estimation of $ARMA(p,q)$ processes requires
a different modelling strategy, relying on the maximum likelihood principle
to estimate ``Box-Jenkins'' models.

For instance, consider the $MA(1)$ process. If we condition on initial
values for the $\epsilon $ 's, this becomes straightforward. The model:

\begin{eqnarray*}
Y_{t} &=&\mu +\epsilon _{t}+\theta \epsilon _{t-1}, \\
&&\epsilon _{t}\symbol{126}N(0,\sigma ^{2}).
\end{eqnarray*}

Let $\Theta =(\mu ,\theta ,\sigma ^{2})^{\prime }$ denote the population
parameters of interest. If $\epsilon _{t-1}$ was known with certainty, $%
Y_{t}|$ $\epsilon _{t-1}\symbol{126}N(\mu +\theta \epsilon _{t-1},\sigma
^{2})$ and we could write 
\[
f(Y_{t}|\epsilon _{t-1};\Theta )=\frac{1}{\sqrt{2\pi \sigma ^{2}}}\exp
\left[ \frac{-\left( y_{t}-\mu -\theta \epsilon _{t-1}\right) ^{2}}{2\sigma
^{2}}\right] 
\]

If we knew for certain that $\epsilon _{0}=0,$ we would know the value of $%
\epsilon _{1}$ with certainty as well, and could in turn calculate the value
of $\epsilon _{2}$ conditional on $\epsilon _{1},$ and so on. The entire
sequence of $\epsilon $ values can then be recursively calculated from 
\[
\epsilon _{t}=y_{t}-\mu -\theta \epsilon _{t-1}
\]
given knowledge of $\mu $ and $\theta .$ Thus the conditional loglikelihood
may be written as
\[
L(\Theta )=-\frac{T}{2}\log (2\pi )-\frac{T}{2}\log (\sigma
^{2})-\sum_{t=1}^{T}\frac{\epsilon _{t}^{2}}{2\sigma ^{2}}
\]
for a particular value of $\Theta .$ Numerical optimization may then be used
to vary elements of $\Theta $ and search for a maximum of the loglikelihood
function. No analytical solution exists---even for the $MA(1)$ ---but the
solution technique should be reliable. This approach conditions the
estimates on the specification that $\epsilon _{0}=0;$ other approaches,
which may be found in more sophisticated software, may use other techniques
for generating the initial conditions for optimization. If $\left| \theta
\right| $ is substantially less than unity, the effect of imposing $\epsilon
_{0}=0$ will quickly die out, and the conditional likelihood will be a good
approximation to the unconditional likelihood for a reasonably large sample
size. In contrast, if $\left| \theta \right| >1,$ the consequences of
imposing $\epsilon _{0}=0$ accumulate over time. If numerical optimization
yields a value in this region, the results must be discarded. An appropriate
starting value for numerical optimization in that case would be $\hat{\theta}%
^{-1}.$

In particular, a $MA(q)$ model will require the specification of $q$
starting values for pre--sample elements of the $\epsilon $ vector. Just as
above, we may generate conditional likelihood estimates on the assumption
that  $\epsilon _{0}=\epsilon _{-1}=...=\epsilon _{-q+1}=0.$ The conditional
loglikelihood is only useful if all values of $z$ for which 
\[
1+\theta _{1}z+\theta _{2}z^{2}+...+\theta _{q}z^{q}=0
\]
lie outside the unit circle.

Numerical optimization to estimate the parameters of a general $ARMA(p,q)$
process require two sets of assumed starting values: the $q$ presample
values of the  $\epsilon $ sequence, and $p$ presample values of the $y$
process. Of course, one may have the prior values of $y$ (that is, you may
start the estimation at least $p$ periods after the start of the data). Box
and Jenkins, in their classic text, recommended setting the presample $%
\epsilon $ 's to zero, but setting the presample $y$ 's to their actual
values. The same caution applies with respect to stability of the resulting
estimates: if the $MA$ polynomial does not satisfy the stability condition
(i.e. invertibility), the estimates should not be trusted. 

\end{document}
