
\documentclass{slides}

\begin{document}

\textbf{EC327:\ Advanced Econometrics, Spring 2007}

\textit{\small Wooldridge, Introductory Econometrics (3rd ed, 2006)}

\bigskip \textbf{Appendix D: \\ Summary of matrix algebra}

 
\textit{Basic definitions}

A {\bf matrix} is a rectangular array of numbers, with $m$ rows and $n$ columns, which are the {\it row dimension} and {\it column dimension}, respectively. The matrix {\bf A} will have typical element $a_{ij}$. 

A {\bf vector} is a matrix with one row and/or one column. Thus a {\it scalar} can be considered a $1 \times 1$ matrix. A $m$-element {\bf row vector} has one row and $m$ columns. A $n$-element {\bf column vector} has $n$ rows and one column. 

A {\bf square matrix} has $m=n$. A {\bf diagonal matrix} is a square matrix with off-diagonal elements equal to 0. It may have $m$ distinct diagonal elements. If those $m$ elements are equal, it is a {\bf scalar matrix}. If they all equal 1, it is an {\bf identity matrix} of order $m$, customarily written as {\bf I} or {\bf I$_m$}. 

A {\bf symmetric matrix} is a square matrix for which $a_{ij} = a_{ji}~ \forall i,j$. The elements above and below its {\it main diagonal} are equal. It is often written in {\it upper triangular} or {\it lower triangular} form, since there is no need to report more than the main diagonal and sub- (super-) diagonal elements. 

A symmetric matrix we often compute in econometrics is the {\it correlation matrix} of a set of variables. The correlation matrix will have {\tt 1}s on its main diagonal (since every variable is perfectly correlated with itself) and off-diagonal values between (-1,+1). 

Another very important matrix is what Stata calls the {\small VCE}, or estimated variance-covariance matrix of the estimated parameters of a regression equation. It is by construction a symmetric matrix with positive elements on the main diagonal (the estimated variances of the parameters $b$, whose positive square roots are the reported standard errors of those parameters). The off-diagonal elements are the estimated covariances of the estimated parameters, used in computing hypothesis tests or confidence intervals involving more than one parameter. This matrix may be examined after a {\tt regress} command in Stata with the command {\tt estat vce}.

The {\bf transpose} of a matrix reflects the matrix around its main diagonal. We write the transpose of {\bf A} as {\bf A'} (or, less commonly, {\bf A$^T$}). If {\bf B = A'}, then  
$b_{ij} = a_{ji}~ \forall i,j$. If {\bf A} is $m \times n$, {\bf B} is of order $n \times m$. The rows of {\bf A} become the columns of {\bf A}, and {\it vice versa}. 

Several relations involving the transpose of a matrix: \\(1) the transpose of a row vector is a column vector; \\(2) the transpose of a transpose is the matrix itself; \\(3) the transpose of a symmetric matrix is the matrix itself; \\(4) The transpose of the sum (difference) is the sum (difference) of the transposes.

A matrix of any row and column dimension with all elements equal to zero is a {\bf zero matrix} or {\bf null matrix}. It plays the role in matrix algebra (or {\it linear algebra}) that the scalar 0 plays in ordinary algebra, in the sense that adding or subtracting a null matrix has no effect, and multiplying by a null matrix returns a null matrix.

The identity matrix {\bf I} plays the role of the number 1 in ordinary algebra: e.g.,  multiplying by {\bf I} has no effect, and we can always insert an {\bf I} of appropriate order in a matrix product without changing the product. 

\textit{Matrix operations}

In matrix algebra, algebraic operations are only defined for matrices or vectors of appropriate order. Since vectors are special cases of matrices, we will only speak of matrices.  Two matrices {\bf A} and {\bf B} are {\bf equal} if they have the same {\it order}---the same number of rows and columns---and if $a_{ij} = b_{ij}~ \forall i,j$. Addition or subtraction can be performed if and only if the matrices are of the same order. The sum (difference)  {\bf C = A} $\pm$ {\bf B} is defined as $c_{ij} = a_{ij} \pm b_{ij}~ \forall i,j$. That is, just as we compare matrices for equality element-by-element, we add (subtract) matrices element-by-element. 

{\bf Scalar multiplication} is defined for any matrix as {\bf C =} k {\bf A}, and involves multiplying each element by that scalar:  $c_{ij} = k \times a_{ij}~ \forall i,j$.

{\bf Matrix multiplication} is defined for matrices {\bf A$_{m \times n}$} and {\bf B$_{n \times q}$} as {\bf C$_{m \times q}$ = A B}. This product is defined since the number of {\it columns} in the first matrix equals the number of {\it rows} in the second matrix.  To define matrix multiplication, we must first define the {\bf dot product} of vectors {\bf u$_{n \times 1}$} and {\bf v$_{n \times 1}$}. The product $d = u' v$ is a scalar, defined as $d = \sum_{i=1}^n u_i v_i$. This implies that if {\bf u = v}, the scalar $d$ will be the sum of squares of the elements of {\bf u}. We may also compute the {\bf outer product} of these vectors, {\bf u v'}, which for vectors of the same length will create a $n \times n$ matrix.

In matrix multiplication, each element of the result matrix is defined as a dot product. If 
{\bf C = A B}, $c_{ij}$ is the dot product of the $i^{th}$ row of {\bf A} and the $j^{th}$ column of {\bf B}: $c_{ij} = \sum_{k=1}^n A_{ik} B_{kj}$. These dot products are only defined for vectors of the same length, which gives rise to the constraint on the number of columns of {\bf A} and the number of rows of {\bf B}. When this constraint is satisfied, the vectors are {\it conformable} for multiplication.

In ordinary algebra, multiplication is {\it commutative}: we can write {\bf x y} or {\bf y x} and receive the same result. In matrix algebra, the product of arbitrary matrices will not exist unless they are {\it conformable}. if one of those products exists, the other will not except in special cases. If {\bf A} and {\bf B} are square matrices of the same order, then both {\bf A B} and {\bf B A} always exist, but they do not yield the same result matrix except under special circumstances.  (A natural exception: we can always write {\bf I A} and {\bf A I}, both of which equal {\bf A}. Likewise for the null matrix.) 

We can also multiply a matrix by a vector. If we write {\bf C = u A}, we are multiplying the $m$-element row vector {\bf u} by matrix {\bf A}, which must have $m$ rows; it may have any number of columns. We are {\it premultiplying} {\bf A} by {\bf u}. We may also {\it postmultiply} {\bf A} by vector {\bf v}. If {\bf A} is $m \times n$, {\bf v} must be a $n$-element column vector. 

Some properties of matrix multiplication: \\ (1) multiplication is {\it associative} with respect to scalars and matrices, as long as they are conformable: \\$k $({\bf A + B}) = $k ${\bf A} + $k${\bf B}, \\ {\bf C ( A + B)} = {\bf C A} + {\bf C B}, \\   {\bf ( A + B) C} = {\bf A C } + {\bf B C}. \\ (2) The transpose of a product is the product of the transposes, in reverse order: \\{\bf (A B)' = B' A'}. \\ (3) For any matrix {\bf X} the products {\bf X'X} \\and {\bf X X'} exist, and are symmetric. 

We cannot speak of dividing one matrix by another (unless they are both scalars, and the rules of ordinary algebra apply). Instead, we define the concept of the {\bf inverse matrix}. A square matrix {\bf A$_{m \times m}$} may possess a unique {\it inverse}, written {\bf A$^{-1}$}, under certain conditions. If it exists, the inverse is defined as that matrix which satisfies {\bf A A$^{-1}$ = A$^{-1}$ A = I$_m$}, and in that sense the inverse operates like the division operator in normal algebra, where $x \times \frac{1}{x} = 1 ~\forall x \neq 0$.  A matrix which possesses an inverse is said to be {\it nonsingular}, or {\it invertible}, and it has a nonzero {\it determinant}. A {\it singular} matrix possesses a zero determinant. We will not go into the computation of inverse matrices or determinants---which is better left to a computer---but we must understand their importance to econometrics and the linear regression model in particular.

Properties of inverses: \\ (1) The inverse of a product is the product of the inverses, in reverse order: {\bf (A B)$^{-1}$ = B$^{-1}$ A$^{-1}$}. The transpose of the inverse is the inverse of the transpose: ({\bf A'})$^{-1}$ = ({\bf A$^{-1})^{'}$}.

The {\it trace} of a square matrix, denoted {\bf tr(A)}, is the scalar sum of its diagonal elements. So, for instance, the trace of the identity matrix {\bf I}$_n$ is $n$. The trace of the sum (difference) of matrices is the sum (difference) of the traces, and the trace of a product yielding a square matrix is not dependent on order: that is, if {\bf A}$_{m \times n}$ and {\bf B}$_{n \times m}$, both the products {\bf AB} and {\bf BA} exist, and have the same trace.

\textit{Linear independence and rank}

The notion of a matrix possessing an inverse is related to the concept of {\it linear independence}. A set of $n$-vectors [{\bf x$_1$, x$_2$, \dots, x$_r$}] is said to be {\it linearly independent} if and only if $\alpha'$ {\bf x = 0} implies that $\alpha$ is the null vector. If $\alpha'$ {\bf x = 0} holds for a set of scalars $\alpha_1, \alpha_2, \dots, \alpha_r$ that are not all zero, then the set of {\bf x}-vectors are {\bf linearly dependent}. This implies that at least one vector in this set can be written as a linear combination of the others. In econometric terms, this is the problem of {\it perfect collinearity}: one or more of the regressors can be expressed as an exact linear combination of the others.

If {\bf A} is a $n \times k$ matrix, the {\it column rank} of {\bf A} is the maximum number of linearly independent columns of {\bf A}. If rank({\bf A}) = $k$, {\bf A} has {\it full column rank}. Row rank is defined similarly. The {\it rank of a matrix} is the minimum of its row and column ranks. When we consider a data matrix {\bf X}$_{n \times k}$, with $n > k$, its rank cannot exceed $k$. If a square matrix {\bf A} of order $k$ is of full rank (rank=$k$), then {\bf A}$^{-1}$ exists: the matrix is invertible. The rank of a product of matrices cannot exceed either of their ranks, and may be zero.

\textit{Quadratic forms and positive definite matrices}

If {\bf A} is a $n \times n$ symmetric matrix, then the {\it quadratic form} associated with {\bf A} is the scalar function
\[
Q =  x' A x = \sum_{i=1}^n a_{ii}x^2_i + 2 \sum_{i=1}^n \sum_{j>i} a_{ij} x_i x_j
\]
where {\bf x} is any $n$-vector. If $x' A x > 0$ for all $n$-vectors {\bf x} except {\bf x=0}, then matrix {\bf A} is said to be {\it positive definite} (p.d.). If $x' A x \geq 0$, then  {\bf A} is said to be {\it positive semi-definite} (p.s.d.). A p.d. matrix {\bf A} has all positive diagonal elements and possesses an inverse {\bf A}$^{-1}$ which is also p.d. and a positive determinant. For any {\bf X}$_{n \times k}$, {\bf X'X} and {\bf XX'} are p.s.d.  If {\bf X}$_{n \times k}$ with $n > k$ has rank $k$, then {\bf X'X} is p.d. and nonsingular, implying that {\bf X'X}$^{-1}$ exists. 

This is the relevant concern for regression, where we have a data matrix {\bf X} of $n$ observations on $k$ variables or regressors, with $n > k$. If those regressors are linearly independent, so that  {\bf X}  is of full rank $k$, we can invert the matrix {\bf X'X}: a key step in computing linear regression estimates.

\pagebreak
\textit{Idempotent matrices}

If {\bf A} is a $n \times n$ symmetric matrix, it is said to be {\it idempotent} if and only if {\bf A A = A}. The identity matrix, playing the role of the number 1 in ordinary algebra, is idempotent. An idempotent matrix has rank equal to its trace, and it is p.s.d. 

If we have a data matrix {\bf X}$_{n \times k}$ with rank({\bf X})=$k$, then
\begin{eqnarray}
P & = & X(X'X)^{-1}X' \\
M & = & I_n - X(X'X)^{-1}X' = I_n - P
\end{eqnarray}
are both symmetric, idempotent matrices. {\bf P}$_{n \times n}${ has rank $k$, while {\bf M}$_{n \times n}$ has rank $n-k$, since the trace of {\bf P} is that of {\bf I}$_k$, and its trace is equal to its rank. 
 
\pagebreak
\textit{Matrix differentiation}

For  $n$-vectors {\bf a} and {\bf x}, define $f(x) = a'x$. Then the derivative of the function with respect to its (vector) argument is
\[
\partial{f} / \partial{x} = a'
\] 
a $1 \times n$ vector. For a $n \times n$ symmetric matrix {\bf A} with quadratic form $Q = x' A x$,  the derivative of {\bf Q} with respect to its vector argument is
\[
\partial{Q} / \partial{x} = 2 x' A
\]
a $1 \times n$ vector. 

\textit{Moments and distributions of random vectors}

Operations on random variables can be expressed in terms of vectors of random variables. If {\bf y} is a $n$-element random vector, then its expected value {\bf E[y]} is merely the $n$-vector of its expected values. This generalizes to a random matrix. A linear transformation of {\bf y} with non-random matrices yields 
\[
E[ A y + b ] = A E[y] + b
\]
If {\bf y} is a $n$-element random vector, then its {\it variance-covariance matrix} or {\small VCE} is the symmetric matrix

Var(y) = \[
\left(
\begin{array}{cccc}
\sigma_1^2  & &   &   \\
\sigma_{21}  & \sigma_2^2  & &  \\
\vdots  &  \vdots &   \\
\sigma_{n1} & \sigma{n2} & \dots \sigma_n^2 \\
\end{array}
\right)
\]
where $\sigma_j^2$ is the variance of {\bf y}$_j$ and  $\sigma_{ij}$ is the covariance of 
{\bf y}$_i$ and {\bf y}$_j$.

Just as we can perform algebra on scalar variances and covariances, we can operate on the {\small VCE} of {\bf y}. Some properties: \\
(1) If {\bf a} is a $n$-element nonrandom vector, {\bf Var(a'y) = [a' Var(y) a]} $\geq 0$. \\
(2) If {\bf Var(a'y)} $> 0~ \forall a \neq ${\bf 0}, {\bf Var(y)} is p.d. and possesses an inverse. \\
(3) If $\mu = ${\bf E[y]}, $Var(y) = E[(y-\mu) (y-\mu)^{'}]$. \\
(4) If the elements of {\bf y} are uncorrelated, {\bf Var(y)} is a diagonal matrix. This is the assumption of independence of the elements of random vector {\bf y}: for instance, of the error terms of a regression equation.\\
(5) If in addition {\bf Var(y$_j$) = }$\sigma^2 ~\forall j$, then {\bf Var(y) = }$\sigma^2 I_n$. This is the assumption of homoskedasticity of the elements of random vector {\bf y}: for instance, of the error terms of a regression equation. \\
(6) For nonrandom {\bf A}$_{m \times n}$ and {\bf b}$_{n \times 1}$, {\bf Var(A y + b) = [ A Var(y) A' ]}.

If {\bf y} is a $n$-element {\it multivariate Normal} random vector with mean vector $\mu$ and {\small VCE} $\Sigma$, we write $y \sim N(\mu, \Sigma)$. Properties of the multivariate normal distribution:\\
(1) If $y \sim N(\mu, \Sigma)$, each element of {\bf y} is Normally distributed.\\
(2) If $y \sim N(\mu, \Sigma)$, any two elements of {\bf y} are independent if and only if they are uncorrelated ($\sigma_{ij} = 0$). \\
(3) If $y \sim N(\mu, \Sigma)$ and {\bf A, b} are nonrandom, then {\bf A y + b }$\sim N(A \mu + b, A \Sigma A')$ \\

A $\chi^2_n$ random variable is the sum of $n$ squared independent standard Normal variables. If $u \sim N(0, I_n)$, then {\bf u'u} $\sim \chi^2_n$.

A {\bf t}-distributed random variable is the ratio of a standard Normal variable {\bf Z} to a $\chi^2_n$ random variable {\bf X}, standardized by its degrees of freedom, where the variables {\bf Z, X} are independent. Let $u \sim N(0, I_n)$, {\bf c} be a nonrandom $n$-vector and {\bf A} be a nonrandom, $n \times n$ symmetric, idempotent matrix with rank $q$, with {\bf Ac = 0}. Then the quantity $[c'u/\sqrt{c'c}] /\sqrt{u'Au} \sim t_q$.

A {\bf F}-distributed random variable is the ratio of two independent $\chi^2$ random variables, standardized by their respective degrees of freedom. If $u \sim N(0, I_n)$ and {\bf A, B} are $n \times n$ nonrandom, symmetric idempotent matrices with rank({\bf A}) = $k_1$ and rank({\bf B}) = $k_2$, then\\ $((u' A u) /k_1) / (u' B u)/ k_2) \sim F^{k_1}_{k_2}$.


\pagebreak
\textbf{Appendix E: \\ The linear regression model in matrix form}

\textit{OLS estimation}

The multiple linear regression model with $k$ regressors can be written as
\[
y_t = \beta_0 + \beta_1 x_{t1} + \beta_2 x_{t2} + \dots + \beta_k x_{tk} + u_t, ~t=1,2,\dots,n
\]
where $y_t$ is the dependent variable for observation $t$ and $x_1 \dots x_k$ are the regressors. $\beta_0$ is the intercept term (constant) and $\beta_1 \dots \beta_k$ are the slope parameters.

We define {\bf x}$_t$ as the $1 \times (k+1)$ row vector $(1, x_{t1},\dots,x_{tk})$ and the column vector \\$\beta =  (\beta_0,\beta_1,\dots,\beta_k)^{'}$. Then the model can be written as
\[
y_t = {\bf x}_t \beta + u_t,~t=1,2,\dots,n
\]
and the entire regression problem as 
\begin{equation}
{\bf y = X \beta + u} \label{linreg}
\end{equation}
where {\bf y} is the $n \times 1$ vector of observations on the dependent variable, {\bf X}$_{n \times (k+1)}$ is the matrix of observations on the regressors, including an initial column of 1s, and  {\bf u} is the $n \times 1$ vector of unobservable errors. 

OLS estimation involves minimizing the sum of squared residuals, where the residuals $e_t = (y_t - x_t {\bf b})$ where {\bf b} is the vector of estimated parameters corresponding to the population parameters ${\bf \beta}$. Minimizing the sum of squared $e_t$ is equivalent to minimizing {\bf SSR = e'e} with respect to the $(k+1)$ elements of {\bf b}. A solution to this optimization problem involves a set of $(k+1)$ {\it first order conditions} (FOC) 
\[
\partial  SSR({\bf b}) / \partial {\bf b} = 0
\]
and setting those conditions equal to zero. The FOCs give rise to a set of $(k+1)$ simultaneous equations in the $(k+1)$ unknowns {\bf b}. In matrix form, the residuals may be written as $e = y - X {\bf b}$, and the FOCs then become
\begin{eqnarray}
X'(y - X {\bf b}) & = & 0 \label{foc} \nonumber \\
X'y & = & (X'X) {\bf b} \nonumber \\
b & = & (X'X)^{-1} X'y 
\end{eqnarray}
if the inverse exists, that is, if and only if {\bf X} is of full (column) rank $(k+1)$. If that condition is satisfied, the {\it cross-products matrix} {\bf X'X} will be a {\it positive definite} matrix. For that to be so, it must be the case that the columns of {\bf X} are linearly independent. This rules out the case of {\it perfect collinearity}, which will arise if one or more of the regressors can be written as a linear combination of the others. 

Recall that the first column of {\bf X} is a vector of 1s. This implies that no other column of {\bf X} can take on a constant value, for it would be a multiple of the first column. Likewise, a situation where the sum of some of the regressors equals a column of ones will violate this assumption. This will occur in the case of the {\it dummy variable trap} where a complete set of {\it mutually exclusive and exhaustive} (MEE)  indicator variables are included in a regression containing a constant term.

Given a solution for {\bf b}, the OLS predicted (fitted) values $\hat{y} = X {\bf b}$, and the calculated residuals equal $e = y - \hat{y}$. From Equation (\ref{foc}), the first order condition may be written as ${\bf X'e = 0}$. Since the first column of {\bf X} is a vector of 1s, the FOC implies that the residuals sum to zero and have an average value of zero. The remaining FOCs imply that each column of {\bf X} (and any linear combinations of those columns) has zero covariance with {\bf e}. This is algebraically implied by OLS, not an assumption. Since $\hat{y}$ is such a linear combination, it is also true that ${\bf \hat{y}'e = 0}$: the residuals have zero covariance with the predicted values.


\pagebreak
\textit{Finite sample properties of OLS}

The assumptions underlying the OLS\ model:
\begin{enumerate}
\item {\it Linear in parameters}: The model can be written as in Equation (\ref{linreg}), with the observed $n$-vector {\bf y}, observed $n \times (k+1)$ matrix {\bf X} and $n$-vector {\bf u} of unobserved errors.

\item {\it No perfect collinearity}: The matrix {\bf X} has rank $(k+1)$.

\item {\it Zero conditional mean}: Conditional on the matrix {\bf X}, each element of {\bf u} has zero mean: $E[u_t | {\bf X}] = 0~\forall t$.

\item {\it Spherical disturbances}: $Var(u|{\bf X}) = \sigma^2 I_n$. This combines the two assumptions of {\it homoskedasticity}, $Var(u_t|{\bf X}) = \sigma^2~\forall t$ and independence, $Cov(u_t,u_s|{\bf X}=0~\forall t \neq s$. The first assumption implies that the $u_t$ are {\it identically} distributed, while the second assumption implies that they are {\it independently} distributed, or in a time series context, free of serial correlation. Taken together, they allow us to say that $u$ is a {\it i.i.d.}\ random variable with a {\it scalar variance-covariance matrix}.

\end{enumerate}

Given these four assumptions, we may prove several theorems related to the OLS\ estimator:

\begin{enumerate}
\item {\it Unbiasedness of OLS}: 
Under assumptions 1, 2 and 3, the OLS\ estimator $b$ is unbiased with respect to $\beta$. 
\begin{eqnarray}
b & = & (X'X)^{-1} X'y \\
& = &  (X'X)^{-1} X'(X \beta + u) \nonumber \\
& = & (X'X)^{-1} (X'X) \beta +  (X'X)^{-1} X'u \label{unbiased} \nonumber
\end{eqnarray}
Taking the conditional expectation,
\begin{eqnarray}
E[b|{\bf X}] & = & \beta + (X'X)^{-1} X'~ E(u|{\bf X}) \\
& = & \beta + (X'X)^{-1} X'0 \nonumber \\
& = & \beta  \nonumber 
\end{eqnarray}
so that $b$ is unbiased.

\item{\it {\small VCE} of the OLS\ estimator}:
Under assumptions 1, 2, 3 and 4, 
\begin{equation}
Var(b|{\bf X}) = \sigma^2 (X'X)^{-1}.
\end{equation}
From the last line of Equation (\ref{unbiased}), we have
\begin{eqnarray}
Var(b|{\bf X}) & = & Var[(X'X)^{-1} X'u|X] \\
& = & (X'X)^{-1} X' [Var(u|X)] X (X'X)^{-1} \nonumber
\end{eqnarray}
Crucially depending on assumption 4, we can then write
\begin{eqnarray}
Var(b|{\bf X}) & = &  (X'X)^{-1} X' (\sigma^2 I_n) X (X'X)^{-1} \nonumber \\
& = & \sigma^2 (X'X)^{-1}(X'X)(X'X)^{-1} \nonumber \\
& = & \sigma^2 (X'X)^{-1}
\end{eqnarray}
This conditional {\small VCE} depends on the unknown parameter $\sigma^2$, but replacing that with its consistent estimate $s^2$ it becomes an operational formula.

\item{\it Gauss--Markov}:
Under assumptions 1, 2, 3 and 4, $b$ is the Best Linear Unbiased Estimator of $\beta$ ($b$ is {\it BLUE}).
Any {\it linear} estimator of $\beta$ can be written as 
\begin{equation}
\hat{\beta} = A'y
\end{equation}
where ${\bf A}$ is a $n \times (k+1)$ matrix whose elements are not functions of {\bf y} but may be functions of {\bf X}. Given the model of Equation (\ref{linreg}), we may write $\hat{\beta}$ as
\begin{equation}
\hat{\beta} = A'(X\beta + u) = (A'X)\beta + A'u. \label{otherest}
\end{equation}
We may then write the conditional expectation of $\hat{\beta}$ as
\begin{eqnarray}
E[\hat{\beta}|X] & = & A'X\beta + E[A'u|X] \nonumber \\
& = & A'X \beta + A' E(u|X) \nonumber \\
& = & A'X\beta
\end{eqnarray}
The last line following from assumption 3. For $\hat{\beta}$ to be an unbiased estimator, it must be that $E[\hat{\beta}|X] = \beta~\forall \beta$. Because ${\bf A'X}$ is a $(k+1) \times (k+1)$ matrix, unbiasedness requires that ${\bf A'X = I_{k+1}}$. 

From Equation (\ref{otherest}) we have
\begin{equation}
Var(\hat{\beta}|X) = A'[Var(u|X)]A = \sigma^2 A'A
\end{equation}
invoking assumption 4 ($i.i.d.$\ disturbances). Therefore
\begin{eqnarray}
Var(\hat{\beta}|X) - \nonumber \\ Var(b|X) & = & \sigma^2 [A'A-(X'X)^{-1}] \nonumber \\
& = & \sigma^2 [A'A- A'X(X'X)^{-1}X'A] \nonumber \\
& = & \sigma^2 A'[I_n - X(X'X)^{-1}X']A \nonumber \\
& = & \sigma^2 A'MA \label{gm}
\end{eqnarray}
where $M = [I_n - X(X'X)^{-1}X']$, a symmetric and idempotent matrix which is positive semi-definite (p.s.d.) for any matrix ${\bf A}$. But Equation (\ref{gm}) represents the difference between the {\small VCE} of any arbitrary linear estimator of $\beta$ and the {\small VCE} of the OLS\ estimator. That difference will be the null matrix if and only if 
$A'A=(X'X)^{-1}$: that is, if we choose $A$ to reproduce the OLS\ estimator. For any other choice of $A$, the difference will be a positive semi-definite matrix, which implies that at least one of the diagonal elements is larger in the first matrix than in the second. That is, the maximum precision estimator of each element of $\beta$ can only be achieved by OLS. Any other linear, unbiased estimator of $\beta$ will have a larger estimated variance for at least one element of $\beta$. Thus OLS\ is {\it BLUE}: the Best (minimum-variance, or most efficient) Linear Unbiased Estimator of $\beta$.

\item{\it Unbiasedness of $s^2$}: The unbiased estimator of the error variance $\sigma^2$ can be calculated as $s^2 = e'e/(n-k-1)$, with $E[s^2 | X] = \sigma^2 ~\forall ~\sigma^2>0$.
\begin{eqnarray}
e & = & y - X b \\
& = & y - X [(X'X)^{-1}X'y] \nonumber \\
& = & My = Mu \nonumber
\end{eqnarray}
where $M$ is defined as above. Since $M$ is symmetric and idempotent,
\begin{equation}
e'e = u'M'Mu = u'M u
\end{equation}
which is a scalar, equal to its trace. So 
\begin{eqnarray}
E[u'Mu|X] & = & E[tr(u'Mu)|X] \\
& = & E[tr(Muu')|X] \nonumber \\
& = & tr[E(Muu')|X] \nonumber \\
& = & tr[M E[uu'|X]] \nonumber \\
& = & tr(M\sigma^2 I_n) = \sigma^2 tr(M) \nonumber \\
& = & \sigma^2 (n-k-1) \nonumber 
\end{eqnarray}
because $tr(M) = tr(I_n) - tr[X [(X'X)^{-1}X'] = n - tr(I_{k+1})$. Therefore,
\begin{equation}
E[s^2 | X] = E [ u'Mu | X)/(n-k-1) = \sigma^2
\end{equation}

\end{enumerate}

\pagebreak

\textit{Statistical inference}

With an additional assumption

5. {\it Normality}: conditional on {\bf X}, the $u_t$ are independently and identically distributed ($i.i.d.$) as Normal[$0,\sigma^2$]. The vector of errors {\bf u}, conditional on {\bf X}, is distributed multivariate Normal[$0, \sigma^2 I_n$].

we may present the theorem

{\it Normality of $b$}: Under assumptions 1,2,3,4,5, $b$ conditional on {\bf X} is distributed as multivariate Normal[$\beta, \sigma^2 (X'X)^{-1}$].

with the corollary that under the null hypothesis, $t$-statistics have a $t_{n-k-1}$ distribution. 

Under normality, $b$ is the {\it minimum variance unbiased estimator} of $\beta$, conditional on {\bf X}, in the sense that it reaches the {\it Cramer-Rao lower bound} (CRLB) for the {\small VCE} of unbiased estimators of $\beta$. The CRLB defines the minimum variance possible for any unbiased estimator---linear or nonlinear. Since OLS\ reaches that bound, it is the most precise unbiased estimator available.


\end{document}
