Least Squares Estimators When trying to determine the mathematical(deterministic) relationship between parameters in a dataset, a commonly encountered problem is ”fitting” a curve that best satisfies the data. One method is employing the use of least squares. Consider Yi − (β0 − β1 Xi ) the deviation of Yi from its expected value(i.e. the model error term). The least squares criterion Q is then defined as the sum of n squared deviations. That is
n
Q=
i=1
(Yi − (β0 − β1 Xi ))2
The idea of the method of least squares is to find the values of the esˆ ˆ timators β0 and β1 that minimizes the criterion Q for a given sample of observations. The normal equations can be simultaneously solved by finding the first order partial derivatives of Q with respect to either β0 or β1 . Thus, we obtain δQ = −2 δβ0 δQ = −2 δβ1
n
(Yi − (β0 − β1 Xi ))
i=1 n
Xi (Yi − (β0 − β1 Xi ))
i=1
ˆ ˆ Setting these partial derivatives to 0,and denoting β0 and β1 as the particular values that minimizes Q, we obtain
n
−2
i=1 n
(Yi − (β0 − β1 Xi )) = 0 Xi (Yi − (β0 − β1 Xi )) = 0
−2
i=1
This simplifies to 1
n
(Yi − (β0 − β1 Xi )) = 0
i=1 n
Xi (Yi − (β0 − β1 Xi )) = 0
i=1
ˆ ˆ Expressions for β0 and β1 can then be derived by further simplifying these equations. Homework 2 Q.1 I’ll omit some of the details since it’s simply a case of applying the method of least squares which is explained above. That’s not to say that you should also omit details in your own attempts. Define Q as
n
Q=
i=1
(Yi − µ)2
Then δQ = −2 δµ
n
(Yi − µ)
i=1
Equating the first order derivative to 0 and letting µ be the value that ˆ minimizes Q, we get
n
−2
i=1 n
(Yi − µ) = 0 ˆ (Yi − µ) = 0 ˆ
i=1 n
Yi − nˆ = 0 µ
i=1
µ= ˆ
n i=1
Yi
n µ=Y ˆ ¯ 2
EXTRA The least squares solution can also be obtained using the associated normal system of matrices M tM v = M ty Then the unique solution v ∗ is given by v ∗ = (M t M )−1 M t y