Weighted Least Squares 2

Published on December 2016 | Categories: Documents | Downloads: 60 | Comments: 0 | Views: 336
of 93
Download PDF   Embed   Report

Comments

Content


Weighted Least-
Squares Regression
A technique for correcting the
problem of heteroskedasticity by log-
likelihood estimation of a weight that
adjusts the errors of prediction
Weighted Least-Squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
Key Concepts
*****
Weighted Least-Squares Regression
OLS
Parameter estimates as:
Unbiased
Efcient
BLUE
Theoretical Sampling distribution of b
Standard error of b
Relationship between the standard error of b and:
The variance of X
The residual sum of squares
The sample size
Gauss-Markov Theorem
Assumptions about the errors (e) in regression analysis and the
consequences of their violation:
e is uncorrelated with X
e has the same variance across all levels of X
The values of e are independent of each other
e is normally distributed
The concepts of homoskedasticity and heteroskedasticity of the
error distributions
The concept of autocorrelation or serial correlation
Spurious relationships
Collinear relationships
Intervening relationships
Techniques for identifying heteroskedasticity
Graphic
Statistical
White’s Test for heteroskedasticity
Rezidualizing a variable
Techniques for identifying WLS weights
Theory, the literature, or prior experience
Regression of e
2
on X and transformation
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
2
Log-likelihood estimation of w
i
SPSS weight estimation procedure
SPSS WLS>> procedure
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
3
Overview
 Theoretical sampling distribution of b
 Assumptions about errors in regression
 Identifying heteroskedasticity
 The concept of weighted least-squares
regression
 Methods for estimating weights
 Regressing e
i
2
on X
 Log-likelihood estimation of weights
 Using WLS>> command in SPSS
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
4
References
White, Halbert (1980) A heteroskedasticity-consistent covariance matrix estimator and a
direct test for heteroskedasticity. Econometrica 48:817-838.
Graybill, Fraklin A. and Iyer, Hariharan K. (1994) Regression Analysis: Concepts and
Applications. Duxbury Press 571-592.
Freund, Rudolf J. and Wilson, William J. (1998) Regression Analysis: Statistical
Modeling of a Response Variable. Academic Press 378-382.
McClendon, McKee J. (1994) Multiple Regression and Causal Analysis. F. E. Peacock
Publishers, Inc. 138-146, 174-181, 189-197.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
5
Violation of OLS Regression Assumptions
Y = a + b
1
X
1
+ b
2
X
2
+ … + b
k
X
k
OLS regression makes various assumptions about
the errors that result from a regression model.
If these assumptions are met …
One can assume that the estimates of the
regression constant (a) and the regression
coefcients (b
k
) are
Unbiased: Replications of the study will
yield values of a and b
k
which will be
distributed on either side of their respective
parameters α and β
k
Efcient: The standard errors of a and
b
k
will neither over- nor underestimate
their associated theoretical standard
errors
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
6
Violation of one or more of these assumptions may
lead to biased and/or inefcient estimates.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
7
Theoretical Sampling Distribution of b
1
Population
Y = α + βX 2
3
m
Theoretical sampling distribution of b b = β
b
68.26%
Theoretical standard error of b
σ
b
= (σ
ε
) / (S
X
n ) σ
ε
= Σ ( Y – Y)
2
/ N
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
8
S
X
= Σ (X – X)
2
/ N
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
9
The Theoretical Standard Error of b
σ
b
= (σ
ε
) / (S
X
n )
The standard error (σ
b
) is directly related
to the standard deviation of the errors produced by
the model (σ
ε
)
The greater the errors produced by the
model, the greater the standard error
of b
The standard error (σ
b
) is inversely related to
the standard deviation of the predictor
variable (S
X
)
As the variability of X increases, the
standard error of b decreases
The standard error (σ
b
) is inversely related to
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
10
the sample size (n)
As the sample size increases, the
standard error of b decreases
Estimation of the Theoretical
Standard Error of b
The theoretical standard error of b (σ
b
) is usually
estimated from a single sample, vis-à-vis a sampling
distribution of b.
SE
b
= (S
e
) / ( TSS
X
)
S
e
= RSS / (n- k)
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
11
TSS
X
= Σ (X – X)
2
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
12
Gauss-Markov Theorem
 b is an unbiased estimate of β. On repeated
estimates, the distribution of b will be centered
around β.
 The sampling distribution of b will be normal
if the samples are large and a sufcient number of
samples are taken.
 OLS provides the best linear unbiased estimate of
β (BLUE)
 “Best” means:
OLS provides the most unbiased and
efcient estimate of β.
Efciency refers to the size of the
standard error of b (σ
b
); neither too large
nor small.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
13
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
14
The Four Assumptions About
Regression Error
e = (Y – Y)
e = prediction error
 e is uncorrelated with X, the independence
assumption.
 e has the same variance (S
e
2
) across the
diferent levels of X, i.e. the variance of e is
homoskedastic v heteroskedastic.
 The values of e are independent of each
other, i.e. not autocorrelated or serially
correlated.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
15
 e is normally distributed.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
16
The Problem of the Correlation of e & X
Y = a + bX
Spurious relationship: e and X may be correlated
because Z is a common cause of X and Y. In this
case b is a biased estimate of β.
spurious relationship

X Y
Z
Collinear Relationship: If X
2
is correlated with X
1
& Y
but is not the cause of either, b
1
will be a biased
estimate of β
1
X
1
Y
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
17
X
2
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
18
Correlation of e with X ( con'd )
Intervening Relationship: X
2
intervenes in the
relationship between X
1
and Y. In this case b
1
will not
be a biased estimate of β, but:
It will refect both the direct and indirect
efects of X
1
on Y.
X
1
X
2
Y
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
19
Homoskedasticity of Errors (e) Over Levels
of X
The “dotted” lines represent the pattern of the
dispersion of the residuals.
0 0
Homoskedastic Heteroskedastic (+)
R
XSe2
> 0.0

0 0
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
20
Heteroskedastic (-) Heteroskedastic
R
XSe2
< 0.0 Hour-Glass Distributed
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
21
Consequences of Heteroscedasticity
b will be an unbiased estimate of β, but SE
b
will be inefcient, too large or small.
SE
b
= Σ (Y-Y )
2
/ (n-k)
TSSx
If SE
b
is overestimated, (R
XSe2
< 0.0) …
b will not be an efcient estimate of β and a
Type II error may occur, since
t = (b / SE
b
)
If SE
b
is underestimated, (R
XSe2
> 0.0) …
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
22
b will not be an efcient estimate of β and a
Type I error may occur, since
t = (b / SE
b
)
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
23
Consequences of the Errors (e) Being
Autocorrelated
An example of a time series:
Y
Time
e at time t will likely be related to e at time
t-1, and so forth.
b will remain an unbiased estimate of β …
But the SEb will be biased and not efcient
since t = b / SE
b
If SE
b
is overestimated, a Type II error
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
24
may occur.
If SE
b
is underestimated, a Type I error
may occur.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
25
The Distribution of the Errors
OLS regression assumes that the errors of
prediction are normally distributed.
This can be tested by saving the errors and
Plotting …
A histogram or
A normal probability plot
Histogram of errors
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
26
Std. Dev = 4.04
Mean = 22.9
N = 70.00
35.0 32.5 30.0 27.5 25.0 22.5 20.0 17.5
Errors as a function of predictions
Predictions
30
20
10
0
Distribution of errors ( con'd )
Normal probability plot of errors
If the errors are non-normally distributed …
b may still be unbiased and efcient if
The homoskedasticity and independence
assumptions are met and the sample is large
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
27
Nora! pro"a"i!it# p!ot of errors
$"served %uu!ative Pro"a"i!it#
1.00 .75 .50 .25 0.00
1.00
.75
.50
.25
0.00
If the sample is small, the use of the t distribution
in determining the signifcance of b and its
confdence interval will be biased.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
28
Summary of Assumptions and
The Consequences of Their Violation
Assumption Violation Consequences
Errors correlated with X
Spurious relationship b biased estimate of β
Collinear relationship b biased estimate of β
Intervening relationship b unbiased estimate of β
but refects both direct &
indirect efects
Heteroskedastisity
(R
XSe2
≠ 0.0)
b unbiased but not
efcient, SE
b
too
small/large, Type I or II
error may result
Autocorrelated errors
b unbiased but not
efcient, SE
b
too
small/large, Type I or II
error may result
Errors non-normally
b may be unbiased if
homoskedasticity & in-
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
29
distributed dependence assumptions
met & N is large. If N is
small, t distribution may
be biased.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
30
Heteroskedastic Errors and
Weighted Least-Squares Regression
If the errors are heteroskedastically distributed
The SE
b
may inefcient, i.e. either too small or
large, which may lead to a Type I or II
error
Ways to detect heteroskedasticity
Scatterplot of X against Y (prior to analysis)
Scatterplot of predictions against residuals, either
unstandardized or standardized
Scatterplot of X against residuals
Scatterplot of X against the absolute value
of the residuals ( e )
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
31
Scatterplot of X against the squared
residuals (e
2
)
White’s Test for homoskedasticiy
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
32
Example
Scatterplot of X Against Y
Sentence length (Y) as a function of
drug dependency (X)
Heteroskedasticity
As drug score increases, the variability in
sentence increases
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
33
D&'S%$&E
12 10 ( ) 4 2 0
30
20
10
0
Example
Scatterplot of Predictions Against Residuals
Sentence length (Y) as a function of
drug dependency (X)
Heteroskedasticity
As predicted sentence becomes longer,
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
34
Predicted *a!ue
9 ( 7 ) 5 4 3 2
20
10
0
+10
variability in residuals becomes greater.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
35
White’s Test for Heteroskedasticity
White, Halbert (1980) A heteroskedasticity-consistent covariance matrix estimator and a
direct test for heteroskedasticity. Econometrica 48:817-838.
Step 1 Compute the regression equation
y = a + bX
Step 2 Save the residuals
(e = Y – Y)
Step 3 Square the residuals (e
2
)
Step 4 Regress e
2
on X and record R
2
from this
analysis (i.e. residualization of X)
Step 5 Calculate White’s chi-square (χ
2
)
χ
2
= (n) (R
2
)
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
36
H
o
= residuals are homoskedastic
n = number of cases
df = number of independent variables
White’s Test for Heteroskedasticity (cont.)
Example
The regression of sentence on dr_score
R
2
= 0.06517
χ
2
= n R
2
= (70) (0.06517) = 4.56
df = 1
p < 0.05
Statistical decision
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
37
Reject the H
o
that the residuals are
homoskedastic
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
38
How Does One Correct the Problem of
Heteroskedastic Errors?
Solution: Weighted Least-Squares Regression (WLS
Regression)
The logic of WLS Regression
Find a weight (w
i
)
That can be used to modify the infuence of
large errors on the estimation of …
The ‘best’ ft values of …
The regression constant (a)
The regression coefcients (b
k
)
OLS is designed to minimize: Σ (Y – Y)
2
In WLS, values of a and b
k
are estimated
which minimize RSS = Σ w
i
(Y – Y)
2

Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
39
This process has the efect of minimizing the
infuence of a case with a large error on the
estimation of a and b
k
And maximizing the infuence of a case with a
small error on the estimation of a and b
k
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
40
Techniques for Estimating a Suitable
Value of a Weight w
i
 From theory, the literature, or experience gained in
prior research.
Rarely will this approach prove successful,
except by trial and error
 Estimate w
i
by regressing e
2
on the ofending
independent variable X and …
Transforming the values of X and Y.
This is called residualizing the variable X.
 Use log-likelihood estimation to determine a
suitable value of w
i

This can be done in SPSS using the
regression weight estimation procedure
coupled with the WLS>> procedure.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
41
 In the following case study both the residualizing
and SPSS WLS>> procedures will be demonstrated.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
42
An Example
*****
The Relationship Between Drug Dependency &
Length of Sentence
The model
Sentence = a + b (drug_score)
The results
Sentence = 1.97 + 0.6438 (drug_score)
For this model to be BLUE, the residuals must be
homoskedastic.
Q Are the residuals homoskedastic?
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
43
An Example (cont.)
Scatterplot of the residuals
Notice how the residuals become larger the
greater the degree of drug dependency. These
are heteroskedastic residuals.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
44
,eteros-edastic &esidua!s
Standardi.ed Predicted *a!ue
1.5 1.0 .5 0.0 +.5 +1.0 +1.5 +2.0
5
4
3
2
1
0
+1
+2
Solving the Problem of Heteroskedasticity
Solution
Residualize the ofending variable X
Steps in the process
1. Plot X against Y to determine the presence of
heteroskedasticity
2. Estimate the following regression equation
and save the residuals (e = Y – Y). In SPSS
the residuals appear as res_1
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
45
Y = a + bX
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
46
Solving the Problem of Heteroskedasticity (cont.)
3. Square the residuals
e
2
= (res_1)
2
= residsq
4. Regress residsq on X and save the predicted
residsq, in SPSS this is called pre_2
Residsq = a + bX
5. Transform X and Y, and compute a weight w
i

called wtsqroot
wtX = X / pre_2
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
47
wtY = Y / pre_2
wtsqroot = 1 / pre_2
Solving the Problem of Heteroskedasticity (cont.)
6. Estimate the following weighted regression
equation through the origin, i.e. with a
regression constant equal to 0.0
wtY = a(wtsqroot) + b(wtX)
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
48
Step 1
Sentence length as a function of
drug dependency
SPSS scatterplot of sentence as a function of
dr_score . This can only be done when there are 2
or less IV.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
49
Sentence as a function of dru/ dependenc#
Dru/ Dependenc#
12 10 ( ) 4 2 0
30
20
10
0
Heteroskedastic The variability in sentence
length increases as the degree of drug dependency
increases.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
50
Step 2
Regress sentence length on drug
dependency, save the residuals (res_1) and
the predictions (pre_1)
sentence = 1.975 + 0.644 dr_score
R
2
= 0.12 (F = 9.24, p = 0.003)
SPSS results for Step 2
Regression
Variables Entered/Removed
b
DR_SCOR
E
a
. Enter
Model
1
Variables
Entered
Variables
Removed Method
All requested variables entered.
a.
Dependent Variable: SE!ECE
b.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
51
Step 2 (cont.)
Model Summary
b
."#$
a
.1%& .1&' #.$(1$
Model
1
R R Square
Ad)usted
R Square
Std. Error o*
the Estimate
+redi,tors: -Constant./ DR_SCORE
a.
Dependent Variable: SE!ECE
b.
ANOVA
b
%&%.01$ 1 %&%.01$ 1.%#& .&&"
a
1#1&."0$ $( %1.11'
1$1%.('1 $1
Re2ression
Residual
!otal
Model
1
Sum o*
Squares d* Mean Square 3 Si2.
+redi,tors: -Constant./ DR_SCORE
a.
Dependent Variable: SE!ECE
b.
Coefficients
a
1.1'0 1.#%0 1."($ .1'&
.$## .%1% ."#$ ".&#& .&&"
-Constant.
DR_SCORE
Model
1
4 Std. Error
5nstandardi6ed
Coe**i,ients
4eta
Standardi
6ed
Coe**i,ien
ts
t Si2.
Dependent Variable: SE!ECE
a.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
52
Casewise Diagnostics
a
".10$ %0.&&
Case umber
$&
Std. Residual SE!ECE
Dependent Variable: SE!ECE
a.
Step 2 (cont.)
Residuals Statistics
a
%.$1(0 (.#1%( 0.10'1 1.'1"% '&
7'.#1%( 1(.01($ 7'.$1E71' #.$#'0 '&
71.1#1 1.#"" .&&& 1.&&& '&
71.0(" ".10$ .&&& .11" '&
+redi,ted Value
Residual
Std. +redi,ted Value
Std. Residual
Minimum Ma8imum Mean Std. Deviation
Dependent Variable: SE!ECE
a.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
53
N.B. The residuals are heteroskedastic. Compare
this scatterplot with the scatterplot of sentence as a
function of dr_score. Notice that the patters are the
same.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
54
,eteros-edastic &esidua!s
Standardi.ed Predicted *a!ue
1.5 1.0 .5 0.0 +.5 +1.0 +1.5 +2.0
5
4
3
2
1
0
+1
+2
Step 3
Calculate the squared residuals
In SPSS, the unstandardized residuals are saved as
res_1.
Step 3 involves squaring the residuals by use of the
data transformation procedure in SPSS.
squared residual = (res_1)
2
= residsq
The SPSS syntax for this transformation is as
follows:
Residsq = res_1**2
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
55
The steps used in this transformation process are
described in the case study associated with this
module.
Step 3 (cont.)
SPSS results for Step 3
pre_1 res_1 residsq
7.76901 -6.76901 45.82
8.41282 -7.41282 54.95
8.41282 -7.41282 54.95
6.48139 -5.48139 30.05
7.76901 -5.76901 33.28
7.12520 -5.12520 26.27
7.76901 -5.76901 33.28
8.41282 -6.41282 41.12
5.83758 -2.83758 8.05
7.76901 -4.76901 22.74
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
56
2.61852 -.61852 .38
2.61852 .38148 .15
3.26233 1.73767 3.02
5.19377 1.80623 3.26
8.41282 -.41282 .17
7.76901 1.23099 1.52
7.12520 2.87480 8.26
6.48139 5.51861 30.46
7.12520 6.87480 47.26
7.12520 7.87480 62.01
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
57
Step 4
Regress the squared residuals on the independent
variable dr-score and save the predictions as (pre_2)
residsq = -7.587 + 4.6685 dr_score
R
2
= 0.065 (F = 4.74, p = 0.0329)
This process is called residualizing a variable.
By OLS defnition, the residuals (residsq)represent
the variance in Y that is unrelated to X.
Therefore, there should be no signifcant relationship
between X and residsq.
If there is, one or more OLS regression
assumptions have been violated.
In this case, the violated assumption is the
homoskedasticity of the residuals.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
58
Step 4 (cont.)
SPSS results for Step 4
Regression
Variables Entered/Removed
b
DR_SCOR
E
a
. Enter
Model
1
Variables
Entered
Variables
Removed Method
All requested variables entered.
a.
Dependent Variable: RES9DS:
b.
Model Summary
b
.%00
a
.&$0 .&01 #'."10"
Model
1
R R Square
Ad)usted
R Square
Std. Error o*
the Estimate
+redi,tors: -Constant./ DR_SCORE
a.
Dependent Variable: RES9DS:
b.
ANOVA
b
1&$#(.(&% 1 1&$#(.(&% #.'#1 .&""
a
10%'#1.# $( %%#$."10
1$""1(.% $1
Re2ression
Residual
!otal
Model
1
Sum o*
Squares d* Mean Square 3 Si2.
+redi,tors: -Constant./ DR_SCORE
a.
Dependent Variable: RES9DS:
b.
Coefficients
a
7'.0(' 1#.#%% 7.0%$ .$&1
#.$$1 %.1## .%00 %.1'' .&""
-Constant.
DR_SCORE
Model
1
4 Std. Error
5nstandardi6ed
Coe**i,ients
4eta
Standardi
6ed
Coe**i,ien
ts
t Si2.
Dependent Variable: RES9DS:
a.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
59
Casewise Diagnostics
a
$.'&$ "#%.1#
Case umber
$&
Std. Residual RES9DS:
Dependent Variable: RES9DS:
a.
Step 4 (cont.)
Residuals Statistics
a
7%.11(1 "1.&1'1 %1.%1&( 1%.#%"& '&
7"(.1%'0 "1'.(#$$ #.1'#E710 #'.&0&$ '&
71.1#1 1.#"" .&&& 1.&&& '&
7.(%1 $.'&$ .&&& .11" '&
+redi,ted Value
Residual
Std. +redi,ted Value
Std. Residual
Minimum Ma8imum Mean Std. Deviation
Dependent Variable: RES9DS:
a.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
60
Step 5
Compute the absolute value of pre_2 and three new
variables wtsent, wtdrug and the weight wtsqroot.
wtsent = (sentence) / abspre_2
wtdrug = (dr_score) / abspre_2
wtsqroot = (1) / abspre_2
pre_2 from the previous step is the information in the
squared residuals (residsq) that is related to the IV
dr_score.
Dividing sentence and dr_score by pre_2 reduces
the infuence of extreme values on the estimation of
a and b.
Finally a third transformation is performed by
creating the variable “wtsqroot.”
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
61
This will serve as a weighting factor.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
62
Step 5 (cont.)
SPSS results for Step 5
abspre_2 wtsent wtdr_sco wtsqroot
34.43. 17 1.53 .17
39.10. 16 1.60 .16
39.10. 16 1.60 .16
25.09 .20 1.40 .20
34.43 .34 1.53 .17
29.76 .37 1.47 .18
34.43 .34 1.53 .17
39.10 .32 1.60 .16
20.42 .66 1.33 .22
34.43 .51 1.53 .17
20.42 .66 1.33 .22
39.10 1.28 1.60 .16
34.43 1.53 1.53 .17
29.76 1.83 1.47 .18
25.09 2.40 1.40 .20
29.76 2.57 1.47 .18
29.76 2.75 1.47 .18
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
63
Step 6
Compute the WLS regression
wtsent = a(wtsqroot) + b (wtdrug)
Results:
wtsent = 1.159 (wtsqroot) + 0.7833 (wtdrug)
R
2
= 0.674 (F = 70.29, p < 0.0001)
SPSS results for Step 6
Regression
Variables Entered/Removed
bc
;!S:RO
O!/
;!DR_SC
O
a
. Enter
Model
1
Variables
Entered
Variables
Removed Method
All requested variables entered.
a.
Dependent Variable: ;!SE!
b.
<inear Re2ression throu2h the Ori2in
,.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
64
Step 6 (cont.)
Model Summary
cd
.(%1
b
.$'# .$$# .1$00
Model
1
R R Square
a
Ad)usted
R Square
Std. Error o*
the Estimate
3or re2ression throu2h the ori2in -the no7inter,ept
model./ R Square measures the proportion o* the
variabilit= in the dependent variable about the ori2in
e8plained b= re2ression. !his CAO! be ,ompared
to R Square *or models >hi,h in,lude an inter,ept.
a.
+redi,tors: ;!S:ROO!/ ;!DR_SCO
b.
Dependent Variable: ;!SE!
,.
<inear Re2ression throu2h the Ori2in
d.
ANOVA
cd
1"1.&#1 % $0.0%& '&.%(0 .&&&
a
$"."1& $( .1"%
11#.#"1
b
'&
Re2ression
Residual
!otal
Model
1
Sum o*
Squares d* Mean Square 3 Si2.
+redi,tors: ;!S:ROO!/ ;!DR_SCO
a.
!his total sum o* squares is not ,orre,ted *or the ,onstant be,ause the ,onstant is
6ero *or re2ression throu2h the ori2in.
b.
Dependent Variable: ;!SE!
,.
<inear Re2ression throu2h the Ori2in
d.
Coefficients
ab
.'(" .1#& .$"' 0.01' .&&&
1.101 .$&0 .%1( 1.11' .&01
;!DR_SCO
;!S:ROO!
Model
1
4 Std. Error
5nstandardi6ed
Coe**i,ients
4eta
Standardi
6ed
Coe**i,ien
ts
t Si2.
Dependent Variable: ;!SE!
a.
<inear Re2ression throu2h the Ori2in
b.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
65
Step 6 (cont.)
Casewise Diagnostics
ab
".'1$ #.11
Case umber
$&
Std. Residual ;!SE!
Dependent Variable: ;!SE!
a.
<inear Re2ression throu2h the Ori2in
b.
Residuals Statistics
ab
1.1"$1 %.&$&$ 1."0(& .1$'1 '&
71."&#$ ".$$#( 1.#("E7&" .10(0 '&
71."1' #.1(0 .&&& 1.&&& '&
71."01 ".'1$ .&&% .11" '&
+redi,ted Value
Residual
Std. +redi,ted Value
Std. Residual
Minimum Ma8imum Mean Std. Deviation
Dependent Variable: ;!SE!
a.
<inear Re2ression throu2h the Ori2in
b.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
66
N.B. The heteroskedasticity has been reduced.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
67
&esidua!s of 0tsent re/ressed on 0tdru/
0tdru/
2.2 2.0 1.( 1.) 1.4 1.2 1.0
4
3
2
1
0
+1
+2
Comparison of the OLS v Residualized
Regression Models
Compare of scatterplots, notice the substantial
reduction of heteroskedasticity
Statistical results
Method a b SE
a
SE
b
R
2
p
OLS 1.975 0.644 1.425 0.212 0.1196 0.0034
Resid-
ualized
1.159 0.783 0.605 0.139 0.6740 0.0001
The residualized model is more efcient, SEs are smaller.
Comparison of 95% confdence intervals
Method
95% Confdence
Interval Diference
OLS 0.221 to 1.066 0.845
Residualized 0.504 to 1.062 0.558
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
68
N.B. The width of the residualized 95% confdence interval
is less than that of the OLS interval.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
69
An Alternative Procedure for Correcting
Heteroskedasticity
Log-Likelihood Estimation of w
i

If it can be assumed that the variance in the DV
Is proportional to the IV,
Log-likelihood estimation can be used to
estimate w
I
.
In this case it is assumed that …
S
y
2
∝ (X)
w
or S
y
2
∝ (1 / X
w
)
(∝ is read “proportional to”)
In log-likelihood estimation of w
i
, the question is:
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
70
What power of X, i.e. w
i
, is most likely to have
produce the proportional relationship between
S
y
2
and X ?
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
71
SPSS Weight Estimation and WLS>>
Regression Procedures
This procedure begins by using log-likelihood
estimation to iteratively determine a weight w
I

To be used in estimating the values of the
regression constant (a) and the regression
coefcient (b) …
Such that the RSS is minimized.
RSS = Σ [ (1 / X
w
i
) (Y – Y)
2
]
This may solve the heteroskedasticity problem if:
S
y
2
∝ (X)
w
or S
y
2
∝ (1 / X
w
)
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
72
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
73
Step 1
Estimation of the weight w
i
using the SPSS weight
estimation procedure
The result
The most likely weight = 1.8
The variance in sentence is estimated to be …
S
y
2
= (dr_score)
1.8
Regression equation
sentence = 0.94 + 0.83 (dr_score)
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
74
For a drug score of 6,the prediction would be
sentence = 0.94 + 0.83 (6) = 5.92 years
Step 1 (cont.)
Examination of weights for individual subjects
Subject dr_score Weight
Jones 10
1/(10)
1.8
= 0. 01585
Smith 1
1/(1)
1.8
= 1.00
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
75
Step 1 (cont.)
SPSS results
!eig"ted #east S$uares
MODEL: MOD_1.
Source variable.. DR_SCORE Dependent variable.. SENTENCE
Log-lieli!ood "unction # -$%&.&'%'() *O+ER value # -).'''
Log-lieli!ood "unction # -$%%.&11')% *O+ER value # -$.,''
Log-lieli!ood "unction # -$%1.&%--&% *O+ER value # -$.-''
Log-lieli!ood "unction # -$),.,11)&- *O+ER value # -$.%''
Log-lieli!ood "unction # -$)..('.--( *O+ER value # -$.$''
Log-lieli!ood "unction # -$)).')'$,. *O+ER value # -$.'''
Log-lieli!ood "unction # -$)'.1,-1(, *O+ER value # -1.,''
Log-lieli!ood "unction # -$$&.)&%--, *O+ER value # -1.-''
Log-lieli!ood "unction # -$$%..(&$(& *O+ER value # -1.%''
Log-lieli!ood "unction # -$$1.,.-1') *O+ER value # -1.$''
Log-lieli!ood "unction # -$1(.1.)-), *O+ER value # -1.'''
Log-lieli!ood "unction # -$1-.%()1-& *O+ER value # -.,''
Log-lieli!ood "unction # -$1).,&,(%' *O+ER value # -.-''
Log-lieli!ood "unction # -$11.)1-.&- *O+ER value # -.%''
Log-lieli!ood "unction # -$',.,1).-. *O+ER value # -.$''
Log-lieli!ood "unction # -$'-.)&(,,- *O+ER value # .'''
Log-lieli!ood "unction # -$'%.'$,&1$ *O+ER value # .$''
Log-lieli!ood "unction # -$'1.&&&$%' *O+ER value # .%''
Log-lieli!ood "unction # -1((.-%&-.) *O+ER value # .-''
Log-lieli!ood "unction # -1(&.--,)1, *O+ER value # .,''
Log-lieli!ood "unction # -1(..,&.$-' *O+ER value # 1.'''
Log-lieli!ood "unction # -1(%.)1)()( *O+ER value # 1.$''
Log-lieli!ood "unction # -1().'%1$$- *O+ER value # 1.%''
Log-lieli!ood "unction # -1($.1$&$-- *O+ER value # 1.-''
Log-lieli!ood "unction # -1(1.-.--,' *O+ER value # 1.,''
Log-lieli!ood "unction # -1(1.&$,1,( *O+ER value # $.'''
Log-lieli!ood "unction # -1($.%.1.). *O+ER value # $.$''
Log-lieli!ood "unction # -1().(%'.(. *O+ER value # $.%''
Log-lieli!ood "unction # -1(-.)'$)&. *O+ER value # $.-''
Log-lieli!ood "unction # -1((.-$)'-' *O+ER value # $.,''
Log-lieli!ood "unction # -$').(.%$$( *O+ER value # ).'''
T!e /alue o0 *O+ER Ma1i2i3ing Log-lieli!ood "unction # 1.,''
log-likelihood estimated weight wi = 1.8
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
76
Step 1 (cont.)
Estimation of the weighted regression model
Source variable.. DR_SCORE *O+ER value # 1.,''
Dependent variable.. SENTENCE
Li4t5i4e Deletion o0 Mi44ing Data
Multiple R .-)-%,
R S6uare .%'.1'
7d8u4ted R S6uare .)(-)-
Standard Error .,%)&.
7nal94i4 o0 /ariance:
D" Su2 o0 S6uare4 Mean S6uare
Regre44ion 1 )$.(-.%-- )$.(-.%--
Re4idual4 -, %,.%'(,$- .&11('(
" # %-.)'.&$ Signi0 " # .''''
------------------ /ariable4 in t!e E6uation ------------------
/ariable : SE : :eta T Sig T
DR_SCORE .,$,%&' .1$1&%& .-)-%&, -.,'. .''''
;Con4tant< .()((&& .)(%.,, $.),$ .'$''
Log-lieli!ood "unction # -1(1.-.--,'
T!e 0ollo5ing ne5 variable4 are being created:
Na2e Label
+=T_1 +eig!t 0or SENTENCE 0ro2 +LS> MOD_1 DR_SCORE?? -1.,''
Weighted equation
Sentence = 0.9399 + 0.8285 (dr_score)
Unweighted equation
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
77
Sentence = 1.97 + 0.6438 (dr_score)
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
78
Step 2
Plot the relationship between dr_score
and the weight w
i

The heteroscdasticity problem …
Recall the previous scatterplot: as the value of drug scores
increases, the variance in sentences increases as well.
The log-likelihood estimated weight is such that
As the value of drug score increases, the weight adjusted
drug score (wgt_1) decreases.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
79
Scatterp!ot of ad1usted dr'score and dr'score
2ei/3t ad1usted dr'score=145dr'score6 771.(
D&'S%$&E
12 10 ( ) 4 2 0
1.2
1.0
.(
.)
.4
.2
0.0
The Efect of the Log-Likelihood
Estimated Weight on the
Regression Errors
OLS regression estimates the best ft values of a
and b by minimizing …
RSS = Σ (Y – Y)
2
WLS regression estimates the best ft values of a
and b by minimizing …
RSS = Σ w
i
(Y – Y)
2
For ofender Jones with a drug score of 10
e
2
= (1/10
1.8
) (Y – Y)
2
e
2
= (0.01585) (Y – Y)
2
Since prediction errors increase as drug score
increases …
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
80
The weight of 1.8 reduces the efect of large
errors on the RSS providing a more efcient
estimate of the SE
b
.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
81
Step 3
The SPSS WLS>> in linear regression
If an appropriate weight w
i
is already known by
another means …
The WLS>> procedure in SPSS linear
regression can be use instead of the SPSS
weight estimation procedure
The procedure
Simply specify the regression model
Enter the known weight-variable under the
WLS>> command and estimate the
model
In this case, the weight variable wgt _1 from
Step 2 will be used
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
82
Step 3 (cont.)
The results of the WLS>> analysis using a weight-
variable wgt_1, w
i
= 1.8, with regression through the
origin
R
2
= 0.668, F = 139.14, p = 0.0001
sentence = 1.036 (dr_score)
SE
b
= ± 0.087
N.B. Since this model does not include a constant
(a), the R
2
and the other statistical results can not be
compared with the associated values of a model that
does use a constant (a).
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
83
Step 3 (cont)
SPSS results
Regression
Variables Entered/Removed
bc
DR_SCOR
E
a
. Enter
Model
1
Variables
Entered
Variables
Removed Method
All requested variables entered.
a.
Dependent Variable: SE!ECE
b.
;ei2hted <east Squares Re2ression 7
;ei2hted b= ;ei2ht *or SE!ECE *rom
;<S/ MOD_1 DR_SCORE?? 71.(&&
,.
Model Summary
bc
.$"$
a
.#&0 ."1$ .(#"'
Model
1
R R Square
Ad)usted
R Square
Std. Error o*
the Estimate
+redi,tors: -Constant./ DR_SCORE
a.
Dependent Variable: SE!ECE
b.
;ei2hted <east Squares Re2ression 7 ;ei2hted b=
;ei2ht *or SE!ECE *rom ;<S/ MOD_1
DR_SCORE?? 71.(&&
,.
ANOVA
bc
"%.1$0 1 "%.1$0 #$."&$ .&&&
a
#(.#1& $( .'1%
(1."'0 $1
Re2ression
Residual
!otal
Model
1
Sum o*
Squares d* Mean Square 3 Si2.
+redi,tors: -Constant./ DR_SCORE
a.
Dependent Variable: SE!ECE
b.
;ei2hted <east Squares Re2ression 7 ;ei2hted b= ;ei2ht *or SE!ECE *rom
;<S/ MOD_1 DR_SCORE?? 71.(&&
,.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
84
Step 3 (cont)
Coefficients
ab
.1#& ."10 %."(% .&%&
.(%( .1%% .$"$ $.(&0 .&&&
-Constant.
DR_SCORE
Model
1
4 Std. Error
5nstandardi6ed
Coe**i,ients
4eta
Standardi
6ed
Coe**i,ien
ts
t Si2.
Dependent Variable: SE!ECE
a.
;ei2hted <east Squares Re2ression 7 ;ei2hted b= ;ei2ht *or SE!ECE *rom
;<S/ MOD_1 DR_SCORE?? 71.(&&
b.
Residuals Statistics
bc
1.'$(# 1.%%#' $.&$#' %.%&#$ '&
7(.%%#' 1(.%$&' 7.1&'0 #.$'"# '&
. . . . &
. . . . &
+redi,ted Value
Residual
Std. +redi,ted Value
a
Std. Residual
a
Minimum Ma8imum Mean Std. Deviation
ot ,omputed *or ;ei2hted <east Squares re2ression.
a.
Dependent Variable: SE!ECE
b.
;ei2hted <east Squares Re2ression 7 ;ei2hted b= ;ei2ht *or SE!ECE *rom
;<S/ MOD_1 DR_SCORE?? 71.(&&
,.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
85
Step 3 (cont)
Saved predicted & residual values, and the weighted
values of dr_score (i.e. wgt_1)
wtg_1 = 1 / (dr_score)
1.8
wgt_1 pre_1 res_1
%&'(') *%+(),' -.%+(),'
%&'/*/ (%,,0)* -*%,,0)*
%&'/*/ (%,,0)* -*%,,0)*
%&+&', )%.+(,. -/%.+(,.
%&'(') *%+(),' -)%+(),'
%&,+)* .%/)..0 -/%/)..0
%&'(') *%+(),' -)%+(),'
%&'/*/ (%,,0)* -.%,,0)*
%&+(./ /%('&*& -,%('&*&
%&,+)* .%/)..0 ,%0+,,)
%&+&', )%.+(,. /%,)&.+
%&,+)* .%/)..0 )%0+,,)
%&,+)* .%/)..0 .%0+,,)
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
86
Step 4
Variable transformations and
plot of the residuals
Unfortunately, the weighted residuals and predictions
produced by the SPSS weight estimation and
WLS>> procedures …
Can not be directly graphed from the saved
residuals and predictions
The residuals and the predictions must frst be
transformed as follows:
Transformed residual = (res_1) (wt)
0.5
Transformed prediction = (pre_1) (wt)
0.5
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
87
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
88
Step 4 (cont.)
SPSS results
Compare the degree of heteroskedasticity in this
scatterplot with …
The plot of the residuals from the un-
weighted regression model.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
89
Scatterp!ot of t3e 8ransfored &esidua!s
8ransfored 2ei/3ted Predictions
1.4 1.3 1.2 1.1 1.0
4
3
2
1
0
+1
+2
Notice the substantial change in the degree of
heteroskedasticity.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
90
Step 4 (cont.)
Transformed variables transres and transpre
transres = res_1*sqrt(wgt_1)
transpre = pre_1*sqrt(wgt_1)
transres transpre
-'%&, '%')
-'%&0 '%')
-'%&0 '%')
-'%&& '%'.
-%*( '%')
-%*) '%')
-%*( '%')
-%(' '%')
%+. '%')
%(' '%'.
%(( '%')
'%'0 '%')
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
91
Comparison of Results:
OLS, Residualized , and Log-Likelihood
Models
Method a b SE
a
SE
b
OLS
1.975 0.644 1.425 0.212
Rezidu-
alized
1.159 0.783 0.605 0.139
Log
Like-
lihood
0.940 0.828 0.394 0.121
N.B. The standard errors of the residualized &
log-likelihood models are lower than the OLS model.
The log-likelihood model produces smaller standard
errors than the residualized model.
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
92
Weighted Least-squares Regression: Charles M. Friel Ph.D., Criminal Justice Center, Sam Houston State University
93

Sponsor Documents

Or use your account on DocShare.tips

Hide

Forgot your password?

Or register your new account on DocShare.tips

Hide

Lost your password? Please enter your email address. You will receive a link to create a new password.

Back to log-in

Close