Chapters 1 and 2
Content
Methods of Comparison:
Controlled Experiments and
Observational Studies
Math 1680
Overview
Introduction
Controlled Experiments
Minimizing Bias
Observational Studies
Association, not Causation
Comparing Rates
Salk Vaccine Trial
Breast Cancer Screening
Summary
Introduction
Suppose a company wanted to market a
weight-loss pill
After design is complete, the company still
needs to answer one basic question
–
Does the pill actually work?
How can the company determine this?
Introduction
One method would be to give the pill to everyone
who wants to lose weight and see if they actually do
–
What’s wrong with this approach?
–First of all, the cost of giving everyone the pill is
prohibitive, and there may be dangerous side effects
for the untested pill.
–Secondly, people who want to lose weight may be
exercising or eating more healthily, which could
overshadow the effects of the pill (This is an example
of a confounding factor.)
Controlled Experiments
To test the pill, two groups are created from a
sample of people
–
–
Control group is given a placebo
Treatment group is given the pill
Controlled Experiments
Why is it important that the people don’t
know which group they are in?
It has been shown repeatedly that a patient's
symptoms can be alleviated by an otherwise
ineffective treatment if the individual expects or
believes that it will work. Conversely, members
of a control group knowingly receiving an inert
substance tend to report a worsening of
symptoms. This phenomenon is called the
placebo effect.
Controlled Experiments
The most ideal form of experiment is one
where neither the subjects nor the evaluators
know who received the treatment
–
This is a double-blind experiment
This design provides the least opportunity for
biased results
Controlled Experiments
After a period of time, the company checks to
see how much weight people in each group
have lost
–
–
If there is a significant difference between the
groups, assume the pill was the reason
If there is no significant difference between the
groups, assume the pill had no effect
Minimizing Bias
What if the control group and the treatment group
differ in some way other than the treatment?
–
Then we have confounding factors in the study, and the
results will be biased and/or invalid
Confounding factors can include…
–
–
–
–
–
–
Age
Race
Gender
Socio-economic status (SES)
Location
Political/religious orientation
Minimizing Bias
In the pill example, suppose all of the men were
placed into the treatment group and all of the women
were placed into the control group
–
What is wrong with this design?
The pill may interact differently with people
based on gender. For example, suppose
that high levels of testosterone diminish the
effect of the pill. Then the results would be
biased against the pill’s effectiveness.
Minimizing Bias
In order to minimize the risk of bias,
researchers assign people to each group
randomly from the pool of subjects
–
If enough people are in the pool, the groups will
look similar in terms of age, race, gender, or any
other demographic which could be a confounding
element
Observational Studies
How would you design a controlled
experiment to test if second-hand smoke
causes lung cancer?
Answer: You can’t, really. The time scale is too large and
it would be virtually impossible to ensure that your
control group never got second-hand smoke exposure.
Observational Studies
In many studies, designing a controlled experiment
is simply not possible
–
–
In such cases, researchers must settle for observational
studies
Observational studies let the subjects select which group to
join
Observational studies should be placed under great
scrutiny because the control and treatment groups
may be quite different in many ways
–
These differences may introduce confounding influences on
the study results
Observational Studies
A TV commercial observes that volunteers
who switched to having to Cheerios for
breakfast reduced their cholesterol level by
4%. Does this show that Cheerios cereal is
effective in reducing your cholesterol level?
Not necessarily. People who
volunteer for such a study may
already be interested in improving
their health and may be taking steps
such as exercising daily. This could
lower their cholesterol level.
Observational Studies
One form of observational study involves
using historical controls to compare a
modern treatment with
–
–
The difference in conditions between past and
present can itself be a confounding factor
For example, in testing a polio vaccine, one would
not want to compare infection rates in the 1950’s
with infection rates in the 1900’s because of the
large differences in standard of living and health
care quality in the two periods
Association, not Causation
An important limitation of all studies is that they can only show
the strength of association between two variables
–
–
Association may point to causation
For example, if exposure to a virus causes a disease, then people
who are exposed should be sicker than similar people who are not
exposed
However, a study alone cannot show causation
–
–
Human inference is needed to connect causal links
Generally, a casual link is accepted if
A scientific theory is derived which explains the association
Enough studies controlling all potential confounding factors are
performed, and all of them show the same association
Association, not Causation
Cervical cancer is more common among
women who have been exposed to the
herpes virus, according to many
observational studies. Is it fair to conclude
that the virus causes cervical cancer?
No. Both herpes and cervical cancer have been shown
to be sexually transmitted diseases. Women who have
many partners are at higher risk for both diseases than
women with few partners, so the number and type of
partners is a confounding influence which explains the
association.
Association, not Causation
Some studies find an association between liver
cancer and smoking. Does this mean smoking
causes liver cancer, or is there a confounding
influence at play?
People who smoke also tend to drink, so drinking is
associated with smoking. It has been shown
repeatedly that over consumption of alcohol can
cause liver cancer.
Comparing Rates
Sometimes the groups in comparison are of
different sizes
–
–
Rather than comparing the actual numbers, use
rates or proportions such as percents
Adjusting to rates allows for a more direct
comparison
Comparing Rates
In the U.S. in 1990, there were 2.1 million deaths
from all causes compared to 1.7 million in 1960
–
According to Census data, there were about 177 million
Americans in 1960 and about 250 million in 1990
True or false: The public’s health got worse over the
period 1960-1990.
False. Convert the death counts to percentage
rates. The death rate in 1960 was 0.96%, while
the death rate in 1990 was 0.84%. If anything,
public health improved.
Salk Vaccine Trial
In 1954, researchers tested the effectiveness of a
vaccine for polio
Approximately 400,000 children were selected
–
–
All children in the study had parental consent to be
vaccinated (Why?)
Children randomly divided into equal sized treatment and
control groups
Control group was given a placebo
Is this a controlled experiment or an observational
study?
Salk Vaccine Trial
Results
Study Group
Population
Polio Cases
Rate per
100,000
Vaccinated
200,745
57
28.4
Placebo
201,229
142
70.6
Was the vaccine effective?
Breast Cancer Screening
In 1963, an insurance company used 62,000
women (all insured by the plan) as subjects
–
–
–
Women ranged in age from 40 to 64
31,000 of the women were used as controls
The others were encouraged to undergo annual
screening
Roughly 20,200 women from the treatment group came
for annual screenings, the other 10,800 refused
After 5 years, the death rates were checked
Breast Cancer Screening
Is this a controlled experiment or an
observational study?
It depends on what questions you are trying to answer.
If you are only interested in the difference between the
treatment and control groups, then you are using the
results of a controlled experiment. However, when
you start drawing inferences between the screened
and refused groups, you are looking at an after-thefact observational study.
Breast Cancer Screening
Deaths (breast
cancer)
Deaths (all other
causes)
Number
Rate per
1,000
Number
Rate per
1,000
31,000
39
1.3
837
27
Examined
20,200
23
1.1
428
21
Refused
10,800
16
1.5
409
38
31,000
63
2
879
28
Treatment
Control
Researchers noted three things:
1. Screening had little impact on diseases other than breast cancer.
2. Poorer women were less likely to accept screening than richer ones.
3. Most diseases fall more heavily on the poor than on the rich.
Breast Cancer Screening
Deaths (breast
cancer)
Deaths (all other
causes)
Number
Rate per
1,000
Number
Rate per
1,000
31,000
39
1.3
837
27
Examined
20,200
23
1.1
428
21
Refused
10,800
16
1.5
409
38
31,000
63
2
879
28
Treatment
Control
Does screening save lives? Which numbers prove
your point?
Breast Cancer Screening
Deaths (breast
cancer)
Deaths (all other
causes)
Number
Rate per
1,000
Number
Rate per
1,000
31,000
39
1.3
837
27
Examined
20,200
23
1.1
428
21
Refused
10,800
16
1.5
409
38
31,000
63
2
879
28
Treatment
Control
Why is the death rate from all other causes in the whole
treatment group about the same as the rate for the control
group?
Breast Cancer Screening
Deaths (breast
cancer)
Deaths (all other
causes)
Number
Rate per
1,000
Number
Rate per
1,000
31,000
39
1.3
837
27
Examined
20,200
23
1.1
428
21
Refused
10,800
16
1.5
409
38
31,000
63
2
879
28
Treatment
Control
Why is the death rate from all other causes higher for the
“refused group” than the examined group?
Breast Cancer Screening
Deaths (breast
cancer)
Deaths (all other
causes)
Number
Rate per
1,000
Number
Rate per
1,000
31,000
39
1.3
837
27
Examined
20,200
23
1.1
428
21
Refused
10,800
16
1.5
409
38
31,000
63
2
879
28
Treatment
Control
Unlike most diseases, breast cancer afflicts the rich more
than the poor. Which numbers in the table confirm this
association?
Case Study: Child care and
behavior
Question: If parents choose to use child care,
are they more likely to see undesirable
behaviors in their children?
Question: How should one find out? By use
of controlled experiments, or by
observational study?
Case Study: Child care and
behavior
Answer: In 1991, a study commenced on 1364
infants (“subjects”) and followed them through their
sixth year in school. 12 years later, an article was
published. “the more time children spent in child care
from birth to age four-and-a-half, the more adults
tended to rate them, both at age four-and-a-half and
at kindergarten, as less likely to get along with
others, as more assertive, as disobedient, and as
aggressive.”
Case Study: Child care and
behavior
A summary of the study noted, “The study
authors noted that their study was not
designed to prove a cause and effect
relationship. …” (that more child care time
causes more aggressive behavior)
Possible confounding factors?
Perhaps… perhaps … perhaps…
Case Study: Child care and
behavior
Question: How about a controlled experiment
instead? (i.e. the researchers select children
to place in child care)
What are the advantages?
What problems might arise?
Summary: Basic Principles of
Statistical Experimental Design
Raise question to answer: Explanatory
variable(s) & Response variable(s).
Leading principle: Comparison.
Other principles: randomize, repeat (or
involve many subjects to reduce chance
variation)
Why:
Summary: Principles of Statistical
Experimental Design
Limitations: Statistical analysis of an experiment
cannot tell us how far the results will generalize to
other settings. (Experiment run in Massachusetts;
how about in the entire USA?)
However, the randomized comparative experiment,
because of its ability to give convincing evidence for
causation, is one of the most important ideas in
statistics.
More Elaborate Statistical Designs
Matched Pairs Design: Pair subjects with the
same age, sex, income, etc. and compare
their responses
Example: Compare two advertisements for
the same product
Common variation of MPD: impose both treatments
on the same subjects, so that each subject serves as
his or her own control. (See HW2, Q9)
More Elaborate Statistical Designs
Question: How does randomization work here?
Answer: Which one of a matched pair sees the first
ad is decided at random. Or, which treatment a
person gets first is decided at random if he or she
also serves as his or her own control.
Additional Example: Matched Pairs for the cell phone
experiment. (Details: trained in using simulator, order
in which a subject drives with and without the phone
be random, two drives be on separate days)
Summary
Statisticians use the method of comparison
–
To determine the effectiveness of a “treatment”, they usually
compare the responses of a treatment group with a control
group
If the control group is comparable to the treatment
group (apart from the treatment), then a significant
difference in the responses of the two groups is likely
to be due to the effect of the treatment
Summary
If the treatment group is different from the
control group with respect to other factors,
the effects of these other factors are likely to
be confounded with the effect of the
treatment
To make sure the treatment group is like the
control group, investigators put subjects into
either group at random
Summary
Whenever possible, the control group is given a
placebo
–
The response should be to the treatment itself rather than to
the idea of the treatment
In a double-blind experiment, the subjects do not
know whether they are in treatment or in control, and
neither do those who evaluate the responses
–
Guards against bias, both in the responses and in the
evaluations
Summary
In an observational study, the investigators
do not assign the subjects into treatment or
control groups
–
Subjects which have the condition are the
treatment group and the ones which do not are
the control
Observational studies can establish
association, which is not necessarily
causation
Summary
Observational studies are particularly susceptible to
confounding factors
–
This can make them very misleading about cause and effect
relationships
With observational studies, try to find out how the
subjects came to be in treatment or control
–
–
–
Are the groups comparable?
What factors are confounded with the treatment?
What adjustments (if any) did the investigators make to
control for these factors?
Controlled Experiments and
Observational Studies
Math 1680
Overview
Introduction
Controlled Experiments
Minimizing Bias
Observational Studies
Association, not Causation
Comparing Rates
Salk Vaccine Trial
Breast Cancer Screening
Summary
Introduction
Suppose a company wanted to market a
weight-loss pill
After design is complete, the company still
needs to answer one basic question
–
Does the pill actually work?
How can the company determine this?
Introduction
One method would be to give the pill to everyone
who wants to lose weight and see if they actually do
–
What’s wrong with this approach?
–First of all, the cost of giving everyone the pill is
prohibitive, and there may be dangerous side effects
for the untested pill.
–Secondly, people who want to lose weight may be
exercising or eating more healthily, which could
overshadow the effects of the pill (This is an example
of a confounding factor.)
Controlled Experiments
To test the pill, two groups are created from a
sample of people
–
–
Control group is given a placebo
Treatment group is given the pill
Controlled Experiments
Why is it important that the people don’t
know which group they are in?
It has been shown repeatedly that a patient's
symptoms can be alleviated by an otherwise
ineffective treatment if the individual expects or
believes that it will work. Conversely, members
of a control group knowingly receiving an inert
substance tend to report a worsening of
symptoms. This phenomenon is called the
placebo effect.
Controlled Experiments
The most ideal form of experiment is one
where neither the subjects nor the evaluators
know who received the treatment
–
This is a double-blind experiment
This design provides the least opportunity for
biased results
Controlled Experiments
After a period of time, the company checks to
see how much weight people in each group
have lost
–
–
If there is a significant difference between the
groups, assume the pill was the reason
If there is no significant difference between the
groups, assume the pill had no effect
Minimizing Bias
What if the control group and the treatment group
differ in some way other than the treatment?
–
Then we have confounding factors in the study, and the
results will be biased and/or invalid
Confounding factors can include…
–
–
–
–
–
–
Age
Race
Gender
Socio-economic status (SES)
Location
Political/religious orientation
Minimizing Bias
In the pill example, suppose all of the men were
placed into the treatment group and all of the women
were placed into the control group
–
What is wrong with this design?
The pill may interact differently with people
based on gender. For example, suppose
that high levels of testosterone diminish the
effect of the pill. Then the results would be
biased against the pill’s effectiveness.
Minimizing Bias
In order to minimize the risk of bias,
researchers assign people to each group
randomly from the pool of subjects
–
If enough people are in the pool, the groups will
look similar in terms of age, race, gender, or any
other demographic which could be a confounding
element
Observational Studies
How would you design a controlled
experiment to test if second-hand smoke
causes lung cancer?
Answer: You can’t, really. The time scale is too large and
it would be virtually impossible to ensure that your
control group never got second-hand smoke exposure.
Observational Studies
In many studies, designing a controlled experiment
is simply not possible
–
–
In such cases, researchers must settle for observational
studies
Observational studies let the subjects select which group to
join
Observational studies should be placed under great
scrutiny because the control and treatment groups
may be quite different in many ways
–
These differences may introduce confounding influences on
the study results
Observational Studies
A TV commercial observes that volunteers
who switched to having to Cheerios for
breakfast reduced their cholesterol level by
4%. Does this show that Cheerios cereal is
effective in reducing your cholesterol level?
Not necessarily. People who
volunteer for such a study may
already be interested in improving
their health and may be taking steps
such as exercising daily. This could
lower their cholesterol level.
Observational Studies
One form of observational study involves
using historical controls to compare a
modern treatment with
–
–
The difference in conditions between past and
present can itself be a confounding factor
For example, in testing a polio vaccine, one would
not want to compare infection rates in the 1950’s
with infection rates in the 1900’s because of the
large differences in standard of living and health
care quality in the two periods
Association, not Causation
An important limitation of all studies is that they can only show
the strength of association between two variables
–
–
Association may point to causation
For example, if exposure to a virus causes a disease, then people
who are exposed should be sicker than similar people who are not
exposed
However, a study alone cannot show causation
–
–
Human inference is needed to connect causal links
Generally, a casual link is accepted if
A scientific theory is derived which explains the association
Enough studies controlling all potential confounding factors are
performed, and all of them show the same association
Association, not Causation
Cervical cancer is more common among
women who have been exposed to the
herpes virus, according to many
observational studies. Is it fair to conclude
that the virus causes cervical cancer?
No. Both herpes and cervical cancer have been shown
to be sexually transmitted diseases. Women who have
many partners are at higher risk for both diseases than
women with few partners, so the number and type of
partners is a confounding influence which explains the
association.
Association, not Causation
Some studies find an association between liver
cancer and smoking. Does this mean smoking
causes liver cancer, or is there a confounding
influence at play?
People who smoke also tend to drink, so drinking is
associated with smoking. It has been shown
repeatedly that over consumption of alcohol can
cause liver cancer.
Comparing Rates
Sometimes the groups in comparison are of
different sizes
–
–
Rather than comparing the actual numbers, use
rates or proportions such as percents
Adjusting to rates allows for a more direct
comparison
Comparing Rates
In the U.S. in 1990, there were 2.1 million deaths
from all causes compared to 1.7 million in 1960
–
According to Census data, there were about 177 million
Americans in 1960 and about 250 million in 1990
True or false: The public’s health got worse over the
period 1960-1990.
False. Convert the death counts to percentage
rates. The death rate in 1960 was 0.96%, while
the death rate in 1990 was 0.84%. If anything,
public health improved.
Salk Vaccine Trial
In 1954, researchers tested the effectiveness of a
vaccine for polio
Approximately 400,000 children were selected
–
–
All children in the study had parental consent to be
vaccinated (Why?)
Children randomly divided into equal sized treatment and
control groups
Control group was given a placebo
Is this a controlled experiment or an observational
study?
Salk Vaccine Trial
Results
Study Group
Population
Polio Cases
Rate per
100,000
Vaccinated
200,745
57
28.4
Placebo
201,229
142
70.6
Was the vaccine effective?
Breast Cancer Screening
In 1963, an insurance company used 62,000
women (all insured by the plan) as subjects
–
–
–
Women ranged in age from 40 to 64
31,000 of the women were used as controls
The others were encouraged to undergo annual
screening
Roughly 20,200 women from the treatment group came
for annual screenings, the other 10,800 refused
After 5 years, the death rates were checked
Breast Cancer Screening
Is this a controlled experiment or an
observational study?
It depends on what questions you are trying to answer.
If you are only interested in the difference between the
treatment and control groups, then you are using the
results of a controlled experiment. However, when
you start drawing inferences between the screened
and refused groups, you are looking at an after-thefact observational study.
Breast Cancer Screening
Deaths (breast
cancer)
Deaths (all other
causes)
Number
Rate per
1,000
Number
Rate per
1,000
31,000
39
1.3
837
27
Examined
20,200
23
1.1
428
21
Refused
10,800
16
1.5
409
38
31,000
63
2
879
28
Treatment
Control
Researchers noted three things:
1. Screening had little impact on diseases other than breast cancer.
2. Poorer women were less likely to accept screening than richer ones.
3. Most diseases fall more heavily on the poor than on the rich.
Breast Cancer Screening
Deaths (breast
cancer)
Deaths (all other
causes)
Number
Rate per
1,000
Number
Rate per
1,000
31,000
39
1.3
837
27
Examined
20,200
23
1.1
428
21
Refused
10,800
16
1.5
409
38
31,000
63
2
879
28
Treatment
Control
Does screening save lives? Which numbers prove
your point?
Breast Cancer Screening
Deaths (breast
cancer)
Deaths (all other
causes)
Number
Rate per
1,000
Number
Rate per
1,000
31,000
39
1.3
837
27
Examined
20,200
23
1.1
428
21
Refused
10,800
16
1.5
409
38
31,000
63
2
879
28
Treatment
Control
Why is the death rate from all other causes in the whole
treatment group about the same as the rate for the control
group?
Breast Cancer Screening
Deaths (breast
cancer)
Deaths (all other
causes)
Number
Rate per
1,000
Number
Rate per
1,000
31,000
39
1.3
837
27
Examined
20,200
23
1.1
428
21
Refused
10,800
16
1.5
409
38
31,000
63
2
879
28
Treatment
Control
Why is the death rate from all other causes higher for the
“refused group” than the examined group?
Breast Cancer Screening
Deaths (breast
cancer)
Deaths (all other
causes)
Number
Rate per
1,000
Number
Rate per
1,000
31,000
39
1.3
837
27
Examined
20,200
23
1.1
428
21
Refused
10,800
16
1.5
409
38
31,000
63
2
879
28
Treatment
Control
Unlike most diseases, breast cancer afflicts the rich more
than the poor. Which numbers in the table confirm this
association?
Case Study: Child care and
behavior
Question: If parents choose to use child care,
are they more likely to see undesirable
behaviors in their children?
Question: How should one find out? By use
of controlled experiments, or by
observational study?
Case Study: Child care and
behavior
Answer: In 1991, a study commenced on 1364
infants (“subjects”) and followed them through their
sixth year in school. 12 years later, an article was
published. “the more time children spent in child care
from birth to age four-and-a-half, the more adults
tended to rate them, both at age four-and-a-half and
at kindergarten, as less likely to get along with
others, as more assertive, as disobedient, and as
aggressive.”
Case Study: Child care and
behavior
A summary of the study noted, “The study
authors noted that their study was not
designed to prove a cause and effect
relationship. …” (that more child care time
causes more aggressive behavior)
Possible confounding factors?
Perhaps… perhaps … perhaps…
Case Study: Child care and
behavior
Question: How about a controlled experiment
instead? (i.e. the researchers select children
to place in child care)
What are the advantages?
What problems might arise?
Summary: Basic Principles of
Statistical Experimental Design
Raise question to answer: Explanatory
variable(s) & Response variable(s).
Leading principle: Comparison.
Other principles: randomize, repeat (or
involve many subjects to reduce chance
variation)
Why:
Summary: Principles of Statistical
Experimental Design
Limitations: Statistical analysis of an experiment
cannot tell us how far the results will generalize to
other settings. (Experiment run in Massachusetts;
how about in the entire USA?)
However, the randomized comparative experiment,
because of its ability to give convincing evidence for
causation, is one of the most important ideas in
statistics.
More Elaborate Statistical Designs
Matched Pairs Design: Pair subjects with the
same age, sex, income, etc. and compare
their responses
Example: Compare two advertisements for
the same product
Common variation of MPD: impose both treatments
on the same subjects, so that each subject serves as
his or her own control. (See HW2, Q9)
More Elaborate Statistical Designs
Question: How does randomization work here?
Answer: Which one of a matched pair sees the first
ad is decided at random. Or, which treatment a
person gets first is decided at random if he or she
also serves as his or her own control.
Additional Example: Matched Pairs for the cell phone
experiment. (Details: trained in using simulator, order
in which a subject drives with and without the phone
be random, two drives be on separate days)
Summary
Statisticians use the method of comparison
–
To determine the effectiveness of a “treatment”, they usually
compare the responses of a treatment group with a control
group
If the control group is comparable to the treatment
group (apart from the treatment), then a significant
difference in the responses of the two groups is likely
to be due to the effect of the treatment
Summary
If the treatment group is different from the
control group with respect to other factors,
the effects of these other factors are likely to
be confounded with the effect of the
treatment
To make sure the treatment group is like the
control group, investigators put subjects into
either group at random
Summary
Whenever possible, the control group is given a
placebo
–
The response should be to the treatment itself rather than to
the idea of the treatment
In a double-blind experiment, the subjects do not
know whether they are in treatment or in control, and
neither do those who evaluate the responses
–
Guards against bias, both in the responses and in the
evaluations
Summary
In an observational study, the investigators
do not assign the subjects into treatment or
control groups
–
Subjects which have the condition are the
treatment group and the ones which do not are
the control
Observational studies can establish
association, which is not necessarily
causation
Summary
Observational studies are particularly susceptible to
confounding factors
–
This can make them very misleading about cause and effect
relationships
With observational studies, try to find out how the
subjects came to be in treatment or control
–
–
–
Are the groups comparable?
What factors are confounded with the treatment?
What adjustments (if any) did the investigators make to
control for these factors?
