EVALUATION OF RESEARCH STUDIES Part II: Observational Studies
Patricia Aikins Murphy, CNM,
In the first column of this series. we disxssed evaluating
key points to consider when experbnental
studies
11).
However, it is often not possible to conduct randomized the relationship
hi&
to a-
behueen exposures
and Leah L. Albers, CNM, DIPH
MS,
Observational
tidies
are ccnsid-
ences between users and “onusen
ered weaker than true experiments or
(commonly
randomized trials in their ability to de-
variables) must be considered.
tect causal relationships
When
the
There
are several iypes of obser-
exposure (also termed-the treatment,
vatin.
ciations between exposures and out-
is not a5-
comes
(2).
signed randomly,
weight association, for exampk.
can-
account for any ajsociation obsewd
prospective
b&wee”
groufx
de-
other factors may
studies
bii
by experimental
confounding
risk factor, or intavention)
and outcomes. The smokingiIow not be studied
termed
that exposure and the outwomen
who
in nature:
of exposed
persons
studies
and the”
They
assoare
identify
and unexposed follow
them over
sigr~ because one cannot randomly
come.
assign women to smoke during preg-
“choae”tobe
nancy. Similarly. the relationshi be. we” lack of prenatal care and ad-
exercise, select CNM care. or deliver in out-of-hospital birth sites) may be
p&a&x
verse pregnancy outcomes cannot be
dirrereni in impntant
way5 from those
vem1s
evaluated experimentally
who do not. These
differences
the tisk ratio (or relative risk:. Cohort
because it
For example,
that examine
Cohort
exposed (whosmoke,
can
is not ethical to deny prenatal care to some women. For reasons such as theso, most health studies are none% perimental or observational designs. where people are studied in their “at-
exert a : trong influence on the study
uml circumstances,
that may be involved
experimental sure.
and *here is no
assignment
of
expo-
resulL% Consider a study that finds a” asscciation beiween oral contmceptive usa and cervical ca”cer. Other
ictors
tima to axertai”
the frequency
of a
outcomein each g&x The ratio of the outcome in the exposed the unexposed group is called
studies offer the opportunfiy to measure exposures as they occur, before tbo paeeg; 2 lmle or kllowledge of an adverse outcome can affect OF’s memory of !xst exposures. Wile” it
in ibis assoc.
is not feasible to do im experiment,
ation are the fad that users of oral
a cohort study may be the next-best
co”tmceptives
choice, particularly
are sax*xal!y active {a
whefi
the
out-
known risk factor for cervical cancer)
come of interest is not rare and does
and
not
usually
do
not
u5e a barrier
method (a possible protective facior).
take
yean
birth weighf
contraceptives.
cxliage.
other
differ-
develop.
Many
pregnancy-related topics. such as low
To clarify the independent role of oral these
to
cesarean seciion,
and premahldiy,
mis-
are well
suited to cohort designs because the period of follow-up IL only about one year. Problems can arise when cohort studies of rare outcomes are undertaken becaw these studies require Ialp popu!abbnrfofkwed for lengthy periods. This can make a study costly and subject to bias if many &bjects drop out of the study over lime. A cohort study ofcancer riskcomparing oral contraceptive use15to users of other birth contiol methods, for example, w&d need to be very large fbecau~ cancer is rare in the age groups most likely to u5e oral contraceptivesl and also of !ong duration fbaause a short study period would miss cancers that take years to develop). Historical cohort studies ty to circumvent these problems by using exisiing records to determine past expmure sIatus.Subjectsare then traced to present locationsand evaluated for the outcome in question. This tvpe of study design has been used to ex-
amine asscctatlcmsbetween WEUpattonal or environmental expasurk-5 (suchas chemical agentsor radiation) and adverse reproductive events or cawer devekmment Such dud&, are very dependent on the quality and completenessof the dd records:Them must be enough informatlon available to classify people properly as either exposed or unexposed. Another critical problem is that of locating people years after the exposure. Inability to find a large number of subjectscan seriously undermine the results.Constder a study that Hnds no association between a &tain chemical agent and subsequent development of cancer. Suppae only half of those who had the expawre years ago were actually found and Interviewed. If the group that could not be located contained a large proportion c4 premature cancer deaths, a potentially impartant association would have been missed. Case-contml studies are retmspecwe. Cases (those u4th the outcome of interest) and contmls (those &aout the outcome) are identified and compared on the frequency of cer!ain exposures. The ratto of exposwe in cases to that of controls is the odds ratio. whichappmdmatesthe relative risk. Case-control studies are best suiied to stud& of ran: outcomessuch as maim birth defect% infant mortaland many cat&. Case-control studies will be more efficient and te;j costly than cohort studies in these situations. llw proper choiceof a controlgroup is the key to a valid case-controlstudy, Controls should represent the same population as the cases in terms of their likelihood of expowre. This is sometimesvery difficult to ensure, especially give” that these studies are often conducted with hospitalizedpatients. Consider a case-control study of the assodattcmbetween oral contraceptivew and breastcancerwhere casesand contmls are ldentifkd and interviewed about past exposures while in the hospiral. Maw wanen who are hmpitaU& for r&a other
ity,
412
Jatmmt of Nurse-Midwifery .
may have chronlcmedlcal pr;ble& that would reduce the likelihood of previousoral contraceptiw “se. If there were a large prop&ion of these women in the control gmup, this shtdy could find a higher rate of oral contraceptive use among women with breast cancer simply because the controls had an unusually low rate of use. not because of a real association. Another drawback of this study destgn Is that sub]eck already know whether or not they have Lhe outcomewhentheyareaskedaboutpast e~cosures. A classic examDIe of the re&“g problem (know”’ as real1 bias) is a studu of maior birth defeck. Mothers whc& newioms have congenital anomalies may be more likely than mothers of healthy newborns to search their memories for any possible exposure. Such differences in rememb&g the mimlte dekik of p%st exposures could produce erroneous or biased findings implicating a medication or job-related aposure in the etiology of birth defects. cross-sectional studies measure exposure and outcome at the same tie ina smpleofa population. These studies do not address whether exposure preceded outcome, because both are measured simultaneously. Thus cause and effect cannot be determined, although such studiesoften provide important clues for fuiher research. For example, the observation that breast cancer rates are hlgher In countties with high dietary fat consumption has led to a number of investigaticns of this association. Cross-sectional etudies are a cornman way of gathering information on the magnitude of acommunity health problem for planning health seni.ces They are also used to evaluate the effectiveness of health intewention prcgrams or educatiu” campzyns. These studieshave been uwd .o dotument use of prenatal care by highrisk populations and to study behavior change and cardiovascular disease prevalence. Crosssectional studies are relatively inexpensive to
Vol. 37, NO. 6. NovemberIDecembe#1992
perform, but they are subject to cer. tin problems. These include lack of precision in measuring exposure and outcome and lack of information on other factors (confounding variables) that could explain assaiations between exposure and outcome. Countries with high dietay fat cons”mption, for example, also tend to be westemtzed,ind”&t&ed nationswith htgherstandardsof living. Many other f&ore could be &ted to @her rates of breast cancer in these countries Cross-sectional studies are also prone to the problem of nonparticipation. In the case of mail or telephone surveys.people who respond, or participate. are often different from those who do not (they may be better educated or more Interested in the subject).Such differencescan bias the walk of the snrdy. When eval”atin9 the merits of an obsenmtionalshady,the reader should consider the foUowing points: 1. What is the basic study design? Were exposed and nonexposed 9ro”p followed foward in tkne before ascertaining outcome (cu hort)? Were and controls chosen and cornpavedon past exposwes (caseconiwl)? or were both exposure and outcome measured at one time on a sdmple of the population (cross-sectional)? Each type of design is prone to specific problems. 2. U it is a cohort design, how precisely was expos”re defined? For example, a note on a chart that a medication was prescribed does not mean it was taken. Were wtent&l changes in expos”re _&t”s considered? Subjects can change jobs, smoktn9 habits, or contraceptive methods during the study period. Such crossover between
Jwmal ofN”reeXidw&xy
l
exposedand unexposedstdhlscan 4. If the study is cross-sectional.was make it difficult to detect assoctthe sample randomly chosenfrom ationswith the outcome of intera w~ulafion? Was there enouqh est. What other factors com,,,on mformation to precisely da& to the exposed group could be resubjects into exposure and outlated to the outcome? A study of come groups?What confounding prenatal stressmust consider that variables may be present? Was women expatencin9 high stress nonparticipation or nonresponse may also have different levels of a problem? tobacco or alcohol “se. or differ5. For any design,sample &e should ent nutritional status, the be noted and the likelihwd of de““nswssed” conbol9ro”p. These tecting stdtitically significant refactor, could be related to any adsults addresazl. As we disclrssed “eise oi;tcane found and must be in the first column of this series(1 I. considered in the rosearch design some “ne9ative” shzdy concluand analysis. Was follow-up long sions are due to lack of statistical enough fur the outcome to depawrandare morepropedyconvelop? Did many subjects drop sidered “inconclusive.” out? How was o”tcome deter. In summary, observational shldy mined? line same cautions about des@nns are prone to a variety of inthe “Ccessityfor cledlfy definedand herent problems that can bias their uniformly applied criteria for dtresults. ‘The dfsc”sshm section of a agnoas that were dixussed in restudy rrport should address these letion to experimental studies (1) Problems, explain steps taken to minapply here as well. imize their effects,and discussthe pa3. For a case-control design. hue tentiel for bias In !he study results. were casesand controls identified The Meet research st”dy does not andchoxn?The mdyshoulddeexist: each published report repescdbothe reasonsfor includin9 and salts a” attenrpt t5 view an eweexcluding people as controls, and sue-ou1cMM r&onshtp fr‘nn a nau the reader should consider theso or different Fers!xcti”e. The critical facton caref”Uv.Did controlshave reader must.be &are of the limitathe same potential for expcsure as tions of observattonal research in orcases?A study of recent oral conder to draw appropriate conclusions tmceptive “se and risk of heari atfor clinical practice. tack, for example, should not include p&menopausal, sterile, or pregnant women in the control REFERENCES group bemuse none of these women are likely to “se oral con1. Murphy P, Alben L. Evaluatkx~of trace@ives (3). Are there differresearchadies. paa I: randanwedbtalr. ences other than exposure etahis J N”rw Midwifey 1992;37:287-90. behueen groups, such as so&2. Ma”mez J, Kramer S. Epidemioleconomic status or smoking habcgy an l”!md”ctOryten Pilweiphti its, that could expkin the findings? Saunders,1% I-. Did the authors account for these 3. S&,esselmanJ. caw co”W saddifferences?is recall bias a patenies:dej@. conduct.an&i%. New York oxford uniwsiiy Pie%, 15x3277. iial problem?
than
Vol. 37. No. 6. NovemberDecembez1992
413
Patricia Aikins Murphy, CNM,
In the first column of this series. we disxssed evaluating
key points to consider when experbnental
studies
11).
However, it is often not possible to conduct randomized the relationship
hi&
to a-
behueen exposures
and Leah L. Albers, CNM, DIPH
MS,
Observational
tidies
are ccnsid-
ences between users and “onusen
ered weaker than true experiments or
(commonly
randomized trials in their ability to de-
variables) must be considered.
tect causal relationships
When
the
There
are several iypes of obser-
exposure (also termed-the treatment,
vatin.
ciations between exposures and out-
is not a5-
comes
(2).
signed randomly,
weight association, for exampk.
can-
account for any ajsociation obsewd
prospective
b&wee”
groufx
de-
other factors may
studies
bii
by experimental
confounding
risk factor, or intavention)
and outcomes. The smokingiIow not be studied
termed
that exposure and the outwomen
who
in nature:
of exposed
persons
studies
and the”
They
assoare
identify
and unexposed follow
them over
sigr~ because one cannot randomly
come.
assign women to smoke during preg-
“choae”tobe
nancy. Similarly. the relationshi be. we” lack of prenatal care and ad-
exercise, select CNM care. or deliver in out-of-hospital birth sites) may be
p&a&x
verse pregnancy outcomes cannot be
dirrereni in impntant
way5 from those
vem1s
evaluated experimentally
who do not. These
differences
the tisk ratio (or relative risk:. Cohort
because it
For example,
that examine
Cohort
exposed (whosmoke,
can
is not ethical to deny prenatal care to some women. For reasons such as theso, most health studies are none% perimental or observational designs. where people are studied in their “at-
exert a : trong influence on the study
uml circumstances,
that may be involved
experimental sure.
and *here is no
assignment
of
expo-
resulL% Consider a study that finds a” asscciation beiween oral contmceptive usa and cervical ca”cer. Other
ictors
tima to axertai”
the frequency
of a
outcomein each g&x The ratio of the outcome in the exposed the unexposed group is called
studies offer the opportunfiy to measure exposures as they occur, before tbo paeeg; 2 lmle or kllowledge of an adverse outcome can affect OF’s memory of !xst exposures. Wile” it
in ibis assoc.
is not feasible to do im experiment,
ation are the fad that users of oral
a cohort study may be the next-best
co”tmceptives
choice, particularly
are sax*xal!y active {a
whefi
the
out-
known risk factor for cervical cancer)
come of interest is not rare and does
and
not
usually
do
not
u5e a barrier
method (a possible protective facior).
take
yean
birth weighf
contraceptives.
cxliage.
other
differ-
develop.
Many
pregnancy-related topics. such as low
To clarify the independent role of oral these
to
cesarean seciion,
and premahldiy,
mis-
are well
suited to cohort designs because the period of follow-up IL only about one year. Problems can arise when cohort studies of rare outcomes are undertaken becaw these studies require Ialp popu!abbnrfofkwed for lengthy periods. This can make a study costly and subject to bias if many &bjects drop out of the study over lime. A cohort study ofcancer riskcomparing oral contraceptive use15to users of other birth contiol methods, for example, w&d need to be very large fbecau~ cancer is rare in the age groups most likely to u5e oral contraceptivesl and also of !ong duration fbaause a short study period would miss cancers that take years to develop). Historical cohort studies ty to circumvent these problems by using exisiing records to determine past expmure sIatus.Subjectsare then traced to present locationsand evaluated for the outcome in question. This tvpe of study design has been used to ex-
amine asscctatlcmsbetween WEUpattonal or environmental expasurk-5 (suchas chemical agentsor radiation) and adverse reproductive events or cawer devekmment Such dud&, are very dependent on the quality and completenessof the dd records:Them must be enough informatlon available to classify people properly as either exposed or unexposed. Another critical problem is that of locating people years after the exposure. Inability to find a large number of subjectscan seriously undermine the results.Constder a study that Hnds no association between a &tain chemical agent and subsequent development of cancer. Suppae only half of those who had the expawre years ago were actually found and Interviewed. If the group that could not be located contained a large proportion c4 premature cancer deaths, a potentially impartant association would have been missed. Case-contml studies are retmspecwe. Cases (those u4th the outcome of interest) and contmls (those &aout the outcome) are identified and compared on the frequency of cer!ain exposures. The ratto of exposwe in cases to that of controls is the odds ratio. whichappmdmatesthe relative risk. Case-control studies are best suiied to stud& of ran: outcomessuch as maim birth defect% infant mortaland many cat&. Case-control studies will be more efficient and te;j costly than cohort studies in these situations. llw proper choiceof a controlgroup is the key to a valid case-controlstudy, Controls should represent the same population as the cases in terms of their likelihood of expowre. This is sometimesvery difficult to ensure, especially give” that these studies are often conducted with hospitalizedpatients. Consider a case-control study of the assodattcmbetween oral contraceptivew and breastcancerwhere casesand contmls are ldentifkd and interviewed about past exposures while in the hospiral. Maw wanen who are hmpitaU& for r&a other
ity,
412
Jatmmt of Nurse-Midwifery .
may have chronlcmedlcal pr;ble& that would reduce the likelihood of previousoral contraceptiw “se. If there were a large prop&ion of these women in the control gmup, this shtdy could find a higher rate of oral contraceptive use among women with breast cancer simply because the controls had an unusually low rate of use. not because of a real association. Another drawback of this study destgn Is that sub]eck already know whether or not they have Lhe outcomewhentheyareaskedaboutpast e~cosures. A classic examDIe of the re&“g problem (know”’ as real1 bias) is a studu of maior birth defeck. Mothers whc& newioms have congenital anomalies may be more likely than mothers of healthy newborns to search their memories for any possible exposure. Such differences in rememb&g the mimlte dekik of p%st exposures could produce erroneous or biased findings implicating a medication or job-related aposure in the etiology of birth defects. cross-sectional studies measure exposure and outcome at the same tie ina smpleofa population. These studies do not address whether exposure preceded outcome, because both are measured simultaneously. Thus cause and effect cannot be determined, although such studiesoften provide important clues for fuiher research. For example, the observation that breast cancer rates are hlgher In countties with high dietary fat consumption has led to a number of investigaticns of this association. Cross-sectional etudies are a cornman way of gathering information on the magnitude of acommunity health problem for planning health seni.ces They are also used to evaluate the effectiveness of health intewention prcgrams or educatiu” campzyns. These studieshave been uwd .o dotument use of prenatal care by highrisk populations and to study behavior change and cardiovascular disease prevalence. Crosssectional studies are relatively inexpensive to
Vol. 37, NO. 6. NovemberIDecembe#1992
perform, but they are subject to cer. tin problems. These include lack of precision in measuring exposure and outcome and lack of information on other factors (confounding variables) that could explain assaiations between exposure and outcome. Countries with high dietay fat cons”mption, for example, also tend to be westemtzed,ind”&t&ed nationswith htgherstandardsof living. Many other f&ore could be &ted to @her rates of breast cancer in these countries Cross-sectional studies are also prone to the problem of nonparticipation. In the case of mail or telephone surveys.people who respond, or participate. are often different from those who do not (they may be better educated or more Interested in the subject).Such differencescan bias the walk of the snrdy. When eval”atin9 the merits of an obsenmtionalshady,the reader should consider the foUowing points: 1. What is the basic study design? Were exposed and nonexposed 9ro”p followed foward in tkne before ascertaining outcome (cu hort)? Were and controls chosen and cornpavedon past exposwes (caseconiwl)? or were both exposure and outcome measured at one time on a sdmple of the population (cross-sectional)? Each type of design is prone to specific problems. 2. U it is a cohort design, how precisely was expos”re defined? For example, a note on a chart that a medication was prescribed does not mean it was taken. Were wtent&l changes in expos”re _&t”s considered? Subjects can change jobs, smoktn9 habits, or contraceptive methods during the study period. Such crossover between
Jwmal ofN”reeXidw&xy
l
exposedand unexposedstdhlscan 4. If the study is cross-sectional.was make it difficult to detect assoctthe sample randomly chosenfrom ationswith the outcome of intera w~ulafion? Was there enouqh est. What other factors com,,,on mformation to precisely da& to the exposed group could be resubjects into exposure and outlated to the outcome? A study of come groups?What confounding prenatal stressmust consider that variables may be present? Was women expatencin9 high stress nonparticipation or nonresponse may also have different levels of a problem? tobacco or alcohol “se. or differ5. For any design,sample &e should ent nutritional status, the be noted and the likelihwd of de““nswssed” conbol9ro”p. These tecting stdtitically significant refactor, could be related to any adsults addresazl. As we disclrssed “eise oi;tcane found and must be in the first column of this series(1 I. considered in the rosearch design some “ne9ative” shzdy concluand analysis. Was follow-up long sions are due to lack of statistical enough fur the outcome to depawrandare morepropedyconvelop? Did many subjects drop sidered “inconclusive.” out? How was o”tcome deter. In summary, observational shldy mined? line same cautions about des@nns are prone to a variety of inthe “Ccessityfor cledlfy definedand herent problems that can bias their uniformly applied criteria for dtresults. ‘The dfsc”sshm section of a agnoas that were dixussed in restudy rrport should address these letion to experimental studies (1) Problems, explain steps taken to minapply here as well. imize their effects,and discussthe pa3. For a case-control design. hue tentiel for bias In !he study results. were casesand controls identified The Meet research st”dy does not andchoxn?The mdyshoulddeexist: each published report repescdbothe reasonsfor includin9 and salts a” attenrpt t5 view an eweexcluding people as controls, and sue-ou1cMM r&onshtp fr‘nn a nau the reader should consider theso or different Fers!xcti”e. The critical facton caref”Uv.Did controlshave reader must.be &are of the limitathe same potential for expcsure as tions of observattonal research in orcases?A study of recent oral conder to draw appropriate conclusions tmceptive “se and risk of heari atfor clinical practice. tack, for example, should not include p&menopausal, sterile, or pregnant women in the control REFERENCES group bemuse none of these women are likely to “se oral con1. Murphy P, Alben L. Evaluatkx~of trace@ives (3). Are there differresearchadies. paa I: randanwedbtalr. ences other than exposure etahis J N”rw Midwifey 1992;37:287-90. behueen groups, such as so&2. Ma”mez J, Kramer S. Epidemioleconomic status or smoking habcgy an l”!md”ctOryten Pilweiphti its, that could expkin the findings? Saunders,1% I-. Did the authors account for these 3. S&,esselmanJ. caw co”W saddifferences?is recall bias a patenies:dej@. conduct.an&i%. New York oxford uniwsiiy Pie%, 15x3277. iial problem?
than
Vol. 37. No. 6. NovemberDecembez1992
413
