Downloaded from www.ajronline.org by 94.56.134.247 on 10/06/15 from IP address 94.56.134.247. Copyright ARRS. For personal use only; all rights reserved
185
High-Signal the Brain:
Foci on MR Images Observer
Variability
of
in Their
Quantification
Victor
F. Zernin Yetkin1 M. Haughton1
Mary
E. Fischer
R. Anne Papke1 David L. Daniels1 Leighton P. Mark1 Lloyd E. Hendnix1 Russell J. Asleson1’3 Johan Johansen1’4
Foci of high signal
in the cerebral
white
matter
are common
incidental
findings
on MR
images of the brain of control subjects or patients with a variety of diseases. Although the number of foci has been reported to correlate with age and several risk factors, the degree of observer variability in quantifying foci has not been reported. We used kappa statistics to determine radiologists’ agreement in counting high-signal-intensity foci on MR images obtained in healthy volunteers and in patients with hypertension. Before interpreting the images, one pair of radiologists studied 30 routine MR images and reached
consensus
on differentiating
high-signal
foci
from
other
foci
of high
intensity
caused by normal structures (e.g., deep gyn or Virchow-Robin spaces). These two observers then independently determined the number of foci in the study group. Using their own criteria, other radiologists independently counted the foci. Agreement between
observers
was determined
with the kappa statistic.
The results showed
fair
agreement
between the radiologists who first reached a consensus in counting foci of hyperintensity and poor agreement between the other observers. We conclude that in order to compare the frequency of foci of hypenntensity in different groups of patients, observer variability must be controlled. Studies without proper control subjects may head to incorrect conclusions regarding the correlation of focal hyperintensities and various risk factors. 159:185-188,
AJR
Foci
July i992
of abnormally
incidentally
high
signal
intensity
in cerebral
on MR images in up to 20% of patients
of hypenintensity ties, white matter
are also referred hypenintensities,
white
or control
matter
subjects.
are
found
These foci
to as leukoanaiosis, periventriculan hypenintensiand “unidentified bright objects.” The prevalence
of foci of hyperintensity
in patients with hypertension, migraine, and dementia and subjects has been studied [i -7]. Foci of hyperintensity have been reported to correlate with some demographic or medical factors. To our knowledge, the effect of observer variability on the number of foci of hypenintensity detected on MR images has not been studied. Accordingly, we measured the degnee of observer variability in determining the number of high-signal foci found in asymptomatic
Received July 22, 1991 ; accepted after February 4, 1992. 1 Department of Radiology, The Medical
revision
3
gists,
of Wisconsin,
Present Colorado
address: Springs,
Milwaukee,
Colorado
WI 53226.
Springs
Aadiolo-
CO 80919.
4 Present address: Department of Radiology, Akershus Central Hospital, Nordbyhagen, Norway.
0361-803X/92/1591-0185 © American Roentgen
on MR images
in healthy
Materials
and Methods
Radiologists
independently
interpreted
MR
images
obtained
in
control
subjects
(313
studies), patients with multiple sclerosis (43 cases), patients with dementia of the Alzheimer type (32 cases), and patients with hypertension (63 cases). The patients with multiple sclerosis fulfilled the criteria of Schumacher et al. [8] for establishing the diagnosis. The patients with hypertension were selected from a group undergoing treatment for essential hypertension; none
of them
had neurologic
signs
or symptoms.
The diagnosis
was documented by standard neurohogic and neuropsychological Each MR study
Ray Society
volunteers.
College
of Wisconsin, Froedtert Memorial Lutheran Hospital, 9200 W. Wisconsin Ave., Milwaukee, WI 53226. Address reprint requests to v. M. Haughton. 2 Biostatistics & Clinical Epidemiology, Medical College
control
included
a sagittal
by a series of 40 axial images
localizer
(2000/20,70)
image
obtained
in patients
with
dementia
(TR/TE)
followed
testing.
with 500/20
with a 5-mm slice thickness
and 1-mm gap,
i 86
cardiac gating, 256 x 256 matrix, and
two
excitations
YETKIN
ET AL.
with
TABLE
obtained
a 1 .5-T scanner(Signa, General Electric Medical Systems, Milwaukee, and assigned a number. All alphanumeric information on the images was obscured. Neuroradiohogists with at least 2 years experience in MR imaging were recruited to interpret the images. One radiologist, designated as observer 1 , interpreted 41 2 studies (mixture of patients and control subjects). Three radiologists reviewed portions of the entire collection
Downloaded from www.ajronline.org by 94.56.134.247 on 10/06/15 from IP address 94.56.134.247. Copyright ARRS. For personal use only; all rights reserved
WI). The studies
(patients
were
and control
in random
placed
subjects
mixed)
observer
1 (designated
without
first conferring
with one
observer
4 for this session)
and
another observer (designated observer 3) reviewed together 30 of the MR studies that were not included in the study sample to reach a consensus on distinguishing foci of hyperintensity from sulci or other
normal
structures.
Then
they
reviewed
independently
number
of small (less than 5 mm) and large (5 mm or larger)
of hyperintensity
per
foci
case, provided a radiologic interpretation for the
study (normal, abnormal, specific images or findings if they wished.
disease), Observer
and commented on the 1 interpreted the studies
2 were compared
to determine
agreement
between
radiologists
who
Observers I , 2, and 3 were also compared to determine agreement in a larger group of radiologists who had not reached consensus. had not reached consensus. 1 (labeled determine
sensus. compared
as observer agreement
The second and third interpretations of observer 1 were to determine intraobserver error. The first and second
Agreement which
I were compared made
observers
with
to determine and
without
is a measure
adjusted
due to chance servers, kappa equals 1 For evaluation purposes, a kappa greater .
than .75 indicates excellent agreement, indicates
agreement
fair
agreement,
and
a kappa
a kappa below
between
.40 and .75
.40 indicates
poor
[9, 101.
Results
Of the 45i
MR studies
enrolled
for this study,
356 were
by one radiologist to determine intraob293 were reviewed by both observers 3
and 4 (who reached a consensus); and i 91 were reviewed by observers i , 2, and 3 (with no consensus). Control subjects and patients were mixed randomly in each group of studies. Agreement between observers i and 2 was poor or fair (Table 1). Agreement was fain for seven on more small foci of hyperintensity, for no large foci of hypenintensity, and for more than two large foci of hyperintensity; it was poor for all other
comparisons. The kappa statistics .531 for small foci of hypenintensity .657 for large foci of hyperintensity.
were between and between
2
41
Poor
.270(141,
.399)
27
15
Poor
.106(000,
.250)
3-6 7+ Mean
30 21
19 25
Poor Fair Poor
.147 (.002, .292) .531 (.389, .673) .263 (.1 74, .352)
72 12 16
73 15 12
Fair Poor Fair
.657 (.536, .778) .300 (.1 1 4, .486) .542 (.370, .714)
Fair
.530 (.41 5, .645)
Large (>5 mm) 0 1 -2 3+
Mean Note--Cl
=
confidence
interval.
TABLE 2: Comparison of Observers 1, 2, and 3 for the Detection of High-Signal Foci in the Cerebral White Matter on MR Images of Total
Size of Foci/No.
Observer
Detected
Observer
Observer
2
3
41 15 19 25
63 22 11 5
Agreement
Kappa
Poor Poor Poor Poor
.227 .005 .063 .348
Poor
.162
Small (5 mm) 0
72
73
85
Fair
.530
1-2
12
15
9
Poor
.266
3+
16
12
6
Fair Fair
.461 .440
Mean
Agreement among observers 1 , 2, and 3 was poor or fair (Table 2). No small foci of hyperintensity were found in 22% of the studies by observer 1 , in 41 % by observer 2, and in
63% by observer 3. Detection of foci 5 mm or larger was less variable (Table i). Observer 1 reported that 72% of studies showed no large foci of hypenintensity, observer 2 reported 73%, and observer 3 reported 85%. Agreement among the observers
interpreted twice server agreement;
1 Observer
a preliminary
was measured
of agreement
on MR Images
22
the agree-
by using the kappa for the agreement alone. If there is complete agreement between ob-
between
Matter
1-2
of observer
4) were compared with those of observer 3 to between radiologists who had reached a con-
interpretations of observer ment between interpretations consensus session. statistic,
The second interpretations
1 and 2 for the Detection White
Agreemen t Kappa (95% CI) Observer
Small (
185
High-Signal the Brain:
Foci on MR Images Observer
Variability
of
in Their
Quantification
Victor
F. Zernin Yetkin1 M. Haughton1
Mary
E. Fischer
R. Anne Papke1 David L. Daniels1 Leighton P. Mark1 Lloyd E. Hendnix1 Russell J. Asleson1’3 Johan Johansen1’4
Foci of high signal
in the cerebral
white
matter
are common
incidental
findings
on MR
images of the brain of control subjects or patients with a variety of diseases. Although the number of foci has been reported to correlate with age and several risk factors, the degree of observer variability in quantifying foci has not been reported. We used kappa statistics to determine radiologists’ agreement in counting high-signal-intensity foci on MR images obtained in healthy volunteers and in patients with hypertension. Before interpreting the images, one pair of radiologists studied 30 routine MR images and reached
consensus
on differentiating
high-signal
foci
from
other
foci
of high
intensity
caused by normal structures (e.g., deep gyn or Virchow-Robin spaces). These two observers then independently determined the number of foci in the study group. Using their own criteria, other radiologists independently counted the foci. Agreement between
observers
was determined
with the kappa statistic.
The results showed
fair
agreement
between the radiologists who first reached a consensus in counting foci of hyperintensity and poor agreement between the other observers. We conclude that in order to compare the frequency of foci of hypenntensity in different groups of patients, observer variability must be controlled. Studies without proper control subjects may head to incorrect conclusions regarding the correlation of focal hyperintensities and various risk factors. 159:185-188,
AJR
Foci
July i992
of abnormally
incidentally
high
signal
intensity
in cerebral
on MR images in up to 20% of patients
of hypenintensity ties, white matter
are also referred hypenintensities,
white
or control
matter
subjects.
are
found
These foci
to as leukoanaiosis, periventriculan hypenintensiand “unidentified bright objects.” The prevalence
of foci of hyperintensity
in patients with hypertension, migraine, and dementia and subjects has been studied [i -7]. Foci of hyperintensity have been reported to correlate with some demographic or medical factors. To our knowledge, the effect of observer variability on the number of foci of hypenintensity detected on MR images has not been studied. Accordingly, we measured the degnee of observer variability in determining the number of high-signal foci found in asymptomatic
Received July 22, 1991 ; accepted after February 4, 1992. 1 Department of Radiology, The Medical
revision
3
gists,
of Wisconsin,
Present Colorado
address: Springs,
Milwaukee,
Colorado
WI 53226.
Springs
Aadiolo-
CO 80919.
4 Present address: Department of Radiology, Akershus Central Hospital, Nordbyhagen, Norway.
0361-803X/92/1591-0185 © American Roentgen
on MR images
in healthy
Materials
and Methods
Radiologists
independently
interpreted
MR
images
obtained
in
control
subjects
(313
studies), patients with multiple sclerosis (43 cases), patients with dementia of the Alzheimer type (32 cases), and patients with hypertension (63 cases). The patients with multiple sclerosis fulfilled the criteria of Schumacher et al. [8] for establishing the diagnosis. The patients with hypertension were selected from a group undergoing treatment for essential hypertension; none
of them
had neurologic
signs
or symptoms.
The diagnosis
was documented by standard neurohogic and neuropsychological Each MR study
Ray Society
volunteers.
College
of Wisconsin, Froedtert Memorial Lutheran Hospital, 9200 W. Wisconsin Ave., Milwaukee, WI 53226. Address reprint requests to v. M. Haughton. 2 Biostatistics & Clinical Epidemiology, Medical College
control
included
a sagittal
by a series of 40 axial images
localizer
(2000/20,70)
image
obtained
in patients
with
dementia
(TR/TE)
followed
testing.
with 500/20
with a 5-mm slice thickness
and 1-mm gap,
i 86
cardiac gating, 256 x 256 matrix, and
two
excitations
YETKIN
ET AL.
with
TABLE
obtained
a 1 .5-T scanner(Signa, General Electric Medical Systems, Milwaukee, and assigned a number. All alphanumeric information on the images was obscured. Neuroradiohogists with at least 2 years experience in MR imaging were recruited to interpret the images. One radiologist, designated as observer 1 , interpreted 41 2 studies (mixture of patients and control subjects). Three radiologists reviewed portions of the entire collection
Downloaded from www.ajronline.org by 94.56.134.247 on 10/06/15 from IP address 94.56.134.247. Copyright ARRS. For personal use only; all rights reserved
WI). The studies
(patients
were
and control
in random
placed
subjects
mixed)
observer
1 (designated
without
first conferring
with one
observer
4 for this session)
and
another observer (designated observer 3) reviewed together 30 of the MR studies that were not included in the study sample to reach a consensus on distinguishing foci of hyperintensity from sulci or other
normal
structures.
Then
they
reviewed
independently
number
of small (less than 5 mm) and large (5 mm or larger)
of hyperintensity
per
foci
case, provided a radiologic interpretation for the
study (normal, abnormal, specific images or findings if they wished.
disease), Observer
and commented on the 1 interpreted the studies
2 were compared
to determine
agreement
between
radiologists
who
Observers I , 2, and 3 were also compared to determine agreement in a larger group of radiologists who had not reached consensus. had not reached consensus. 1 (labeled determine
sensus. compared
as observer agreement
The second and third interpretations of observer 1 were to determine intraobserver error. The first and second
Agreement which
I were compared made
observers
with
to determine and
without
is a measure
adjusted
due to chance servers, kappa equals 1 For evaluation purposes, a kappa greater .
than .75 indicates excellent agreement, indicates
agreement
fair
agreement,
and
a kappa
a kappa below
between
.40 and .75
.40 indicates
poor
[9, 101.
Results
Of the 45i
MR studies
enrolled
for this study,
356 were
by one radiologist to determine intraob293 were reviewed by both observers 3
and 4 (who reached a consensus); and i 91 were reviewed by observers i , 2, and 3 (with no consensus). Control subjects and patients were mixed randomly in each group of studies. Agreement between observers i and 2 was poor or fair (Table 1). Agreement was fain for seven on more small foci of hyperintensity, for no large foci of hypenintensity, and for more than two large foci of hyperintensity; it was poor for all other
comparisons. The kappa statistics .531 for small foci of hypenintensity .657 for large foci of hyperintensity.
were between and between
2
41
Poor
.270(141,
.399)
27
15
Poor
.106(000,
.250)
3-6 7+ Mean
30 21
19 25
Poor Fair Poor
.147 (.002, .292) .531 (.389, .673) .263 (.1 74, .352)
72 12 16
73 15 12
Fair Poor Fair
.657 (.536, .778) .300 (.1 1 4, .486) .542 (.370, .714)
Fair
.530 (.41 5, .645)
Large (>5 mm) 0 1 -2 3+
Mean Note--Cl
=
confidence
interval.
TABLE 2: Comparison of Observers 1, 2, and 3 for the Detection of High-Signal Foci in the Cerebral White Matter on MR Images of Total
Size of Foci/No.
Observer
Detected
Observer
Observer
2
3
41 15 19 25
63 22 11 5
Agreement
Kappa
Poor Poor Poor Poor
.227 .005 .063 .348
Poor
.162
Small (5 mm) 0
72
73
85
Fair
.530
1-2
12
15
9
Poor
.266
3+
16
12
6
Fair Fair
.461 .440
Mean
Agreement among observers 1 , 2, and 3 was poor or fair (Table 2). No small foci of hyperintensity were found in 22% of the studies by observer 1 , in 41 % by observer 2, and in
63% by observer 3. Detection of foci 5 mm or larger was less variable (Table i). Observer 1 reported that 72% of studies showed no large foci of hypenintensity, observer 2 reported 73%, and observer 3 reported 85%. Agreement among the observers
interpreted twice server agreement;
1 Observer
a preliminary
was measured
of agreement
on MR Images
22
the agree-
by using the kappa for the agreement alone. If there is complete agreement between ob-
between
Matter
1-2
of observer
4) were compared with those of observer 3 to between radiologists who had reached a con-
interpretations of observer ment between interpretations consensus session. statistic,
The second interpretations
1 and 2 for the Detection White
Agreemen t Kappa (95% CI) Observer
Small (
