Validity of Hematologic Data in Veterans Administration Hospital Laboratories A Veterans Administration
Cooperative Study
LEONARD S. KAPLOW, M.D., MURIEL K. SCHAUBLE, M.D., AN,D JACK M. BECKTEL, M.S.
DESPITE their common linkage, Veterans Administration Hospital Laboratory Services differ in workloads and in calibers and sizes of technical and professional staffs. Some have active residency training programs or other educational functions or both, whereas others Received October 17, 1977; received revised manuscript and accepted for publication January 30, 1978. Supported by the V. A. Cooperative Studies Program. V. A. Central Office, Washington, D. C. Address reprint requests to Dr. Kaplow: Chief, Laboratory Service, Veterans Administration Hospital, West Haven, Connecticut 06516.
Laboratory Service, V. A. Hospital, and Departments of Pathology and Laboratory Medicine, Yale University School of Medicine, New Haven, Connecticut; Clinical Pathology, V. A. Hospital and Department of Pathology, Albany Medical College, Albany, New York; Cooperative Studies Program Coordinating Center, V. A. Hospital, Hines, Illinois
Executive Committee: Schauble. Muriel K.. M.D.. Chairman. Chief, Clinical Pathology, VAH, Albany, New York: Bearman. Jacob E., Ph.D., Professor, Department of Biometry. University of Minnesota, Minneapolis, Minnesota; Becktel, Jack M.. M.S., Statistician. Cooperative Studies Program Coordinating Center, VAH. Hines, Illinois: Gullick. Herbert D.. M.D.. Chief. Surgical Service, VAH, Albany, New York: Henry. John B., M.D., Director of Clinical Pathology, Upstate Medical Center. SUNY. Syracuse, New York: James, Kenneth, Ph.D., Chief. Cooperative Studies Program Coordinating Center, VAH, Hines, Illinois: Kaplow, Leonard S., M.D.*: Steele, Bernard W., M.D.Central Office Coordinators: Hagans. James, M.D., Ph.D.. Chief. Cooperative Studies Program. VACO, Washington, D. C ; Mendel. Julius, Ph.D., Asst. Director. Pathology Service. VACO, Washington. D. C. Participants: Beals. Theodore F.. M.D.. Asst. Chief. Laboratory Service, VAH, Ann Arbor, Michigan; Franco-Jove, Donna, M.D.. Clinical Pathologist, VAH. Atlanta. Georgia; Gill, Edward, Supervisory Chemist, VAH, Buffalo, New York; Hargreaves, Thomas, M.D.. Consultant Chemist. VAH. Palo Alto, California: Horvath. Beni, M.D., Clinical Pathologist, VAH. Boston. Mass.: Jarvis, Bruce W., M.D., Chief, Laboratory Service, VAH, New Orleans, La.; Jimenez, Fidelio, M.D., Chief. Laboratory Service, VAH. Brooklyn, New York; Kaplow, Leonard S.. M.D.,* Chief, Laboratory Service, VAH, West Haven, Conn.; Kassouny. Dicran, M.D., Clinical Pathologist, VAH, New York, New York; Krisko, Istvan, M.D., Staff Pathologist. Laboratory Service. VAH, Houston, Texas; Lingeman, Ralph, M.D.. Clinical Pathologist, VAH, Washington, D. C ; Maldonato, Herbert. M.D.. Clinical Pathologist, VAH, San Juan, Puerto Rico; Mir, Juan A.. M.D., Chief, Laboratory Service, VAH, Chicago, Illinois; Price. Preston, M.D., Chief, Laboratory Service, Oteen. North Carolina; Robinson, C. A., Jr., M.D.. Laboratory Service, VAH, Birmingham, Alabama: Rubnitz, Myron, M.D., Chief, Laboratory Service, VAH, Hines, Illinois; Schatzki, P. F., Ph.D., M.D., Staff Pathologist. VAH. Richmond, Va.; Skerrett. P. V., M.D., Chief, Laboratory Service, VAH, Philadelphia, Pa.; Steele, Bernard W.. M.D.,* Chief, Clinical Chemistry, VAH, Minneapolis, Minn.; White. Harold J., M.D., Chief, Laboratory Service, VAH, Little Rock, Ark. * Listed both in Executive Committee and as Participants.
291
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 6, 2016
Kaplow, Leonard S., Schauble, Muriel K., and Becktel, Jack M.: Validity of hematologic data in Veterans Administration Hospital laboratories. A Veterans Administration cooperative study. Am J Clin Pathol 71:291-300, 1979. Twenty laboratories in Veterans Administration Hospitals, similar in sophistication, were assessed for accuracy and precision in performing hemoglobin (Hb), hematocrit (Hct), erythrocyte count, mean corpuscular volume (MCV), and leukocyte count. Nineteen of the laboratories used the Coulter S counter. Ten laboratories were classified as "superior" by past performance in College of American Pathologists (CAP) surveys. The remainder were selected from other V. A. hospitals. All laboratories analyzed two "masked" and two "open" samples from two pools (normal and abnormal) on the same day, once weekly, for three consecutive weeks (12 samples/pool/lab). Results from 19 laboratories were evaluated for within-day, day-to-day, and interlaboratory variability, effects of masking, and previous performance in CAP surveys. Overall study means were in agreement with target values. An unexpected finding was that withinday variability was greater than day-to-day variability, which was greater than interlaboratory variability (variance components). Precisions for Hb, Hct, erythrocyte count, and MCV were greater than that for leukocyte count. Interlaboratory Standard deviations were less than or close to '' medical needs.'' The data indicated that for three laboratories, "open" samples resulted in a significant bias. It is concluded that V. A. hospital laboratories using similar instrumentation perform at the level of the "state of the art" and could serve as a common resource for cooperative studies requiring Coulter S data. (Key words: Laboratory performance; Laboratory surveys; Clinical hematology laboratory evaluation; Laboratory ranking; Laboratory variation.)
292
KAPLOW, SCHAUBLE, AND BECKTEL
Table I. Average Hospital Profile* Number of beds
700
Square feet of laboratory space
12,000
Workload (weighted tests)* Total Hematology
7.3 million 1.2 million
Productivity (average weighted)* Units/tech-clerical yr (VA recommended standard
172,000 94,000*)
Staffing Pathologists (total) Clinical Pathologists Scientists Supervisors Technologists Technicians Clerks
5.2 1.7 2.5 6.0 20.0 21.0 3.5
Total:
59.9 90% 80%
Research Programs
58%
* At the time of the study.
do not. V. A. Hospitals are particularly suited for cooperative studies, and an efficient centrally directed mechanism for funding and monitoring such programs has been operative for many years. Such studies have made important contributions to medical knowledge. Cooperative research studies may involve a few or numerous participants widely dispersed across the country. They are rarely devoted solely to laboratory aspects of disease, but often utilize data submitted by laboratories from the participating hospitals. Routine hematologic data such as those derived from the Coulter S countert are probably more commonly used for this purpose than any other laboratory procedure. The validity of pooling data from such diverse facilities for statistical use in cooperative research projects is uncertain. The present study was undertaken to determine the precision and accuracy of V. A. Hospital laboratories in performing basic hematologic procedures in an effort to assess their potential reliability as participants in cooperative research activities. It was also intended to evaluate the "state of the art" in such laboratories and to compare their performances with previous results in CAP proficiency surveys and with those of other hematology laboratories for within-day, day-today, and short-term (three weeks) interlaboratory variability. In addition, studies were undertaken to determine whether masking had any influence on the results reported by the various participating laboratories. A t Coulter Electronics, Inc., Hialeah, Fla.
companion study evaluating clinical chemistry laboratory performance was also undertaken.9 Materials and Methods Twenty V. A. Hospital laboratories of diverse geographic locations but similar in sophistication participated in the study. They represent a cross-section of 60 V. A. Hospitals active in cooperative research studies. Ten laboratories had previously been classified as superior, based on past performances in national proficiency surveys. The remaining ten were randomly selected from the other laboratories in the group of 60. A profile of the average participating laboratory is given in Table 1. Commercial^ pools of normal and abnormal stabilized blood cell suspensions were shipped to the Albany V. A. Hospital, which served as the coordinating laboratory, where preliminary tests were carried out to determine the effects of storage on the pooled cell suspensions. Analyses were performed daily at Albany during the three-week study period and for 22 additional consecutive days. However, these data were not included in the study. The outdating period indicated by the supplier expired one day after the last day of the actual study. Materials were monitored and shipped to participants from Albany. Replicate aliquots of the two pools were prepared at Albany, and four such aliquots of each pool were shipped to participating laboratories on Monday for three consecutive weeks. The cell suspensions were stored at 5 ± 3 C and warmed to room temperature prior to analysis. All samples were analyzed on the same day by all participating laboratories during the week of mailing. Ail but one laboratory used the Coulter S counter for analysis. This laboratory (Laboratory 18) used a Hemalog§ instrument. The values reported by this laboratory are indicated in graphic form in Figure 1. However, their data were not used in the statistical analysis. Two samples of each pool were labeled "study specimen" but were otherwise treated as routine samples and represented unmasked specimens. The other two samples of each pool were given fictitious patients' names and were introduced into the normal workload without further identification. These were considered masked samples. Thus, the study accumulated 2,400 bits of information (120 per laboratory), of which 2,280 were used for statistical analysis. All laboratories used Coulter 4C to standardize their instruments. Results were mailed each week to Albany, where they were collated and transmitted to the Hines, Illinois, V. A. $ Dade Corporation, Miami, Fla. 33152. § Technicon Corp., Tarrytown, N. Y.
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 6, 2016
Training Programs (% of hospitals) Clinical Pathology Residency Medical Technology
A.J.C.P. • March 1979
Vol. 71 . No. 3
Individual laboratories were judged for their performances with respect to accuracy and precision on the basis of their summed squared standard deviation index (SDI)2. These values were used rather than SDI values since they are more likely to distinguish truly superior from inferior laboratory performance. This occurs because SDI are approximately standard normal variables; i.e., they may be considered to have (approximately) a gaussian distribution with mean zero and unit standard deviation. In contrast, sums of squares of independent standard normal variables have chi-squared distributions with as many degrees of freedom as the number of variables (i.e., number of SDI/ laboratory; in our case, 10). Tables of chi-squared distributions are readily available and may indicate whether a sum is sufficiently extreme to be of concern. On the other hand, decisions as to whether extreme values are random or significantly deviant are more difficult when one uses SDI values. Laboratories were also ranked for accuracy which was appraised by a scoring technic whereby each lab-
oratory was given a rating of 4 for all values falling within ±1 SD of the "true" value and 2 for each value falling between ± 1 and ±2 SD. No credit was given for values more than ±2 SD from the "true" value. The sum of the ratings for all 120 determinations of each laboratory was considered its overall score and was used to evaluate laboratories for accuracy performance. Finally, accuracy was additionally evaluated by counting the number of individual determinations whose values were more than ±2 SD from the "true" mean and considered unacceptable. By relating the number of such unacceptable results to the total number (120) of determinations, we were able to obtain a percentage index representing a degree of unacceptable performance. These were converted to percentages of acceptable performance by subtracting from 100. Results Preliminary tests at Albany demonstrated only minimal changes in sample values due to storage for three weeks beyond the outdating period (Table 2). Figure 1 indicates the total data derived from the study. The overall study mean ("truth") for 19 laboratories for both normal and abnormal pools for all five parameters are represented by the centrally located horizontal lines calculated from data that excluded outliers. The area outlined by the other two horizontal lines represents 2 SD around this mean. The basic data for each hospital laboratory are located within the vertical rectangles, and the means are indicated by the short horizontal lines. The three daily means derived from these data are indicated by the three solid circles. The data reported by Laboratory 18 are included and represented by the dashed rectangle. However, their values were not used in arriving at the "truth" or in any other statistical analysis. Extreme values reported by individual laboratories are indicated outside the rectangles. They were not excluded in assessing the performance of each laboratory. Thus, these figures are a pictorial representation of within-day, average day-today, and overall study precision and accuracy. The interlaboratory means, standard deviations, and coefficients of variation (CV) are given in Table 3 and are compared with reference values and medical-needs requirements. The overall means, SD, and CV for all five constituents for masked samples were compared with similar data for open samples (Table 4). Based on F tests, statistically significant consistent differences were not observed. However, careful inspection of basic data from individual laboratories indicated a consistent smaller variance for three laboratories for all open
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 6, 2016
Cooperative Study Program Support Center for statistical analysis. Means, standard deviations (SD) and coefficients of variation (CV) were computed for the entire study as well as for within-day, day-to-day, and interlaboratory variability. Values obtained were compared with (1) stated values supplied by Dade, (2) values determined by Albany during the preliminary study, (3) medical-needs variability as defined by Barnett1 for hemoglobin (Hb), and in addition, for all parameters using the concept devised by Cotlove.3 The latter assumes that the SD of the analytic error should be less than or the same as half the standard deviation of the biologic variation. When this is true, the analytic-error component contribution to the total variation is 12% or less. The theoretical Cotlove medicalneeds values were derived from standard deviations obtained from another investigation,8 in which biologic variation for a group of 381 healthy male subjects, aged 20 to 29 years, was determined by participants in this study. Using all data derived from the Coulter S counters, the intralaboratory means and SD for the five normal and five abnormal parameters were determined first. Outliers were categorized as those values lying more than 2 SD from the mean. These observations were excluded and means and SD were then recalculated. These values were considered to be closest to the "truth." However, all data, including outliers, were used to determine the means and SD for individual laboratories for comparison with the "truth" to assess accuracy. In addition, the relative contribution of within-day, day-to-day, and interlaboratory variance to total analytic error was estimated as described for the clinical chemistry study.9
I
293
V.A. HEMATOLOGY LABORATORY PERFORMANCE
I«J6
2
I
3
1
4
1
5
4 43
7
1
8
8
1
y.
• 1*1
9
9
1
•
4 33
1
1
»
•
LL
. 2
» J II1
*
. « m 6 • _ *. • 2
4 1
r
363
i i
m
.
•
19 20
_
L
m... 4*1*" Tali
12 13 14 15 16 17 18
U-»I3 2
10 II
1
1
•
• T ••
'
I7
30.1
1
4 1.7
425
44.1
43.3
«_ 46.3
E 47'7
48.9
3.03
-
4 89
4.37
4.63
"• 4.73
I
J I
*
2
3
L 2 3
4
M
5
-f
8
9
10 II
12 13 14 IS 16 17 18 19 20
_| _J _l 10 II 12 13 14 15 16 17 18 19 20
LABORATORY I.D. NUMBER
7
_J
4M
! !• Ill
:•!
flfHSK
i i 121 i i
—
94
86
88
90 -
92
94
96
96
6
6
1
[_l_
{T * >
1
1
•
'
RED BLOOD COUNT
1
i
jj
2. y 1 •
#
m.
.....
'
31.3
S.I 3
U 4.SI IS
2
3 |
1 1
•
[ [•
•
m»
L 17
! •
Ld 1
Lu
,
1 •
L_ J 15 16
23 3
J.
; i < 1
#
• 4
. 2 .
•
_l 14
*
13
213
«4
m
•
•..
L_ _L II 12
iii:
A
LABORATORY ID. NUMBER
7
1
i.
J.
• 2
7
J 10
227
•ffil
*- v % z. • U 3• 4
?
3 1
VOLUME
7
H
FIG. I. Study means (N = 228) and two standard deviations for normal (above) and abnormal (below) pools for all five modalities tested (hemoglobin, hematocrit, erythrocyte count, mean corpuscular volume, and leukocyte count). Within-day values are enclosed in rectangles with digits representing the number of duplicate or replicate values (N = 12/day). The solid circles represent three daily means for each laboratory (N = 3). Note: The results for Laboratory 18 are depicted but were not used in deriving the study mean. The values for deviant results are indicated above or below the broken end of the appropriate rectangle.
87
89
91
93
93
97
101
103
2 3.6
24.5
2 5.4
26.3
— 28.1 S! _ 27.2 u
29.0
29.9
m m
9A
9.2
30.8
9.6
HEMOGLOBIN
ABNORMAL
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 6, 2016
IP/
A.J.C.P. • March 1979
KAPLOW SCHAUBLE, AND BECKTEL
296
Table 2. Results of Prel iminary Albany Study*
Hemoglobin
22-day Abnormal pool 3-week 22-day
X
SD
X
SD
X
SD
X
SD
14.5 (2.25) 14.4 (2.23)
0.12 (.019) 0.14 (.023)
45.7
0.9
4.9
0.05
93.5
1.9
7.1
0.16
45.6
0.8
4.8
0.05
94.0
1.3
7.5
0.27
8.8 (1-36) 8.8 (1.36)
0.17 (.026) 0.16 (.025)
27.2
0.5
2.88
0.06
93.5
0.6
15.8
0.1
27.7
0.7
2.92
0.07
94.0
1.8
15.4
0.4
Hematocri
Erythrocyte count
Mean corpuscular volume
Leukocyte count
I07MI 10'VI
cu fim ft
1071
% xlfr*
samples as compared with masked samples—an event unlikely to be due to chance alone. The precision of the "superior" and "other" laboratories for the parameters in the abnormal pool was unrelated to their previous performance ranking. However, in the normal range, the "superior" laboratories demonstrated more precision than the "other" laboratories for all parameters except leukocyte count. When the accuracies of the two groups were compared (Table 5), the "other" group had a significantly higher percentage of unacceptable results for normal mean corpuscular volume (MCV) only. Table 6 compares rankings based on summed SDP values with those derived from sums of absolute values of SDI accuracy points and percentages of acceptable results. Overall average performance rankings are also indicated. Components of analytic variability, expressed as percentages, for all determinations, both normal and abnormal, contributed by within-day, day-to-day and interlaboratory variability are given in Table 7. Discussion The results reported by Laboratory 18 were derived from a Hemalog instrument. All other laboratories used the Coulter S. Data reported by Laboratory 18 were more deviant from the target values than those reported by any of the other 19 laboratories. This was particularly true for hematocrit and MCV in both normal and abnormal ranges. The hematocrit on the Hemalog is determined, as in manual methods, by centrifugation, and MCV is obtained mathematically from the erythrocyte count and Hct, whereas the MCV is elec-
All the results for the three-week study were within 6% of the poststudy results. The expiration date indicated by the supplier occurred on the first day after Iheend of the three-week study period.
tronically measured on the Coulter S and the Hct is derived from erythrocyte count and MCV analyses. Thus, theoretically the results of Laboratory 18 might be expected to be closer to manual reference values for these parameters than those of other laboratories. However, the actual performance of Laboratory 18 for both MCV and Hct was among the poorest of all participants. It was thus decided to exclude the data from Laboratory 18 in all statistical analyses. We felt that in this way, the results would be more meaningful in assessing performance in cooperative studies where all laboratories utilized similar instrumentation. The designation of " t r u e " or target values to determine the accuracy of performance is an arbitrary decision, since the actual true values are unknown. In the present study, " t r u e " values were considered to be those values derived from all data excluding those from Laboratory 18 and after eliminating all values lying outside 2 SD. It was considered that this approach would yield reliable target values. In contrast, in assessing performance of individual laboratories, all data were used, even unacceptable values, regardless of their distance from the " t r u e " mean. In fact, the number of such unacceptable results was used as one mechanism for assessing accuracy. It was felt that this strategy would be more likely to detect and give weight to transcription as well as to analytic errors. The " t r u e " means for all five normal and abnormal parameters agreed closely with the overall interlaboratory, stated Dade, and preliminary Albany means (Table 3). The three Hb values were essentially identical, in contrast to MCV, which showed the widest range.
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 6, 2016
Hemoglobin gAII mmol/
Leukocyte Count
SD
* Unils:
Conventional: SI (in parentheses):
Mean Corpuscular Volume
X
Study Normal pool 3-week
Hematocrit
Erythrocyte Count
297
V.A. HEMATOLOGY LABORATORY PERFORMANCE
Vol. 71 . No. 3
It should be emphasized that the "truth" or target values were based on results determined by Coulter S instruments calibrated with Coulter 4C stabilized erythrocytes. Recent evidence7 that such calibration introduces a bias when compared with manual reference methods must be borne in mind. These differences are apparently due to the slower rate of lysis of stabilized erythrocytes as compared with fresh erythrocytes when exposed to stromatolyzing agents.7 The calibration procedure recommended by Bull2 avoids the use of commercial stabilized erythrocytes and utilizes relatively fresh human erythrocytes as an internal standard. This procedure should yield results closer to true values than the target values derived from Coulter S instruments, such as those obtained by the technic described in this report. Based on CAP survey results, Koepke6 suggests
a maximum deviation of ±0.4 g/dl (0.062 mmol/l) as an absolute level of good precision for Hb and ±0.8 g/dl (0.124 mmol/l) as acceptable. In the present study six of 228 results for the normal pool range and three of 228 in the abnormal study range exceeded 0.8 g/dl (0.124 mmol/l). Based on overall study SDs of 0.36 g/dl (0.056 mmol/l) (normal pool) and 0.27 g/dl (0.042 mmol/l) (abnormal pool), 5%, or 11 of 228 values, would normally be expected to lie more than 0.72 g/dl (0.112 mmol/l) (normal pool) or 0.54 g/dl (0.084 mmol/l) (abnormal pool) from the mean. Furthermore, Koepke indicates it is unlikely that additional improvement in instrumentation or methodology will result in significant improvement in precision. It appears that V. A. laboratories perform adequately in analyzing for hemoglobin. The overall interlaboratory SD were better than or were close to the medical requirements for abnormal
Normal Pool
Abnormal Pool
Mean Corpuscular Volume
Mean Corpuscular Volume
Leukocyte Count
Hemoglobin
Hematocrit
Erythrocyte Count
95
7.2
8.8 (1.36)
27.2
2.92
92
15.4
4.88
95
7.1
8.8 (1.36)
27.2
2.91
92
15.3
44.4
4.94
90
7.4
8.7 (1.35)
26.3
2.89
91
15.6
45.7
4.82
94
7.1
8.8 (1.36)
27.2
2.88
94
15.8
0.36 (.056)
1.4
0.12
2.5
0.47
0.27 (.042)
1.1
0.09
2.4
0.78
Day-to-day
0.35 (.054)
1.3
0.11
1.8
0.40
0.25 (.039)
0.9
0.09
1.7
0.71
Within-day
0.29 (•045) 0.12 (.019) 0.46 (.071)
1.0
0.09
0.7
0.29
0.8
0.08
0.8
0.54
0.9
0.05
1.9
0.16
0.5
0.06
0.6
0.10
1.1
0.16
1.6
0.77
0.21 (.033) 0.17 (.026) 0.46 (.071)
1.1
0.16
1.6
0.77
3.1 2.8 2.2 2.4 2.5
2.5 2.3 1.8 1.3 2.0
2.6 1.9 0.7 0.6
6.5 5.6 4.1 2.8
3.1 2.8 2.4 1.6
4.0 3.3 2.9 2.4
3.2 3.1 2.7 3.1
2.6 1.8 0.9 2.0
5.1 4.6 3.5 2.8
—
—
Mean "Truth" Interlaboratory Dade Albany SD Interlaboratory
Albany Med needs Cotlovet CV (%) Interlaboratory Day-to-day Within-day Stated V.A.t CAP 1974
Hemoglobin
Hematocrit
Erythrocyte Count
14.4 (2.23) 14.4 (2.23) 14.5 (2.25) 14.5 (2.25)
46.5
4.89
46.4
2.5 2.4 2.0 1.6 1.4
* Unils: Hemoglobin Conventional: SI (in parentheses):
g/dl mmoiyl
Erythrocyte Hematocrit count % x 10"'
10 V I IO'VI
—
Mean corpuscular volume
Leukocyte count
cu ttm 11
lOV/d 10»/l
—
—
—
—
t See Methods section for explanation of the source of these figures. t Based on previous quality control studies.
Leukocyte Count
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 6, 2016
Table 3. Comparison of Study Means, Standard Deviations, and Coefficients of Variation with Dade, Albany, CAP, and Medical Needs*
298
A.J.C.P. • March 1979
KAPLOW, SCHAUBLE, AND BECKTEL
Table 4. Precision Performance: Comparison of Means, Standard Deviations, and Coefficients of Variation for Masked vs. Open Specimens*
Normal pool Hemoglobin Open
SD
CV (%)
FTest
P
Cooperative Study
LEONARD S. KAPLOW, M.D., MURIEL K. SCHAUBLE, M.D., AN,D JACK M. BECKTEL, M.S.
DESPITE their common linkage, Veterans Administration Hospital Laboratory Services differ in workloads and in calibers and sizes of technical and professional staffs. Some have active residency training programs or other educational functions or both, whereas others Received October 17, 1977; received revised manuscript and accepted for publication January 30, 1978. Supported by the V. A. Cooperative Studies Program. V. A. Central Office, Washington, D. C. Address reprint requests to Dr. Kaplow: Chief, Laboratory Service, Veterans Administration Hospital, West Haven, Connecticut 06516.
Laboratory Service, V. A. Hospital, and Departments of Pathology and Laboratory Medicine, Yale University School of Medicine, New Haven, Connecticut; Clinical Pathology, V. A. Hospital and Department of Pathology, Albany Medical College, Albany, New York; Cooperative Studies Program Coordinating Center, V. A. Hospital, Hines, Illinois
Executive Committee: Schauble. Muriel K.. M.D.. Chairman. Chief, Clinical Pathology, VAH, Albany, New York: Bearman. Jacob E., Ph.D., Professor, Department of Biometry. University of Minnesota, Minneapolis, Minnesota; Becktel, Jack M.. M.S., Statistician. Cooperative Studies Program Coordinating Center, VAH. Hines, Illinois: Gullick. Herbert D.. M.D.. Chief. Surgical Service, VAH, Albany, New York: Henry. John B., M.D., Director of Clinical Pathology, Upstate Medical Center. SUNY. Syracuse, New York: James, Kenneth, Ph.D., Chief. Cooperative Studies Program Coordinating Center, VAH, Hines, Illinois: Kaplow, Leonard S., M.D.*: Steele, Bernard W., M.D.Central Office Coordinators: Hagans. James, M.D., Ph.D.. Chief. Cooperative Studies Program. VACO, Washington, D. C ; Mendel. Julius, Ph.D., Asst. Director. Pathology Service. VACO, Washington. D. C. Participants: Beals. Theodore F.. M.D.. Asst. Chief. Laboratory Service, VAH, Ann Arbor, Michigan; Franco-Jove, Donna, M.D.. Clinical Pathologist, VAH. Atlanta. Georgia; Gill, Edward, Supervisory Chemist, VAH, Buffalo, New York; Hargreaves, Thomas, M.D.. Consultant Chemist. VAH. Palo Alto, California: Horvath. Beni, M.D., Clinical Pathologist, VAH. Boston. Mass.: Jarvis, Bruce W., M.D., Chief, Laboratory Service, VAH, New Orleans, La.; Jimenez, Fidelio, M.D., Chief. Laboratory Service, VAH. Brooklyn, New York; Kaplow, Leonard S.. M.D.,* Chief, Laboratory Service, VAH, West Haven, Conn.; Kassouny. Dicran, M.D., Clinical Pathologist, VAH, New York, New York; Krisko, Istvan, M.D., Staff Pathologist. Laboratory Service. VAH, Houston, Texas; Lingeman, Ralph, M.D.. Clinical Pathologist, VAH, Washington, D. C ; Maldonato, Herbert. M.D.. Clinical Pathologist, VAH, San Juan, Puerto Rico; Mir, Juan A.. M.D., Chief, Laboratory Service, VAH, Chicago, Illinois; Price. Preston, M.D., Chief, Laboratory Service, Oteen. North Carolina; Robinson, C. A., Jr., M.D.. Laboratory Service, VAH, Birmingham, Alabama: Rubnitz, Myron, M.D., Chief, Laboratory Service, VAH, Hines, Illinois; Schatzki, P. F., Ph.D., M.D., Staff Pathologist. VAH. Richmond, Va.; Skerrett. P. V., M.D., Chief, Laboratory Service, VAH, Philadelphia, Pa.; Steele, Bernard W.. M.D.,* Chief, Clinical Chemistry, VAH, Minneapolis, Minn.; White. Harold J., M.D., Chief, Laboratory Service, VAH, Little Rock, Ark. * Listed both in Executive Committee and as Participants.
291
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 6, 2016
Kaplow, Leonard S., Schauble, Muriel K., and Becktel, Jack M.: Validity of hematologic data in Veterans Administration Hospital laboratories. A Veterans Administration cooperative study. Am J Clin Pathol 71:291-300, 1979. Twenty laboratories in Veterans Administration Hospitals, similar in sophistication, were assessed for accuracy and precision in performing hemoglobin (Hb), hematocrit (Hct), erythrocyte count, mean corpuscular volume (MCV), and leukocyte count. Nineteen of the laboratories used the Coulter S counter. Ten laboratories were classified as "superior" by past performance in College of American Pathologists (CAP) surveys. The remainder were selected from other V. A. hospitals. All laboratories analyzed two "masked" and two "open" samples from two pools (normal and abnormal) on the same day, once weekly, for three consecutive weeks (12 samples/pool/lab). Results from 19 laboratories were evaluated for within-day, day-to-day, and interlaboratory variability, effects of masking, and previous performance in CAP surveys. Overall study means were in agreement with target values. An unexpected finding was that withinday variability was greater than day-to-day variability, which was greater than interlaboratory variability (variance components). Precisions for Hb, Hct, erythrocyte count, and MCV were greater than that for leukocyte count. Interlaboratory Standard deviations were less than or close to '' medical needs.'' The data indicated that for three laboratories, "open" samples resulted in a significant bias. It is concluded that V. A. hospital laboratories using similar instrumentation perform at the level of the "state of the art" and could serve as a common resource for cooperative studies requiring Coulter S data. (Key words: Laboratory performance; Laboratory surveys; Clinical hematology laboratory evaluation; Laboratory ranking; Laboratory variation.)
292
KAPLOW, SCHAUBLE, AND BECKTEL
Table I. Average Hospital Profile* Number of beds
700
Square feet of laboratory space
12,000
Workload (weighted tests)* Total Hematology
7.3 million 1.2 million
Productivity (average weighted)* Units/tech-clerical yr (VA recommended standard
172,000 94,000*)
Staffing Pathologists (total) Clinical Pathologists Scientists Supervisors Technologists Technicians Clerks
5.2 1.7 2.5 6.0 20.0 21.0 3.5
Total:
59.9 90% 80%
Research Programs
58%
* At the time of the study.
do not. V. A. Hospitals are particularly suited for cooperative studies, and an efficient centrally directed mechanism for funding and monitoring such programs has been operative for many years. Such studies have made important contributions to medical knowledge. Cooperative research studies may involve a few or numerous participants widely dispersed across the country. They are rarely devoted solely to laboratory aspects of disease, but often utilize data submitted by laboratories from the participating hospitals. Routine hematologic data such as those derived from the Coulter S countert are probably more commonly used for this purpose than any other laboratory procedure. The validity of pooling data from such diverse facilities for statistical use in cooperative research projects is uncertain. The present study was undertaken to determine the precision and accuracy of V. A. Hospital laboratories in performing basic hematologic procedures in an effort to assess their potential reliability as participants in cooperative research activities. It was also intended to evaluate the "state of the art" in such laboratories and to compare their performances with previous results in CAP proficiency surveys and with those of other hematology laboratories for within-day, day-today, and short-term (three weeks) interlaboratory variability. In addition, studies were undertaken to determine whether masking had any influence on the results reported by the various participating laboratories. A t Coulter Electronics, Inc., Hialeah, Fla.
companion study evaluating clinical chemistry laboratory performance was also undertaken.9 Materials and Methods Twenty V. A. Hospital laboratories of diverse geographic locations but similar in sophistication participated in the study. They represent a cross-section of 60 V. A. Hospitals active in cooperative research studies. Ten laboratories had previously been classified as superior, based on past performances in national proficiency surveys. The remaining ten were randomly selected from the other laboratories in the group of 60. A profile of the average participating laboratory is given in Table 1. Commercial^ pools of normal and abnormal stabilized blood cell suspensions were shipped to the Albany V. A. Hospital, which served as the coordinating laboratory, where preliminary tests were carried out to determine the effects of storage on the pooled cell suspensions. Analyses were performed daily at Albany during the three-week study period and for 22 additional consecutive days. However, these data were not included in the study. The outdating period indicated by the supplier expired one day after the last day of the actual study. Materials were monitored and shipped to participants from Albany. Replicate aliquots of the two pools were prepared at Albany, and four such aliquots of each pool were shipped to participating laboratories on Monday for three consecutive weeks. The cell suspensions were stored at 5 ± 3 C and warmed to room temperature prior to analysis. All samples were analyzed on the same day by all participating laboratories during the week of mailing. Ail but one laboratory used the Coulter S counter for analysis. This laboratory (Laboratory 18) used a Hemalog§ instrument. The values reported by this laboratory are indicated in graphic form in Figure 1. However, their data were not used in the statistical analysis. Two samples of each pool were labeled "study specimen" but were otherwise treated as routine samples and represented unmasked specimens. The other two samples of each pool were given fictitious patients' names and were introduced into the normal workload without further identification. These were considered masked samples. Thus, the study accumulated 2,400 bits of information (120 per laboratory), of which 2,280 were used for statistical analysis. All laboratories used Coulter 4C to standardize their instruments. Results were mailed each week to Albany, where they were collated and transmitted to the Hines, Illinois, V. A. $ Dade Corporation, Miami, Fla. 33152. § Technicon Corp., Tarrytown, N. Y.
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 6, 2016
Training Programs (% of hospitals) Clinical Pathology Residency Medical Technology
A.J.C.P. • March 1979
Vol. 71 . No. 3
Individual laboratories were judged for their performances with respect to accuracy and precision on the basis of their summed squared standard deviation index (SDI)2. These values were used rather than SDI values since they are more likely to distinguish truly superior from inferior laboratory performance. This occurs because SDI are approximately standard normal variables; i.e., they may be considered to have (approximately) a gaussian distribution with mean zero and unit standard deviation. In contrast, sums of squares of independent standard normal variables have chi-squared distributions with as many degrees of freedom as the number of variables (i.e., number of SDI/ laboratory; in our case, 10). Tables of chi-squared distributions are readily available and may indicate whether a sum is sufficiently extreme to be of concern. On the other hand, decisions as to whether extreme values are random or significantly deviant are more difficult when one uses SDI values. Laboratories were also ranked for accuracy which was appraised by a scoring technic whereby each lab-
oratory was given a rating of 4 for all values falling within ±1 SD of the "true" value and 2 for each value falling between ± 1 and ±2 SD. No credit was given for values more than ±2 SD from the "true" value. The sum of the ratings for all 120 determinations of each laboratory was considered its overall score and was used to evaluate laboratories for accuracy performance. Finally, accuracy was additionally evaluated by counting the number of individual determinations whose values were more than ±2 SD from the "true" mean and considered unacceptable. By relating the number of such unacceptable results to the total number (120) of determinations, we were able to obtain a percentage index representing a degree of unacceptable performance. These were converted to percentages of acceptable performance by subtracting from 100. Results Preliminary tests at Albany demonstrated only minimal changes in sample values due to storage for three weeks beyond the outdating period (Table 2). Figure 1 indicates the total data derived from the study. The overall study mean ("truth") for 19 laboratories for both normal and abnormal pools for all five parameters are represented by the centrally located horizontal lines calculated from data that excluded outliers. The area outlined by the other two horizontal lines represents 2 SD around this mean. The basic data for each hospital laboratory are located within the vertical rectangles, and the means are indicated by the short horizontal lines. The three daily means derived from these data are indicated by the three solid circles. The data reported by Laboratory 18 are included and represented by the dashed rectangle. However, their values were not used in arriving at the "truth" or in any other statistical analysis. Extreme values reported by individual laboratories are indicated outside the rectangles. They were not excluded in assessing the performance of each laboratory. Thus, these figures are a pictorial representation of within-day, average day-today, and overall study precision and accuracy. The interlaboratory means, standard deviations, and coefficients of variation (CV) are given in Table 3 and are compared with reference values and medical-needs requirements. The overall means, SD, and CV for all five constituents for masked samples were compared with similar data for open samples (Table 4). Based on F tests, statistically significant consistent differences were not observed. However, careful inspection of basic data from individual laboratories indicated a consistent smaller variance for three laboratories for all open
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 6, 2016
Cooperative Study Program Support Center for statistical analysis. Means, standard deviations (SD) and coefficients of variation (CV) were computed for the entire study as well as for within-day, day-to-day, and interlaboratory variability. Values obtained were compared with (1) stated values supplied by Dade, (2) values determined by Albany during the preliminary study, (3) medical-needs variability as defined by Barnett1 for hemoglobin (Hb), and in addition, for all parameters using the concept devised by Cotlove.3 The latter assumes that the SD of the analytic error should be less than or the same as half the standard deviation of the biologic variation. When this is true, the analytic-error component contribution to the total variation is 12% or less. The theoretical Cotlove medicalneeds values were derived from standard deviations obtained from another investigation,8 in which biologic variation for a group of 381 healthy male subjects, aged 20 to 29 years, was determined by participants in this study. Using all data derived from the Coulter S counters, the intralaboratory means and SD for the five normal and five abnormal parameters were determined first. Outliers were categorized as those values lying more than 2 SD from the mean. These observations were excluded and means and SD were then recalculated. These values were considered to be closest to the "truth." However, all data, including outliers, were used to determine the means and SD for individual laboratories for comparison with the "truth" to assess accuracy. In addition, the relative contribution of within-day, day-to-day, and interlaboratory variance to total analytic error was estimated as described for the clinical chemistry study.9
I
293
V.A. HEMATOLOGY LABORATORY PERFORMANCE
I«J6
2
I
3
1
4
1
5
4 43
7
1
8
8
1
y.
• 1*1
9
9
1
•
4 33
1
1
»
•
LL
. 2
» J II1
*
. « m 6 • _ *. • 2
4 1
r
363
i i
m
.
•
19 20
_
L
m... 4*1*" Tali
12 13 14 15 16 17 18
U-»I3 2
10 II
1
1
•
• T ••
'
I7
30.1
1
4 1.7
425
44.1
43.3
«_ 46.3
E 47'7
48.9
3.03
-
4 89
4.37
4.63
"• 4.73
I
J I
*
2
3
L 2 3
4
M
5
-f
8
9
10 II
12 13 14 IS 16 17 18 19 20
_| _J _l 10 II 12 13 14 15 16 17 18 19 20
LABORATORY I.D. NUMBER
7
_J
4M
! !• Ill
:•!
flfHSK
i i 121 i i
—
94
86
88
90 -
92
94
96
96
6
6
1
[_l_
{T * >
1
1
•
'
RED BLOOD COUNT
1
i
jj
2. y 1 •
#
m.
.....
'
31.3
S.I 3
U 4.SI IS
2
3 |
1 1
•
[ [•
•
m»
L 17
! •
Ld 1
Lu
,
1 •
L_ J 15 16
23 3
J.
; i < 1
#
• 4
. 2 .
•
_l 14
*
13
213
«4
m
•
•..
L_ _L II 12
iii:
A
LABORATORY ID. NUMBER
7
1
i.
J.
• 2
7
J 10
227
•ffil
*- v % z. • U 3• 4
?
3 1
VOLUME
7
H
FIG. I. Study means (N = 228) and two standard deviations for normal (above) and abnormal (below) pools for all five modalities tested (hemoglobin, hematocrit, erythrocyte count, mean corpuscular volume, and leukocyte count). Within-day values are enclosed in rectangles with digits representing the number of duplicate or replicate values (N = 12/day). The solid circles represent three daily means for each laboratory (N = 3). Note: The results for Laboratory 18 are depicted but were not used in deriving the study mean. The values for deviant results are indicated above or below the broken end of the appropriate rectangle.
87
89
91
93
93
97
101
103
2 3.6
24.5
2 5.4
26.3
— 28.1 S! _ 27.2 u
29.0
29.9
m m
9A
9.2
30.8
9.6
HEMOGLOBIN
ABNORMAL
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 6, 2016
IP/
A.J.C.P. • March 1979
KAPLOW SCHAUBLE, AND BECKTEL
296
Table 2. Results of Prel iminary Albany Study*
Hemoglobin
22-day Abnormal pool 3-week 22-day
X
SD
X
SD
X
SD
X
SD
14.5 (2.25) 14.4 (2.23)
0.12 (.019) 0.14 (.023)
45.7
0.9
4.9
0.05
93.5
1.9
7.1
0.16
45.6
0.8
4.8
0.05
94.0
1.3
7.5
0.27
8.8 (1-36) 8.8 (1.36)
0.17 (.026) 0.16 (.025)
27.2
0.5
2.88
0.06
93.5
0.6
15.8
0.1
27.7
0.7
2.92
0.07
94.0
1.8
15.4
0.4
Hematocri
Erythrocyte count
Mean corpuscular volume
Leukocyte count
I07MI 10'VI
cu fim ft
1071
% xlfr*
samples as compared with masked samples—an event unlikely to be due to chance alone. The precision of the "superior" and "other" laboratories for the parameters in the abnormal pool was unrelated to their previous performance ranking. However, in the normal range, the "superior" laboratories demonstrated more precision than the "other" laboratories for all parameters except leukocyte count. When the accuracies of the two groups were compared (Table 5), the "other" group had a significantly higher percentage of unacceptable results for normal mean corpuscular volume (MCV) only. Table 6 compares rankings based on summed SDP values with those derived from sums of absolute values of SDI accuracy points and percentages of acceptable results. Overall average performance rankings are also indicated. Components of analytic variability, expressed as percentages, for all determinations, both normal and abnormal, contributed by within-day, day-to-day and interlaboratory variability are given in Table 7. Discussion The results reported by Laboratory 18 were derived from a Hemalog instrument. All other laboratories used the Coulter S. Data reported by Laboratory 18 were more deviant from the target values than those reported by any of the other 19 laboratories. This was particularly true for hematocrit and MCV in both normal and abnormal ranges. The hematocrit on the Hemalog is determined, as in manual methods, by centrifugation, and MCV is obtained mathematically from the erythrocyte count and Hct, whereas the MCV is elec-
All the results for the three-week study were within 6% of the poststudy results. The expiration date indicated by the supplier occurred on the first day after Iheend of the three-week study period.
tronically measured on the Coulter S and the Hct is derived from erythrocyte count and MCV analyses. Thus, theoretically the results of Laboratory 18 might be expected to be closer to manual reference values for these parameters than those of other laboratories. However, the actual performance of Laboratory 18 for both MCV and Hct was among the poorest of all participants. It was thus decided to exclude the data from Laboratory 18 in all statistical analyses. We felt that in this way, the results would be more meaningful in assessing performance in cooperative studies where all laboratories utilized similar instrumentation. The designation of " t r u e " or target values to determine the accuracy of performance is an arbitrary decision, since the actual true values are unknown. In the present study, " t r u e " values were considered to be those values derived from all data excluding those from Laboratory 18 and after eliminating all values lying outside 2 SD. It was considered that this approach would yield reliable target values. In contrast, in assessing performance of individual laboratories, all data were used, even unacceptable values, regardless of their distance from the " t r u e " mean. In fact, the number of such unacceptable results was used as one mechanism for assessing accuracy. It was felt that this strategy would be more likely to detect and give weight to transcription as well as to analytic errors. The " t r u e " means for all five normal and abnormal parameters agreed closely with the overall interlaboratory, stated Dade, and preliminary Albany means (Table 3). The three Hb values were essentially identical, in contrast to MCV, which showed the widest range.
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 6, 2016
Hemoglobin gAII mmol/
Leukocyte Count
SD
* Unils:
Conventional: SI (in parentheses):
Mean Corpuscular Volume
X
Study Normal pool 3-week
Hematocrit
Erythrocyte Count
297
V.A. HEMATOLOGY LABORATORY PERFORMANCE
Vol. 71 . No. 3
It should be emphasized that the "truth" or target values were based on results determined by Coulter S instruments calibrated with Coulter 4C stabilized erythrocytes. Recent evidence7 that such calibration introduces a bias when compared with manual reference methods must be borne in mind. These differences are apparently due to the slower rate of lysis of stabilized erythrocytes as compared with fresh erythrocytes when exposed to stromatolyzing agents.7 The calibration procedure recommended by Bull2 avoids the use of commercial stabilized erythrocytes and utilizes relatively fresh human erythrocytes as an internal standard. This procedure should yield results closer to true values than the target values derived from Coulter S instruments, such as those obtained by the technic described in this report. Based on CAP survey results, Koepke6 suggests
a maximum deviation of ±0.4 g/dl (0.062 mmol/l) as an absolute level of good precision for Hb and ±0.8 g/dl (0.124 mmol/l) as acceptable. In the present study six of 228 results for the normal pool range and three of 228 in the abnormal study range exceeded 0.8 g/dl (0.124 mmol/l). Based on overall study SDs of 0.36 g/dl (0.056 mmol/l) (normal pool) and 0.27 g/dl (0.042 mmol/l) (abnormal pool), 5%, or 11 of 228 values, would normally be expected to lie more than 0.72 g/dl (0.112 mmol/l) (normal pool) or 0.54 g/dl (0.084 mmol/l) (abnormal pool) from the mean. Furthermore, Koepke indicates it is unlikely that additional improvement in instrumentation or methodology will result in significant improvement in precision. It appears that V. A. laboratories perform adequately in analyzing for hemoglobin. The overall interlaboratory SD were better than or were close to the medical requirements for abnormal
Normal Pool
Abnormal Pool
Mean Corpuscular Volume
Mean Corpuscular Volume
Leukocyte Count
Hemoglobin
Hematocrit
Erythrocyte Count
95
7.2
8.8 (1.36)
27.2
2.92
92
15.4
4.88
95
7.1
8.8 (1.36)
27.2
2.91
92
15.3
44.4
4.94
90
7.4
8.7 (1.35)
26.3
2.89
91
15.6
45.7
4.82
94
7.1
8.8 (1.36)
27.2
2.88
94
15.8
0.36 (.056)
1.4
0.12
2.5
0.47
0.27 (.042)
1.1
0.09
2.4
0.78
Day-to-day
0.35 (.054)
1.3
0.11
1.8
0.40
0.25 (.039)
0.9
0.09
1.7
0.71
Within-day
0.29 (•045) 0.12 (.019) 0.46 (.071)
1.0
0.09
0.7
0.29
0.8
0.08
0.8
0.54
0.9
0.05
1.9
0.16
0.5
0.06
0.6
0.10
1.1
0.16
1.6
0.77
0.21 (.033) 0.17 (.026) 0.46 (.071)
1.1
0.16
1.6
0.77
3.1 2.8 2.2 2.4 2.5
2.5 2.3 1.8 1.3 2.0
2.6 1.9 0.7 0.6
6.5 5.6 4.1 2.8
3.1 2.8 2.4 1.6
4.0 3.3 2.9 2.4
3.2 3.1 2.7 3.1
2.6 1.8 0.9 2.0
5.1 4.6 3.5 2.8
—
—
Mean "Truth" Interlaboratory Dade Albany SD Interlaboratory
Albany Med needs Cotlovet CV (%) Interlaboratory Day-to-day Within-day Stated V.A.t CAP 1974
Hemoglobin
Hematocrit
Erythrocyte Count
14.4 (2.23) 14.4 (2.23) 14.5 (2.25) 14.5 (2.25)
46.5
4.89
46.4
2.5 2.4 2.0 1.6 1.4
* Unils: Hemoglobin Conventional: SI (in parentheses):
g/dl mmoiyl
Erythrocyte Hematocrit count % x 10"'
10 V I IO'VI
—
Mean corpuscular volume
Leukocyte count
cu ttm 11
lOV/d 10»/l
—
—
—
—
t See Methods section for explanation of the source of these figures. t Based on previous quality control studies.
Leukocyte Count
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 6, 2016
Table 3. Comparison of Study Means, Standard Deviations, and Coefficients of Variation with Dade, Albany, CAP, and Medical Needs*
298
A.J.C.P. • March 1979
KAPLOW, SCHAUBLE, AND BECKTEL
Table 4. Precision Performance: Comparison of Means, Standard Deviations, and Coefficients of Variation for Masked vs. Open Specimens*
Normal pool Hemoglobin Open
SD
CV (%)
FTest
P
