HEMATOPATHOLOGY Original Article
A Method for the Independent Assessment of the Accuracy of Hematology Whole-Blood Calibrators BRIAN S. BULL, M.D.,1 A. RICHARDSON-JONES, M.D.,2 MALCOLM GIBSON, NZ FIMLT,2 AND DEAN TWEDT, MT (ASCP)2
proach proved both workable and effective in detecting inaccuracies of less than 1%. Few technologist-users have the time or the equipment to verify independently the accuracy of hematologic calibrators. The manufacturers should perform ongoing, independent assessments of the accuracy of their products. The statistical analysis of patient data provides manufacturers with a suitable method. (Key words: Hemoglobin; Erythrocyte indices; Quality control; Laboratory instrumentation) Am J Clin Pathol 1992; 98:623-629
Commercial procedures for assigning values to wholeblood calibrators are inherently self-referential. Both the users and the manufacturers of this material exist in a world of mirrors. The users' quality control programs provided and supported by manufacturers verify with great precision only that the values assigned to the calibrators are being recovered from the users' instruments— instruments that were calibrated to give those values. The manufacturers are only slightly better off. They maintain laboratories capable of implementing reference methods on fresh whole blood. These blood samples are then used to calibrate multichannel analyzers, which are used, in turn, to assign values to the controls and calibrators that the manufacturer provides to the end user. Thus any systematic error in the application of the results of the reference methodology to a particular manufacturer's instruments will be propagated through the sequence from manufacturer to user. Then, via the quality assurance programs provided for the users by the manufacturer, any
systematic errors will be recovered again, but not identified by the manufacturer, because such errors will have been propagated through the entire system. There is no external verification mechanism used by either manufacturer or user that will ensure that the values that one assigns and the other recovers are, in fact, correct. This situation is far from ideal. If the manufacturer's process of assigning reference values to commercial calibrators should develop a bias in one or more parameters, this departure from truth is unlikely to be spotted by users unless they verify independently the manufacturer's assigned values. This is a daunting task that few users attempt. Indeed, it is a task that few users are capable of accomplishing with sufficient accuracy to establish credibly that the manufacturer's value-assignment process is in error. Furthermore, the role of whistle blower is timeconsuming, expensive, and usually thankless. To make matters worse, there is an even more daunting problem confronting a laboratory that tries to monitor the accuracy of manufacturer-produced calibrators. The various proficiency testing programs favored now by state From the 'Department of Pathology, Loma2 Linda University School and national regulatory agencies all assume that the maof Medicine, Loma Linda. California, and Coulter Electronics. Inc.. terial sent out for proficiency testing is stable during shipHialeah, Florida. ment and that if it has been labeled by the manufacturer Received August 19, 1991; revised manuscript accepted for publication then the values supplied are correct. Thus agencies conApril 29, 1992. Address reprint requests to Dr. Bull: Room 2516, LLU Medical Center, clude that an individual user is in error if he or she fails Loma Linda, California 92350. to recover the same value as other users of similar ma623
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 5, 2016
An independent assessment of the accuracy with which a large manufacturer assigned values to hematologic calibrators was performed. Data were collected from 1,767 hospitals and clinics distributed over North America. Statistical analysis of the mean corpuscular volume, mean corpuscular hemoglobin, and mean corpuscular hemoglobin concentration of these patient data confirmed the accuracy of the values for hemoglobin, erythrocyte count, hematocrit, and erythrocyte indices for the calibrators manufactured and released during a 9-month period. This ap-
HEMATOPATHOLOGY
624
Article
MATERIALS AND METHODS A major manufacturer of controls and calibrators for whole-blood multichannel analyzers (Coulter Corp., Hialeah, FL) participated in the 9-month study that took place from April through December 1990. At the beginning of the study, a letter was sent to all North American participants in the manufacturer's Interlaboratory Quality Assurance Program. The letter requested that each participant provide, at the end of each month, copies of the last 20 patient complete blood count (CBC) results that were analyzed. These copies and the participants' quality
A.J.C.P. •
control data were mailed to the manufacturer. All participants used Coulter multichannel analyzers. Several efforts were employed to reduce possible bias. The first was that the participants were not told the purpose of the study. Furthermore, for each group of 20 CBC results submitted, only the values of the 20th sample were entered into the database. By the conclusion of the 9-month study, 1,767 hospitals and clinics had submitted 3,936 sets of 20 CBC results. One thousand twenty-three of the participating institutions submitted only one data set. The remaining 744 institutions supported the study, with varying degrees of regularity, by including sets of 20 CBC results with their monthly quality control data. The erythrocyte indices to be used in the study (i.e., the last value from each group of 20 CBC results) were then grouped, in order of receipt by the manufacturer, into subsets of 20 members. The arithmetic mean indices for MCV, MCH, and MCHC on each subset of 20 patient results were plotted on scattergrams against the time during the 9-month period when the data were acquired. In addition, a histogram for each erythrocyte index was produced, showing the distribution of all the results used in the study (i.e., the 20th result from each submitted CBC set) compared with the average results from each of the n = 20 subsets. Finally, the erythrocyte indices from each n = 20 subset were analyzed by the X B process.3 RESULTS The scattergrams of the raw erythrocyte index data for the subsets of 20 patient results, plotted against the time the data were collected, are shown in Figures \A, 2A, and 3A. Histograms for each erythrocyte index comparing the distribution of all the values used in the study versus the averages of the subsets of data from 20 patients are shown in Figures IB, IB, and 3B. The X B transformation of the data from the subsets of 20 values is shown in Figures 1C, 2C, and 3C. As can be seen from the histograms, the mean values for MCV, MCH, and MCHC are 89.9 fL, 30.5 pg, and 33.9 g/dL, respectively. These values are, reassuringly, in agreement with previous estimates of mean erythrocyte indices in large patient populations (i.e., MCV = 89.5, MCH = 30.5, and MCHC = 34.0). These earlier estimates were based on 1,500 patient samples drawn randomly in groups of 500 from only three institutions rather than from more than 1,700, as in the present case.4 For the grouped subsets of 20 patients, the 95% confidence limits around these means are ± 0.9 fL for the MCV, ± 0.4 pg for the MCH, and ± 0 . 3 g/dL for the MCHC. Inspection of the X B plots of the same data indicates that a drift from 0.3% below the mean MCH and
«r 1992
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 5, 2016
chines (all calibrated to the same manufacturer-supplied calibrators). Few laboratories have the stomach for such an unequal contest with an adversary as formidable as this governmental-industrial complex. Few users will notice when biases develop in commercial calibrators and, because of the impediments already mentioned, even fewer will complain. Fortunately, in the field of hematology whole-blood calibrators and controls, there is an elegant and effective corrective to this problem. It uses the average erythrocyte indices of a patient population as an independent gauge of the accuracy of the values assigned to manufactured controls and calibrators. The larger the patient population involved, the more reliable and effective the method. We here describe the use of a patient population spread out over North America. This process of checking the accuracy of the value assignment process requires collaboration between the manufacturers and the users of calibrators and controls. This is an inconvenience but not a fatal flaw. The bulk of the effort required is the manufacturer's responsibility, the party best equipped to bear it. When applied in their usual formulation, the parameters that the method verifies include the erythrocyte count, hematocrit, hemoglobin, mean corpuscular volume (MCV), mean corpuscular hemoglobin (MCH), and mean corpuscular hemoglobin concentration (MCHC).' In addition, an extended version of the method has the potential to identify a developing bias in the remaining parameters of hematology calibrators.2 This independent verification process provides ongoing information to the user, manufacturer, and regulatory agency that the entire process of value assignment and value recovery is proceeding appropriately. We report a continent-wide trial of such an independent verification process. It proved capable of verifying the accuracy with which a manufacturer assigned values to the several lots of calibrators that were produced and distributed during a 9-month period. This method has proved feasible and reassuringly sensitive to the detection of small deviations in the value assignment process.
BULL ET AL. Independent Assessment of Hematology Whole-Blood Calibrators -I
1
i
1
r-
H
625
h
- f
* • • —• •••••• -O
MCV Units: fL
-H
1
eo
1
1(July]
120
140
1BO
1B0
(D«c.l
Cumulative Number of n=20 Data Sets (Month ol Collection)
FIGS. \A-C. Mean corpuscular volume. A. A scattergram of the subsets of 20 values taken from the 3,936 data sets received. The symbol "i" represents an individual patient MCV. An asterisk identifies the mean of each 20i subset. Ninety-five percent confidence limits were calculated as ±2 SD. The cumulative number of n = 20 subsets gathered are shown on the X axis along with the months of collection. B. The 3,936 patient MCV values (i) are plotted as a distribution (. . .). Each data point in the smaller distribution (_) represents an average of a subset of 20i patient values selected sequentially from the larger distribution. C. A plot of average patient MCV values from the subsets of 20 patient^ values smoothed by application of the XB algorithm. Note that throughout the 9-month period there is no discernable change in the mean MCV value.
-I
1
1
1
1
1
1-
Population Mean: 89.92 fL
Number of ™>°-\ Cases
oJ—I 78
1
1
1
78
BO
82
DATAi DATA 20i
1 ••' 04
I SB
M
BO
BZ
04
OQ
100
102
104
MCV Units: fL
B
MCV
X B 20, 89.9 fL L
Cumulative Number of n=20 Data Sets (Month of Collection)
Vol. 98 • No. 6
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 5, 2016
(April)
DATA (20i) 95%H = 90.8 95%L= 89.0 Mean: 89.9
626
HEMATOPATHOLOGY Original Article H
MCH Units: pg
n.
h
•*• » \ » %•%«
i.=-
# •••••• ••••• -O
•rftt - - * - % - * •
V . " " "1
ao
•O
(July)
120
140
160
ISO
(D»c.)
Cumulative Number of n=20 Data Sets (Month of Collection)
Population Mean: 30.51 pg
Number of Cases
B
DATAi DATA 20i
MCH Units: pg
X B High: 3 0 . 6 7 pg [Sets 130 t o 155: August]
MCH Units: pg
X B 20 n X = 30.5 Pg
30 M
- '
X B Low: 3 0 . 4 5 pg [ S e t s 20 t o 70: May -
(AorllJ
20
40
BO
—I (July)
June]
1
1
1
120
140
160
1— 1B0
(Dae.)
Cumulative Number of n=20 Data Sets (Month of Collection)
A.J.C.P. • December 1992
FIGS. 2A-C. Mean corpuscular hemoglobin. A. A scattergram of the subsets of 20 values taken from the 3,936 data sets received. The symbol "i" represents an individual patient MCH. An asterisk identifies the mean of each 20i subset. Ninety-five percent confidence limits were calculated as ±2 SD. The cumulative number of n = 20 subsets gathered are shown on the X axis along with the months of collection. B. The 3,936 patient MCH values (i) are plotted as a distribution (. . .). Each data point in the smaller distribution (_) represents an average of a subset of 20i patient values selected sequentially from the larger distribution. C. A plot of average patient MCH values from the subsets of 20 patient values smoothed by application of the XB algorithm. Note that throughout the 9month period there is a drift from 30.45 pg to 30.67 pg, an overall drift of about 0.7%.
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 5, 2016
(Apfii)
DATA (20i) 95%H = 30.9 95%L= 30.1 Mean: 30.5
BULL ET
627
AL.
Independent Assessment of Hematology Whole-Blood Calibrators -I
h
MCHC Units: g/dL
»""j.«"
40
BO
SO
1
i
1
h
•*,
(July)
ISO
140
ISO
1>0
* ••••
DATA (20i) 9 5 % H = 34.2
•••• —O
95%L= 33.6 Mean: 33.9
(D«c.)
Cumulative Number of n=20 Data Sets (Month of Collection)
Number of Cases
Populat ion Mean: 33.89 g/dL
DATAi DATA 20i
aoo-
31
92
B
MCHC Units: g/dL H
h-
X B High: 34.00 g/dL [Sets 130 t o 155: August]
MCHC Units: g/dL
* B 20 n 9. = 33.9 g/dl
5?B Low: 3 3 . 8 3 g/dL [ s e t s 20 t o 70: May - June]
lAorll)
[July)
20
120
Cumulative Number of n=20 Data Sets (Month of Collection) Vol. 98 • No. 6
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 5, 2016
FIGS. 3 A-C. Mean corpuscular hemoglobin concentration. A. A scattergram of the subsets of 20 values taken from the 3,936 data sets received. The symbol " i " represents an individual patient MCHC. An asterisk identifies the mean of each 20i subset. Ninety-five percent confidence limits were calculated as ±2 SD. The cumulative number of n = 20 subsets gathered are shown on the X axis along with the months of collection. B. The 3,936 patient MCHC values (i) are plotted as a distribution (. . .). Each data point in the smaller distribution (_) represents an average of a subset of 20i patient values selected sequentially from the larger distribution. C. A plot of average patient MCHC values from the subsets of 20 patient values smoothed by application of the X B algorithm. Note that throughout the 9month period there is an overall drift of about 0.6%, similar to the MCH drift.
H
HEMATOPATHOLOGY
628
Article MCHC to 0.3% above took place during the 9-month period of the study, with a return to the average just as the study ended. DISCUSSION
The analyzers used in the earlier study, like those used in the present one, were all conductance-orifice machines. Furthermore, they were all produced by the same manufacturer (Coulter Corp.). Although other manufacturers have not yet undertaken quality control investigations on a scale similar to these two studies, there are no theoretical reasons to expect that either the approach or the target values will differ with different analyzers. Indeed, pilot experiments with the recovery of patient MCV, MCH, and MCHC values along with the 1984 College of American Pathologists surveys H-50 and H-51 did not identify any differences between several conductance-orifice analyzers and laser-based instruments. 5 Similar conclusions were reached earlier by Koepke and associates,6 who re-
A.J.C.P. •
From the same data set it is possible to derive an estimate of the sensitivity of this independent assessment of the accuracy of hematology calibrators to the detection of bias, should bias be present. Inspection of Figures 1C, 2C, and 3C discloses that although the MCV assignment process showed no detectible drift during the 9-month period of the study, there was a clearly detectable drift in the plots of MCH and MCHC. At the beginning of the study, the mean MCH of the recovered patient data was approximately 30.45 pg. By the seventh month of the study, this had increased to about 30.67 pg, a change of 0.7%. Similar changes are evident on the MCHC plots as well. If this drift had been observed on a whole-blood analyzer in a single user's laboratory, it could have represented either an upward drift in the hemoglobin value or a downward drift in the erythrocyte count. This is the case because an individual analyzer does not determine hemoglobin, hematocrit, and erythrocyte count by separate, primary measurements. Rather the analyzer measures two of these, the erythrocyte count and the hemoglobin value, and then back calculates the hematocrit from the erythrocyte count and a measurement of the MCV. For this reason it would have been impossible to determine whether the erythrocyte count or the hemoglobin value had drifted. This data set, however, does not come from a single user machine. It represents all user machines across North America. The data set reflects faithfully what the manu-
1992
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 5, 2016
The absolute values recovered for the patient mean MCV, MCH, and MCHC values agree with those determined previously from a study using 500 patient results from each of three university hospitals that were located in Japan, the United States, and the United Kingdom. The analyzers used in this earlier international study were calibrated by reference methods specified by the National Committee for Clinical Laboratory Standards for hemoglobin, erythrocyte count, and hematocrit. The agreement between the results of the international study (conducted in 1984) and the North American study (presented in this paper) strongly suggests that the erythrocyte index values (i.e., MCV, MCH, and MCHC) assigned to the patient population by the analyzers used by the 1,767 clinics and hospitals during the North American study were unbiased. Had the manufacturer not assigned accurate values to the calibrators, the average values from the population at large would have differed from the target values determined by reference methodology from the 1,500 patient results used by the international study. An erythrocyte count assigned to the manufactured calibrators used in the North American study that was 5% high would have given rise to population means for MCH and MCHC that were 5% lower than the known population target values for these measurements. Because the means for MCH and MCV are on target, it follows that the manufacturer's calibrants were labeled correctly for erythrocyte count. A similar argument can be made in the case of the hemoglobin values, or else the MCH and the MCHC would differ from the expected target values. Similar arguments apply to the hematocrit values placed on the calibrators.
ported results from laser-based analyzers used in a veterans hospital laboratory for routine patient testing. The values assigned to the commercial calibrators used by this group of 1,767 institutions were unbiased estimates of the erythrocyte count, the hemoglobin, and the hematocrit of those calibrators. That the calibrators were accurately labeled is more certain than it would be if a user in only one of these institutions had recovered the expected target values for the mean erythrocyte indices on just its patient population. When a single user recovers the expected mean indices, there is still the possibility of a consistent processing error (such as inadequately resuspending blood samples before analysis), making all of the primary determinations equally high or low. This uncertainty arises because of the machine-accomplished dilution step(s) that are common to hemoglobin, erythrocyte count, and the hematocrit channels in the user laboratory. This is not the case in the manufacturer's reference laboratory. There each erythrocyte-related parameter is individually analyzed by reference methodology. Each reference method uses separate processes such that precisely compensating errors in three independent reference methodologies would be very unlikely.
BULL ET AL. Independent Assessment of Hematology Whole-Blood Calibrators
This pilot project documented that access to the mean erythrocyte indices of a continent-wide patient population can provide independent evidence that a manufacturer's value-assignment process is correct. The labor involved in collecting the individual patient data is minimal when performed in conjunction with manufacturer-sponsored quality control programs. Can the process be extended to other measurements performed by multichannel whole-blood analyzers? The available statistical evidence suggests that it can. The datacollection process will need modification, however, because the coefficient of variation for these remaining parameters is considerably larger than the approximate 4% characteristic of the erythrocyte indices. Coefficients of variation as large as 40% to 50% are often characteristic of leukocyte and platelet counts in a patient population. These coefficients of variation are so large that data from approximately 500 patients must be averaged to determine the patient mean leukocyte and platelet counts. The process thus becomes unwieldy for the individual laboratory. Even in a large hospital or clinic, an estimate of the accuracy of the leukocyte count channel would only be available once or twice a day at best—far too infrequently for meaningful process control. These limitations do not apply to the manufacturer who is drawing from a continent-wide patient population. If each institution merely submitted the XB average value for leukocyte and platelet counts on a single 20-patient
batch, the statistical distribution of those values would closely approximate that of the single patient values for the erythrocyte indices. The leukocyte and platelet counts from each institution would now show a coefficient of variation of about 5%. These data could then be handled as detailed in this initial study and should prove equally useful as an independent verification process of the constancy of the value-assignment process used by the manufacturer. CONCLUSIONS We have drawn the following conclusions based on our pilot program. Recovery of erythrocyte index data from random patient samples in the continent-at-large is feasible as part of a manufacturer's quality control program. The averages of individual patient erythrocyte indices are stable over time, and this stability provides an independent check on the repeatability of the manufacturer's process for assigning MCV, MCH, MCHC, hemoglobin, hematocrit, and red blood cell count values to calibrants. The closeness with which the averages of patient erythrocyte indices match previously determined values for these indices in large patient populations provides an independent check of the accuracy of the manufacturer's value-assignment process. There is evidence that the simultaneous recovery of XB averages of leukocyte and platelet counts on a single set of 20 random patient specimens from each institution would make it feasible to extend this approach to these, and possibly other, analyzer channels. Acknowledgment. The authors thank Coulter Electronics, who bore the expense of data collection and part of the initial data analysis.
REFERENCES 1. Bull BS, Hay K.L. The blood count, its quality control and related methods: X B calibration and control of the multichannel haematology analysers. In: Chanarin 1, ed. Laboratory Hematology: An Account of Laboratory Techniques, I. London: ChurchillLivingston, 1989, pp 3-9. 2. Bull BS. Quality assurance strategies. In: Koepke JA, ed. Practical Laboratory Hematology. New York: Churchill-Livingston, 1991, pp 3-29. 3. Bull BS, Elashoff RM, Heilbron DC, Couperus J. A study of various estimators for the derivation of quality control procedures from patient erythrocyte indices. Am J Clin Pathol 1975;61:473-481. 4. Bull BS, Hay KL. Are red cell indices international? Arch Pathol Lab Med 1985;109:604-606. 5. Levy WC, Bull BS, Koepke JA. The incorporation of RBC index mean data into quality control programs. Am J Clin Pathol 1986;86:193-199. 6. Koepke JA, Protexter TJ. Quality assurance for multichannel hematology instruments. Four years' experience with patient mean erythrocyte indices. Am J Clin Pathol 1981;75:28-33.
Vol. 98 • No. 6
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 5, 2016
facturer's "machine," the entire value assignment process, has produced. Because the manufacturer analyzes directly for erythrocyte count, hemoglobin, and hematocrit, the most probable explanation that satisfies the data is that the instability arose in the hemoglobin assignment process. Had the instability arisen in the erythrocyte count, then the MCV, rather than the MCHC, would have shown an effect similar to that seen in the MCH. Furthermore, the data show that the method is sensitive enough to detect and quantify an instability in the hemoglobin assignment process of ±0.3%. This is sensitivity well below the level of clinical significance and at a level that would be undetectable to all users. Indeed, to detect a change in the value-assignment process at the 1% level, a user would have to be equipped to perform reference assays on large numbers of fresh whole-blood samples—numbers on the same order as those used by the manufacturer during the value-assignment process. Even so, it would be difficult for a user to establish that such a minute difference existed, much less document that any difference found was due to an error in the manufacturer's assigned value.
629
A Method for the Independent Assessment of the Accuracy of Hematology Whole-Blood Calibrators BRIAN S. BULL, M.D.,1 A. RICHARDSON-JONES, M.D.,2 MALCOLM GIBSON, NZ FIMLT,2 AND DEAN TWEDT, MT (ASCP)2
proach proved both workable and effective in detecting inaccuracies of less than 1%. Few technologist-users have the time or the equipment to verify independently the accuracy of hematologic calibrators. The manufacturers should perform ongoing, independent assessments of the accuracy of their products. The statistical analysis of patient data provides manufacturers with a suitable method. (Key words: Hemoglobin; Erythrocyte indices; Quality control; Laboratory instrumentation) Am J Clin Pathol 1992; 98:623-629
Commercial procedures for assigning values to wholeblood calibrators are inherently self-referential. Both the users and the manufacturers of this material exist in a world of mirrors. The users' quality control programs provided and supported by manufacturers verify with great precision only that the values assigned to the calibrators are being recovered from the users' instruments— instruments that were calibrated to give those values. The manufacturers are only slightly better off. They maintain laboratories capable of implementing reference methods on fresh whole blood. These blood samples are then used to calibrate multichannel analyzers, which are used, in turn, to assign values to the controls and calibrators that the manufacturer provides to the end user. Thus any systematic error in the application of the results of the reference methodology to a particular manufacturer's instruments will be propagated through the sequence from manufacturer to user. Then, via the quality assurance programs provided for the users by the manufacturer, any
systematic errors will be recovered again, but not identified by the manufacturer, because such errors will have been propagated through the entire system. There is no external verification mechanism used by either manufacturer or user that will ensure that the values that one assigns and the other recovers are, in fact, correct. This situation is far from ideal. If the manufacturer's process of assigning reference values to commercial calibrators should develop a bias in one or more parameters, this departure from truth is unlikely to be spotted by users unless they verify independently the manufacturer's assigned values. This is a daunting task that few users attempt. Indeed, it is a task that few users are capable of accomplishing with sufficient accuracy to establish credibly that the manufacturer's value-assignment process is in error. Furthermore, the role of whistle blower is timeconsuming, expensive, and usually thankless. To make matters worse, there is an even more daunting problem confronting a laboratory that tries to monitor the accuracy of manufacturer-produced calibrators. The various proficiency testing programs favored now by state From the 'Department of Pathology, Loma2 Linda University School and national regulatory agencies all assume that the maof Medicine, Loma Linda. California, and Coulter Electronics. Inc.. terial sent out for proficiency testing is stable during shipHialeah, Florida. ment and that if it has been labeled by the manufacturer Received August 19, 1991; revised manuscript accepted for publication then the values supplied are correct. Thus agencies conApril 29, 1992. Address reprint requests to Dr. Bull: Room 2516, LLU Medical Center, clude that an individual user is in error if he or she fails Loma Linda, California 92350. to recover the same value as other users of similar ma623
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 5, 2016
An independent assessment of the accuracy with which a large manufacturer assigned values to hematologic calibrators was performed. Data were collected from 1,767 hospitals and clinics distributed over North America. Statistical analysis of the mean corpuscular volume, mean corpuscular hemoglobin, and mean corpuscular hemoglobin concentration of these patient data confirmed the accuracy of the values for hemoglobin, erythrocyte count, hematocrit, and erythrocyte indices for the calibrators manufactured and released during a 9-month period. This ap-
HEMATOPATHOLOGY
624
Article
MATERIALS AND METHODS A major manufacturer of controls and calibrators for whole-blood multichannel analyzers (Coulter Corp., Hialeah, FL) participated in the 9-month study that took place from April through December 1990. At the beginning of the study, a letter was sent to all North American participants in the manufacturer's Interlaboratory Quality Assurance Program. The letter requested that each participant provide, at the end of each month, copies of the last 20 patient complete blood count (CBC) results that were analyzed. These copies and the participants' quality
A.J.C.P. •
control data were mailed to the manufacturer. All participants used Coulter multichannel analyzers. Several efforts were employed to reduce possible bias. The first was that the participants were not told the purpose of the study. Furthermore, for each group of 20 CBC results submitted, only the values of the 20th sample were entered into the database. By the conclusion of the 9-month study, 1,767 hospitals and clinics had submitted 3,936 sets of 20 CBC results. One thousand twenty-three of the participating institutions submitted only one data set. The remaining 744 institutions supported the study, with varying degrees of regularity, by including sets of 20 CBC results with their monthly quality control data. The erythrocyte indices to be used in the study (i.e., the last value from each group of 20 CBC results) were then grouped, in order of receipt by the manufacturer, into subsets of 20 members. The arithmetic mean indices for MCV, MCH, and MCHC on each subset of 20 patient results were plotted on scattergrams against the time during the 9-month period when the data were acquired. In addition, a histogram for each erythrocyte index was produced, showing the distribution of all the results used in the study (i.e., the 20th result from each submitted CBC set) compared with the average results from each of the n = 20 subsets. Finally, the erythrocyte indices from each n = 20 subset were analyzed by the X B process.3 RESULTS The scattergrams of the raw erythrocyte index data for the subsets of 20 patient results, plotted against the time the data were collected, are shown in Figures \A, 2A, and 3A. Histograms for each erythrocyte index comparing the distribution of all the values used in the study versus the averages of the subsets of data from 20 patients are shown in Figures IB, IB, and 3B. The X B transformation of the data from the subsets of 20 values is shown in Figures 1C, 2C, and 3C. As can be seen from the histograms, the mean values for MCV, MCH, and MCHC are 89.9 fL, 30.5 pg, and 33.9 g/dL, respectively. These values are, reassuringly, in agreement with previous estimates of mean erythrocyte indices in large patient populations (i.e., MCV = 89.5, MCH = 30.5, and MCHC = 34.0). These earlier estimates were based on 1,500 patient samples drawn randomly in groups of 500 from only three institutions rather than from more than 1,700, as in the present case.4 For the grouped subsets of 20 patients, the 95% confidence limits around these means are ± 0.9 fL for the MCV, ± 0.4 pg for the MCH, and ± 0 . 3 g/dL for the MCHC. Inspection of the X B plots of the same data indicates that a drift from 0.3% below the mean MCH and
«r 1992
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 5, 2016
chines (all calibrated to the same manufacturer-supplied calibrators). Few laboratories have the stomach for such an unequal contest with an adversary as formidable as this governmental-industrial complex. Few users will notice when biases develop in commercial calibrators and, because of the impediments already mentioned, even fewer will complain. Fortunately, in the field of hematology whole-blood calibrators and controls, there is an elegant and effective corrective to this problem. It uses the average erythrocyte indices of a patient population as an independent gauge of the accuracy of the values assigned to manufactured controls and calibrators. The larger the patient population involved, the more reliable and effective the method. We here describe the use of a patient population spread out over North America. This process of checking the accuracy of the value assignment process requires collaboration between the manufacturers and the users of calibrators and controls. This is an inconvenience but not a fatal flaw. The bulk of the effort required is the manufacturer's responsibility, the party best equipped to bear it. When applied in their usual formulation, the parameters that the method verifies include the erythrocyte count, hematocrit, hemoglobin, mean corpuscular volume (MCV), mean corpuscular hemoglobin (MCH), and mean corpuscular hemoglobin concentration (MCHC).' In addition, an extended version of the method has the potential to identify a developing bias in the remaining parameters of hematology calibrators.2 This independent verification process provides ongoing information to the user, manufacturer, and regulatory agency that the entire process of value assignment and value recovery is proceeding appropriately. We report a continent-wide trial of such an independent verification process. It proved capable of verifying the accuracy with which a manufacturer assigned values to the several lots of calibrators that were produced and distributed during a 9-month period. This method has proved feasible and reassuringly sensitive to the detection of small deviations in the value assignment process.
BULL ET AL. Independent Assessment of Hematology Whole-Blood Calibrators -I
1
i
1
r-
H
625
h
- f
* • • —• •••••• -O
MCV Units: fL
-H
1
eo
1
1(July]
120
140
1BO
1B0
(D«c.l
Cumulative Number of n=20 Data Sets (Month ol Collection)
FIGS. \A-C. Mean corpuscular volume. A. A scattergram of the subsets of 20 values taken from the 3,936 data sets received. The symbol "i" represents an individual patient MCV. An asterisk identifies the mean of each 20i subset. Ninety-five percent confidence limits were calculated as ±2 SD. The cumulative number of n = 20 subsets gathered are shown on the X axis along with the months of collection. B. The 3,936 patient MCV values (i) are plotted as a distribution (. . .). Each data point in the smaller distribution (_) represents an average of a subset of 20i patient values selected sequentially from the larger distribution. C. A plot of average patient MCV values from the subsets of 20 patient^ values smoothed by application of the XB algorithm. Note that throughout the 9-month period there is no discernable change in the mean MCV value.
-I
1
1
1
1
1
1-
Population Mean: 89.92 fL
Number of ™>°-\ Cases
oJ—I 78
1
1
1
78
BO
82
DATAi DATA 20i
1 ••' 04
I SB
M
BO
BZ
04
OQ
100
102
104
MCV Units: fL
B
MCV
X B 20, 89.9 fL L
Cumulative Number of n=20 Data Sets (Month of Collection)
Vol. 98 • No. 6
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 5, 2016
(April)
DATA (20i) 95%H = 90.8 95%L= 89.0 Mean: 89.9
626
HEMATOPATHOLOGY Original Article H
MCH Units: pg
n.
h
•*• » \ » %•%«
i.=-
# •••••• ••••• -O
•rftt - - * - % - * •
V . " " "1
ao
•O
(July)
120
140
160
ISO
(D»c.)
Cumulative Number of n=20 Data Sets (Month of Collection)
Population Mean: 30.51 pg
Number of Cases
B
DATAi DATA 20i
MCH Units: pg
X B High: 3 0 . 6 7 pg [Sets 130 t o 155: August]
MCH Units: pg
X B 20 n X = 30.5 Pg
30 M
- '
X B Low: 3 0 . 4 5 pg [ S e t s 20 t o 70: May -
(AorllJ
20
40
BO
—I (July)
June]
1
1
1
120
140
160
1— 1B0
(Dae.)
Cumulative Number of n=20 Data Sets (Month of Collection)
A.J.C.P. • December 1992
FIGS. 2A-C. Mean corpuscular hemoglobin. A. A scattergram of the subsets of 20 values taken from the 3,936 data sets received. The symbol "i" represents an individual patient MCH. An asterisk identifies the mean of each 20i subset. Ninety-five percent confidence limits were calculated as ±2 SD. The cumulative number of n = 20 subsets gathered are shown on the X axis along with the months of collection. B. The 3,936 patient MCH values (i) are plotted as a distribution (. . .). Each data point in the smaller distribution (_) represents an average of a subset of 20i patient values selected sequentially from the larger distribution. C. A plot of average patient MCH values from the subsets of 20 patient values smoothed by application of the XB algorithm. Note that throughout the 9month period there is a drift from 30.45 pg to 30.67 pg, an overall drift of about 0.7%.
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 5, 2016
(Apfii)
DATA (20i) 95%H = 30.9 95%L= 30.1 Mean: 30.5
BULL ET
627
AL.
Independent Assessment of Hematology Whole-Blood Calibrators -I
h
MCHC Units: g/dL
»""j.«"
40
BO
SO
1
i
1
h
•*,
(July)
ISO
140
ISO
1>0
* ••••
DATA (20i) 9 5 % H = 34.2
•••• —O
95%L= 33.6 Mean: 33.9
(D«c.)
Cumulative Number of n=20 Data Sets (Month of Collection)
Number of Cases
Populat ion Mean: 33.89 g/dL
DATAi DATA 20i
aoo-
31
92
B
MCHC Units: g/dL H
h-
X B High: 34.00 g/dL [Sets 130 t o 155: August]
MCHC Units: g/dL
* B 20 n 9. = 33.9 g/dl
5?B Low: 3 3 . 8 3 g/dL [ s e t s 20 t o 70: May - June]
lAorll)
[July)
20
120
Cumulative Number of n=20 Data Sets (Month of Collection) Vol. 98 • No. 6
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 5, 2016
FIGS. 3 A-C. Mean corpuscular hemoglobin concentration. A. A scattergram of the subsets of 20 values taken from the 3,936 data sets received. The symbol " i " represents an individual patient MCHC. An asterisk identifies the mean of each 20i subset. Ninety-five percent confidence limits were calculated as ±2 SD. The cumulative number of n = 20 subsets gathered are shown on the X axis along with the months of collection. B. The 3,936 patient MCHC values (i) are plotted as a distribution (. . .). Each data point in the smaller distribution (_) represents an average of a subset of 20i patient values selected sequentially from the larger distribution. C. A plot of average patient MCHC values from the subsets of 20 patient values smoothed by application of the X B algorithm. Note that throughout the 9month period there is an overall drift of about 0.6%, similar to the MCH drift.
H
HEMATOPATHOLOGY
628
Article MCHC to 0.3% above took place during the 9-month period of the study, with a return to the average just as the study ended. DISCUSSION
The analyzers used in the earlier study, like those used in the present one, were all conductance-orifice machines. Furthermore, they were all produced by the same manufacturer (Coulter Corp.). Although other manufacturers have not yet undertaken quality control investigations on a scale similar to these two studies, there are no theoretical reasons to expect that either the approach or the target values will differ with different analyzers. Indeed, pilot experiments with the recovery of patient MCV, MCH, and MCHC values along with the 1984 College of American Pathologists surveys H-50 and H-51 did not identify any differences between several conductance-orifice analyzers and laser-based instruments. 5 Similar conclusions were reached earlier by Koepke and associates,6 who re-
A.J.C.P. •
From the same data set it is possible to derive an estimate of the sensitivity of this independent assessment of the accuracy of hematology calibrators to the detection of bias, should bias be present. Inspection of Figures 1C, 2C, and 3C discloses that although the MCV assignment process showed no detectible drift during the 9-month period of the study, there was a clearly detectable drift in the plots of MCH and MCHC. At the beginning of the study, the mean MCH of the recovered patient data was approximately 30.45 pg. By the seventh month of the study, this had increased to about 30.67 pg, a change of 0.7%. Similar changes are evident on the MCHC plots as well. If this drift had been observed on a whole-blood analyzer in a single user's laboratory, it could have represented either an upward drift in the hemoglobin value or a downward drift in the erythrocyte count. This is the case because an individual analyzer does not determine hemoglobin, hematocrit, and erythrocyte count by separate, primary measurements. Rather the analyzer measures two of these, the erythrocyte count and the hemoglobin value, and then back calculates the hematocrit from the erythrocyte count and a measurement of the MCV. For this reason it would have been impossible to determine whether the erythrocyte count or the hemoglobin value had drifted. This data set, however, does not come from a single user machine. It represents all user machines across North America. The data set reflects faithfully what the manu-
1992
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 5, 2016
The absolute values recovered for the patient mean MCV, MCH, and MCHC values agree with those determined previously from a study using 500 patient results from each of three university hospitals that were located in Japan, the United States, and the United Kingdom. The analyzers used in this earlier international study were calibrated by reference methods specified by the National Committee for Clinical Laboratory Standards for hemoglobin, erythrocyte count, and hematocrit. The agreement between the results of the international study (conducted in 1984) and the North American study (presented in this paper) strongly suggests that the erythrocyte index values (i.e., MCV, MCH, and MCHC) assigned to the patient population by the analyzers used by the 1,767 clinics and hospitals during the North American study were unbiased. Had the manufacturer not assigned accurate values to the calibrators, the average values from the population at large would have differed from the target values determined by reference methodology from the 1,500 patient results used by the international study. An erythrocyte count assigned to the manufactured calibrators used in the North American study that was 5% high would have given rise to population means for MCH and MCHC that were 5% lower than the known population target values for these measurements. Because the means for MCH and MCV are on target, it follows that the manufacturer's calibrants were labeled correctly for erythrocyte count. A similar argument can be made in the case of the hemoglobin values, or else the MCH and the MCHC would differ from the expected target values. Similar arguments apply to the hematocrit values placed on the calibrators.
ported results from laser-based analyzers used in a veterans hospital laboratory for routine patient testing. The values assigned to the commercial calibrators used by this group of 1,767 institutions were unbiased estimates of the erythrocyte count, the hemoglobin, and the hematocrit of those calibrators. That the calibrators were accurately labeled is more certain than it would be if a user in only one of these institutions had recovered the expected target values for the mean erythrocyte indices on just its patient population. When a single user recovers the expected mean indices, there is still the possibility of a consistent processing error (such as inadequately resuspending blood samples before analysis), making all of the primary determinations equally high or low. This uncertainty arises because of the machine-accomplished dilution step(s) that are common to hemoglobin, erythrocyte count, and the hematocrit channels in the user laboratory. This is not the case in the manufacturer's reference laboratory. There each erythrocyte-related parameter is individually analyzed by reference methodology. Each reference method uses separate processes such that precisely compensating errors in three independent reference methodologies would be very unlikely.
BULL ET AL. Independent Assessment of Hematology Whole-Blood Calibrators
This pilot project documented that access to the mean erythrocyte indices of a continent-wide patient population can provide independent evidence that a manufacturer's value-assignment process is correct. The labor involved in collecting the individual patient data is minimal when performed in conjunction with manufacturer-sponsored quality control programs. Can the process be extended to other measurements performed by multichannel whole-blood analyzers? The available statistical evidence suggests that it can. The datacollection process will need modification, however, because the coefficient of variation for these remaining parameters is considerably larger than the approximate 4% characteristic of the erythrocyte indices. Coefficients of variation as large as 40% to 50% are often characteristic of leukocyte and platelet counts in a patient population. These coefficients of variation are so large that data from approximately 500 patients must be averaged to determine the patient mean leukocyte and platelet counts. The process thus becomes unwieldy for the individual laboratory. Even in a large hospital or clinic, an estimate of the accuracy of the leukocyte count channel would only be available once or twice a day at best—far too infrequently for meaningful process control. These limitations do not apply to the manufacturer who is drawing from a continent-wide patient population. If each institution merely submitted the XB average value for leukocyte and platelet counts on a single 20-patient
batch, the statistical distribution of those values would closely approximate that of the single patient values for the erythrocyte indices. The leukocyte and platelet counts from each institution would now show a coefficient of variation of about 5%. These data could then be handled as detailed in this initial study and should prove equally useful as an independent verification process of the constancy of the value-assignment process used by the manufacturer. CONCLUSIONS We have drawn the following conclusions based on our pilot program. Recovery of erythrocyte index data from random patient samples in the continent-at-large is feasible as part of a manufacturer's quality control program. The averages of individual patient erythrocyte indices are stable over time, and this stability provides an independent check on the repeatability of the manufacturer's process for assigning MCV, MCH, MCHC, hemoglobin, hematocrit, and red blood cell count values to calibrants. The closeness with which the averages of patient erythrocyte indices match previously determined values for these indices in large patient populations provides an independent check of the accuracy of the manufacturer's value-assignment process. There is evidence that the simultaneous recovery of XB averages of leukocyte and platelet counts on a single set of 20 random patient specimens from each institution would make it feasible to extend this approach to these, and possibly other, analyzer channels. Acknowledgment. The authors thank Coulter Electronics, who bore the expense of data collection and part of the initial data analysis.
REFERENCES 1. Bull BS, Hay K.L. The blood count, its quality control and related methods: X B calibration and control of the multichannel haematology analysers. In: Chanarin 1, ed. Laboratory Hematology: An Account of Laboratory Techniques, I. London: ChurchillLivingston, 1989, pp 3-9. 2. Bull BS. Quality assurance strategies. In: Koepke JA, ed. Practical Laboratory Hematology. New York: Churchill-Livingston, 1991, pp 3-29. 3. Bull BS, Elashoff RM, Heilbron DC, Couperus J. A study of various estimators for the derivation of quality control procedures from patient erythrocyte indices. Am J Clin Pathol 1975;61:473-481. 4. Bull BS, Hay KL. Are red cell indices international? Arch Pathol Lab Med 1985;109:604-606. 5. Levy WC, Bull BS, Koepke JA. The incorporation of RBC index mean data into quality control programs. Am J Clin Pathol 1986;86:193-199. 6. Koepke JA, Protexter TJ. Quality assurance for multichannel hematology instruments. Four years' experience with patient mean erythrocyte indices. Am J Clin Pathol 1981;75:28-33.
Vol. 98 • No. 6
Downloaded from http://ajcp.oxfordjournals.org/ by guest on June 5, 2016
facturer's "machine," the entire value assignment process, has produced. Because the manufacturer analyzes directly for erythrocyte count, hemoglobin, and hematocrit, the most probable explanation that satisfies the data is that the instability arose in the hemoglobin assignment process. Had the instability arisen in the erythrocyte count, then the MCV, rather than the MCHC, would have shown an effect similar to that seen in the MCH. Furthermore, the data show that the method is sensitive enough to detect and quantify an instability in the hemoglobin assignment process of ±0.3%. This is sensitivity well below the level of clinical significance and at a level that would be undetectable to all users. Indeed, to detect a change in the value-assignment process at the 1% level, a user would have to be equipped to perform reference assays on large numbers of fresh whole-blood samples—numbers on the same order as those used by the manufacturer during the value-assignment process. Even so, it would be difficult for a user to establish that such a minute difference existed, much less document that any difference found was due to an error in the manufacturer's assigned value.
629
