Clin Chem Lab Med 2015; aop
Arne Åsberg, Ingrid Hov Odsæter*, Sven Magnus Carlsen and Gustav Mikkelsen
Using the likelihood ratio to evaluate allowable total error – an example with glycated hemoglobin (HbA1c) DOI 10.1515/cclm-2014-1125 Received November 17, 2014; accepted March 12, 2015
Abstract Background: Allowable total error is derived in many ways, often from data on biological variation in normal individuals. We present a new principle for evaluating allowable total error: What are the diagnostic consequences of allowable total errors in terms of errors in likelihood ratio (LR)? Glycated hemoglobin A1c in blood (HbA1c) in diagnosing diabetes mellitus is used as an example. Allowable total error for HbA1c is 3.0% derived from data on biological variation compared to 6.0% as defined by National Glycohemoglobin Standardization Program (NGSP). Methods: We estimated a function for LR of HbA1c in diagnosing diabetes mellitus using logistic regression with a clinical database (n = 572) where diabetes status was defined by WHO criteria. Then we estimated errors in LR that correspond to errors in the measurement of HbA1c. Results: Measuring HbA1c 3% too low at HbA1c of 6.5 percentage points (the suggested diagnostic limit) gives a LR of 0.36 times the correct LR, while measuring HbA1c 3% too high gives a LR of 2.77 times the correct LR. The corresponding errors in LR for allowable total error of 6% are 0.13 and 7.69 times the correct LR, respectively. Conclusions: These principles of evaluating allowable total error can be applied to any diagnostically used analyte where the distribution of the analyte’s concentration *Corresponding author: Ingrid Hov Odsæter, Department of Clinical Chemistry, Trondheim University Hospital, 7006 Trondheim, Norway; and Faculty of Medicine, Department of Cancer Research and Molecular Medicine, Norwegian University of Science and Technology, Trondheim, Norway, E-mail: [email protected] Arne Åsberg and Gustav Mikkelsen: Department of Clinical Chemistry, Trondheim University Hospital, Trondheim, Norway Sven Magnus Carlsen: Department of Endocrinology, Trondheim University Hospital, Trondheim, Norway; and Department of Cancer Research and Molecular Medicine, Unit for Applied Clinical Research, Faculty of Medicine, Norwegian University of Science and Technology, Trondheim, Norway
is known in patients with and without the disease in a clinically relevant population. In the example used, the allowable total error of 6% leads to very erroneous LRs, suggesting that the NGSP limits of ±6% are too liberal. Keywords: assessment, healthcare quality; diagnostic errors; hemoglobin A, glycosylated; logistic regression.
Introduction In clinical medicine the likelihood ratios (LRs) of various signs, symptoms and test results can be used to estimate the probability of disease, as the post-test odds is pre-test odds multiplied by the LR [1]. If the concentration of an analyte is measured with error, the associated LR is also in error, resulting in errors in post-test odds. If the analytical result is used for diagnosing a particular disease, the clinician is (or should be) concerned about the deviation of the result from the correct value. Whether the deviation is due to bias or imprecision, or both, is not important to the clinician dealing with one analytical result in a diagnostic situation. In that situation, the clinical needs are best characterized by the total analytical error. The total error of an analytical result is defined in several ways [2], e.g., as the maximum absolute deviation of the 95% less deviating results from repeated analyses of the same sample. Allowable total error has been determined in several ways; however, the diagnostic consequences of the allowable errors have not been assessed in terms of errors in LR [3]. Take, for instance, the relative concentration of glycated hemoglobin in blood (HbA1c), which for decades now has been used to monitor patients with diabetes mellitus [4]. In 2009 an International Expert Committee recommended the use of HbA1c to diagnose diabetes mellitus [5], with a diagnostic limit at 6.5 percentage points (pp) when the method is standardized and aligned to level of the Diabetes Control and Complications Trial (DCCT), corresponding to 48 mmol HbA1c/mol hemoglobin on the scale of the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC). This decision was affirmed
Brought to you by | New York University Bobst Library Technical Services Authenticated Download Date | 6/23/15 12:15 AM
2 Åsberg et al.: Assessing analytical quality goals by World Health Organization (WHO) in 2011 [6]. In the US, for manufacturers or laboratories to pass certification the National Glycohemoglobin Standardization Program (NGSP) demands that “37 of 40 results (38 of 40 for Level I laboratories) need to be within ±6% (relative) of the NGSP SRL” [7] (SRL is secondary reference laboratory). The fractions 37/40 and 38/40 are 0.925 and 0.95, respectively, so the NGSP claims can be regarded as allowable total error. In contrast, the allowable total error based on biological variation is 3.0%, derived from within-subject variation (CVI) of 1.9% and between-subject variation (CVG) of 5.7% [8]. Allowing for a bias of 0.25 ⋅( CVI2 + CVG2 ) 0.5 = 1.5% and imprecision (CVA) of 0.5·CVI = 0.9%, the allowable total error is calculated as 1.5%+1.65·0.9% = 3%. Neither the NGSP claim [9] nor claims based on biological variation are derived from data on the diagnostic performance of HbA1c. The aim of this study was to present the principle of evaluating the diagnostic consequences of allowable total error by estimating the resulting error in LR. HbA1c in diagnosing diabetes mellitus was used as an example.
Materials and methods Population In order to study how measurement errors may create diagnostic errors we need to know the distribution of the analyte concentration in patients with and without the disease in a clinically relevant population. We have established such a clinical database by collecting laboratory data from January 1, 2006 to October 13, 2014 from outpatients, 18 years or older, with a complete data set of analytical results from the following three tests ordered in the same request form: HbA1c, fasting s-glucose and s-glucose 2 h after a 75 g oral glucose load. Patients with more than one complete data set were included with the oldest set only. Individuals were diagnosed with diabetes mellitus using the diagnostic limits of WHO [10]: Fasting s-glucose ≥ 7.0 mmol/L or 2-h s-glucose ≥ 11.1 mmol/L. The laboratory database was searched for results of s-creatinine for the included individuals in the same time period. S-creatinine
Arne Åsberg, Ingrid Hov Odsæter*, Sven Magnus Carlsen and Gustav Mikkelsen
Using the likelihood ratio to evaluate allowable total error – an example with glycated hemoglobin (HbA1c) DOI 10.1515/cclm-2014-1125 Received November 17, 2014; accepted March 12, 2015
Abstract Background: Allowable total error is derived in many ways, often from data on biological variation in normal individuals. We present a new principle for evaluating allowable total error: What are the diagnostic consequences of allowable total errors in terms of errors in likelihood ratio (LR)? Glycated hemoglobin A1c in blood (HbA1c) in diagnosing diabetes mellitus is used as an example. Allowable total error for HbA1c is 3.0% derived from data on biological variation compared to 6.0% as defined by National Glycohemoglobin Standardization Program (NGSP). Methods: We estimated a function for LR of HbA1c in diagnosing diabetes mellitus using logistic regression with a clinical database (n = 572) where diabetes status was defined by WHO criteria. Then we estimated errors in LR that correspond to errors in the measurement of HbA1c. Results: Measuring HbA1c 3% too low at HbA1c of 6.5 percentage points (the suggested diagnostic limit) gives a LR of 0.36 times the correct LR, while measuring HbA1c 3% too high gives a LR of 2.77 times the correct LR. The corresponding errors in LR for allowable total error of 6% are 0.13 and 7.69 times the correct LR, respectively. Conclusions: These principles of evaluating allowable total error can be applied to any diagnostically used analyte where the distribution of the analyte’s concentration *Corresponding author: Ingrid Hov Odsæter, Department of Clinical Chemistry, Trondheim University Hospital, 7006 Trondheim, Norway; and Faculty of Medicine, Department of Cancer Research and Molecular Medicine, Norwegian University of Science and Technology, Trondheim, Norway, E-mail: [email protected] Arne Åsberg and Gustav Mikkelsen: Department of Clinical Chemistry, Trondheim University Hospital, Trondheim, Norway Sven Magnus Carlsen: Department of Endocrinology, Trondheim University Hospital, Trondheim, Norway; and Department of Cancer Research and Molecular Medicine, Unit for Applied Clinical Research, Faculty of Medicine, Norwegian University of Science and Technology, Trondheim, Norway
is known in patients with and without the disease in a clinically relevant population. In the example used, the allowable total error of 6% leads to very erroneous LRs, suggesting that the NGSP limits of ±6% are too liberal. Keywords: assessment, healthcare quality; diagnostic errors; hemoglobin A, glycosylated; logistic regression.
Introduction In clinical medicine the likelihood ratios (LRs) of various signs, symptoms and test results can be used to estimate the probability of disease, as the post-test odds is pre-test odds multiplied by the LR [1]. If the concentration of an analyte is measured with error, the associated LR is also in error, resulting in errors in post-test odds. If the analytical result is used for diagnosing a particular disease, the clinician is (or should be) concerned about the deviation of the result from the correct value. Whether the deviation is due to bias or imprecision, or both, is not important to the clinician dealing with one analytical result in a diagnostic situation. In that situation, the clinical needs are best characterized by the total analytical error. The total error of an analytical result is defined in several ways [2], e.g., as the maximum absolute deviation of the 95% less deviating results from repeated analyses of the same sample. Allowable total error has been determined in several ways; however, the diagnostic consequences of the allowable errors have not been assessed in terms of errors in LR [3]. Take, for instance, the relative concentration of glycated hemoglobin in blood (HbA1c), which for decades now has been used to monitor patients with diabetes mellitus [4]. In 2009 an International Expert Committee recommended the use of HbA1c to diagnose diabetes mellitus [5], with a diagnostic limit at 6.5 percentage points (pp) when the method is standardized and aligned to level of the Diabetes Control and Complications Trial (DCCT), corresponding to 48 mmol HbA1c/mol hemoglobin on the scale of the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC). This decision was affirmed
Brought to you by | New York University Bobst Library Technical Services Authenticated Download Date | 6/23/15 12:15 AM
2 Åsberg et al.: Assessing analytical quality goals by World Health Organization (WHO) in 2011 [6]. In the US, for manufacturers or laboratories to pass certification the National Glycohemoglobin Standardization Program (NGSP) demands that “37 of 40 results (38 of 40 for Level I laboratories) need to be within ±6% (relative) of the NGSP SRL” [7] (SRL is secondary reference laboratory). The fractions 37/40 and 38/40 are 0.925 and 0.95, respectively, so the NGSP claims can be regarded as allowable total error. In contrast, the allowable total error based on biological variation is 3.0%, derived from within-subject variation (CVI) of 1.9% and between-subject variation (CVG) of 5.7% [8]. Allowing for a bias of 0.25 ⋅( CVI2 + CVG2 ) 0.5 = 1.5% and imprecision (CVA) of 0.5·CVI = 0.9%, the allowable total error is calculated as 1.5%+1.65·0.9% = 3%. Neither the NGSP claim [9] nor claims based on biological variation are derived from data on the diagnostic performance of HbA1c. The aim of this study was to present the principle of evaluating the diagnostic consequences of allowable total error by estimating the resulting error in LR. HbA1c in diagnosing diabetes mellitus was used as an example.
Materials and methods Population In order to study how measurement errors may create diagnostic errors we need to know the distribution of the analyte concentration in patients with and without the disease in a clinically relevant population. We have established such a clinical database by collecting laboratory data from January 1, 2006 to October 13, 2014 from outpatients, 18 years or older, with a complete data set of analytical results from the following three tests ordered in the same request form: HbA1c, fasting s-glucose and s-glucose 2 h after a 75 g oral glucose load. Patients with more than one complete data set were included with the oldest set only. Individuals were diagnosed with diabetes mellitus using the diagnostic limits of WHO [10]: Fasting s-glucose ≥ 7.0 mmol/L or 2-h s-glucose ≥ 11.1 mmol/L. The laboratory database was searched for results of s-creatinine for the included individuals in the same time period. S-creatinine
