AUTHOR REPLY
Reply to “Better Method for Evaluating a New Laboratory Test for Syphilis” Laboratory of G. Enders & Colleagues, MZV and Institute of Virology, Infectious Diseases and Epidemiology e. V., Stuttgart, Germanya; Labor Schottdorf MZV GmbH, Augsburg, Germanyb; Central Institute for Blood Transfusion and Immunology, Innsbruck, Austriac; Thai Red Cross Society, Bangkok, Thailandd; Faculty of Medicine, Ege University, Smyrna, Turkeye; The Greater Romagna Area Hub Laboratory, Unit of Microbiology, Pievesestina, Cesena, Italyf
T
his article is in response to the correspondence from Zhang et al. (1) relating to the methodology used in the study by Enders et al. (2). The correspondence from Zhang et al. does not present a better method. Instead, it criticizes a well-known and widely used method for evaluating diagnostic tests. The argument proposed by Zhang et al. contains several flaws, which we feel are important to address. They assert that it is not appropriate to assess sensitivity based on samples positive for syphilis infection or to assess specificity based on samples negative for syphilis infection. However, the European Union common technical specifications and the FDA guidance for diagnostic accuracy agree that these parameters should be used to assess sensitivity and specificity (3, 4). Zhang et al. also suggest that sensitivity and specificity do not adequately describe the performance of an assay in clinical practice. While we agree that sensitivity and specificity are technical measures of diagnostic accuracy and cannot completely reflect assay performance in routine use, we maintain that best-in-class sensitivity and specificity data are prerequisites for any assay considered for clinical use. As Zhang et al. describe, the negative predictive value (NPV) and positive predictive value (PPV) of an assay are dependent on disease prevalence and cannot be evaluated in isolation to assess the quality of a diagnostic method. Sensitivity and specificity, as calculated by Enders et al., provide standard information that can be used to compare diagnostic tests regardless of the clinical setting. Assessment of a new assay typically compares its performance to that of the existing accepted gold standard diagnostic method. Currently, there is no such accepted gold standard diagnostic method for syphilis, and if the assessment of the Elecsys syphilis assay by Enders et al. contains possible bias, we believe this to be the most likely source. We agree that it is important to utilize the correct target population for the assay to minimize bias and to make the assessment of the assay as relevant as possible. We assessed the specificity of the assay in samples from a wide range of relevant populations (blood banks, prenatal screening, routine diagnostic requests, patients assessed at HIV clinics). Overestimation of specificity is therefore unlikely. It is more reasonable to suggest that the specificity of the comparator assays has been overestimated, when those assays were evaluated in samples from blood banks, where a majority of repeat donors can be assumed, resulting in a preselection bias favoring the routine assay due to donors with reactive results (including false reactive) being deferred from repeat donations. Regarding the sensitivity assessment, an enrichment strat-
May 2015 Volume 22 Number 5
egy was implemented in the study design. This was intended to overcome the rare occurrence of syphilis-positive samples in a study design that includes all patients from the target populations. The enrichment strategy also allowed for the inclusion of samples from various stages of syphilis infection, minimizing the potential bias caused by including only samples with easily detectable, severe infection. The absence of a diagnostic gold standard for syphilis infection means that it is possible that a small number of the precharacterized positive samples were falsely included and may represent false-positive samples. Underestimation of the sensitivity, as it compares with that determined for samples characterized using an imperfect existing standard, is therefore possible. It is correct that, for individual clinicians, the clinically relevant NPV and PPV require local prevalence data in specific patient cohorts. Although calculating these numbers in cohorts, given the knowledge about the individual prevalence, is not difficult, calculating prevalence and NPV or PPV across all the different cohorts, particularly the sensitivity-enriched cohorts, used in the study by Enders et al. and comparing the results to CDC statistics from the United States (as exemplified in the introduction of the article) is in our opinion very adventurous. Instead of extrapolating and supposing prevalence values, we accurately present sensitivity and specificity, calculated using the methods suggested by the FDA and European Union guidelines. If the prevalence of disease in each cohort must be calculated before the effectiveness of a new diagnostic assay is accepted, evaluation of new assays will become highly complex. REFERENCES 1. Zhang Q, Zhang Y-F, Chen F-Y, Liu L, Yang T-C, Niu J-J, Liu L-L. 2015. Better method for evaluating a new laboratory test for syphilis. Clin Vaccine Immunol 22:606. http://dx.doi.org/10.1128/CVI.00014-15. 2. Enders M, Hunjet A, Gleich M, Imdahl R, Mühlbacher A, Schennach H, Chaiwong K, Sakuldamrongpanich T, Turhan A, Sertöz R, Wolf E,
Citation Enders M, Gleich M, Mühlbacher A, Sakuldamrongpanich T, Turhan A, Sertöz R, Semprini S, Sambri V. 2015. Reply to “Better method for evaluating a new laboratory test for syphilis.” Clin Vaccine Immunol 22:607– 608. doi:10.1128/CVI.00109-15. Editor: T. S. Alexander Address correspondence to Vittorio Sambri, [email protected]. This is a response to a letter by Zhang et al. (doi:10.1128/CVI.00014-15). Copyright © 2015, American Society for Microbiology. All Rights Reserved. doi:10.1128/CVI.00109-15
Clinical and Vaccine Immunology
cvi.asm.org
607
Downloaded from http://cvi.asm.org/ on May 27, 2015 by NANYANG TECHNOLOGICAL UNIVERSITY
Martin Enders,a Michael Gleich,b Annelies Mühlbacher,c Tasanee Sakuldamrongpanich,d Ajda Turhan,e Rüchan Sertöz,e Simona Semprini,f Vittorio Sambrif
Author Reply
Mayer W, Tao C, Wang LL, Semprini S, Sambri V. 2015. Performance evaluation of the Elecsys syphilis assay for the detection of total antibodies to Treponema pallidum. Clin Vaccine Immunol 22:17–26. http://dx .doi.org/10.1128/CVI.00505-14. 3. Official Journal of the European Union. 2009. Commission decision of 27 November 2009 amending Decision 2002/364/EC on common technical specifications for in vitro diagnostic medical devices.
http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri⫽OJ:L:2009 :318:0025:0040:EN:PDF. Accessed 17 February 2015. 4. U.S. Food and Drug Administration. Statistical guidance on reporting results from studies evaluating diagnostic tests (guidance for industry and FDA staff—version 13 March 2007). http://www.fda.gov/MedicalDevices /DeviceRegulationandGuidance/GuidanceDocuments/ucm071148.htm. Accessed 17 February 2015.
Downloaded from http://cvi.asm.org/ on May 27, 2015 by NANYANG TECHNOLOGICAL UNIVERSITY
608
cvi.asm.org
Clinical and Vaccine Immunology
May 2015 Volume 22 Number 5
Reply to “Better Method for Evaluating a New Laboratory Test for Syphilis” Laboratory of G. Enders & Colleagues, MZV and Institute of Virology, Infectious Diseases and Epidemiology e. V., Stuttgart, Germanya; Labor Schottdorf MZV GmbH, Augsburg, Germanyb; Central Institute for Blood Transfusion and Immunology, Innsbruck, Austriac; Thai Red Cross Society, Bangkok, Thailandd; Faculty of Medicine, Ege University, Smyrna, Turkeye; The Greater Romagna Area Hub Laboratory, Unit of Microbiology, Pievesestina, Cesena, Italyf
T
his article is in response to the correspondence from Zhang et al. (1) relating to the methodology used in the study by Enders et al. (2). The correspondence from Zhang et al. does not present a better method. Instead, it criticizes a well-known and widely used method for evaluating diagnostic tests. The argument proposed by Zhang et al. contains several flaws, which we feel are important to address. They assert that it is not appropriate to assess sensitivity based on samples positive for syphilis infection or to assess specificity based on samples negative for syphilis infection. However, the European Union common technical specifications and the FDA guidance for diagnostic accuracy agree that these parameters should be used to assess sensitivity and specificity (3, 4). Zhang et al. also suggest that sensitivity and specificity do not adequately describe the performance of an assay in clinical practice. While we agree that sensitivity and specificity are technical measures of diagnostic accuracy and cannot completely reflect assay performance in routine use, we maintain that best-in-class sensitivity and specificity data are prerequisites for any assay considered for clinical use. As Zhang et al. describe, the negative predictive value (NPV) and positive predictive value (PPV) of an assay are dependent on disease prevalence and cannot be evaluated in isolation to assess the quality of a diagnostic method. Sensitivity and specificity, as calculated by Enders et al., provide standard information that can be used to compare diagnostic tests regardless of the clinical setting. Assessment of a new assay typically compares its performance to that of the existing accepted gold standard diagnostic method. Currently, there is no such accepted gold standard diagnostic method for syphilis, and if the assessment of the Elecsys syphilis assay by Enders et al. contains possible bias, we believe this to be the most likely source. We agree that it is important to utilize the correct target population for the assay to minimize bias and to make the assessment of the assay as relevant as possible. We assessed the specificity of the assay in samples from a wide range of relevant populations (blood banks, prenatal screening, routine diagnostic requests, patients assessed at HIV clinics). Overestimation of specificity is therefore unlikely. It is more reasonable to suggest that the specificity of the comparator assays has been overestimated, when those assays were evaluated in samples from blood banks, where a majority of repeat donors can be assumed, resulting in a preselection bias favoring the routine assay due to donors with reactive results (including false reactive) being deferred from repeat donations. Regarding the sensitivity assessment, an enrichment strat-
May 2015 Volume 22 Number 5
egy was implemented in the study design. This was intended to overcome the rare occurrence of syphilis-positive samples in a study design that includes all patients from the target populations. The enrichment strategy also allowed for the inclusion of samples from various stages of syphilis infection, minimizing the potential bias caused by including only samples with easily detectable, severe infection. The absence of a diagnostic gold standard for syphilis infection means that it is possible that a small number of the precharacterized positive samples were falsely included and may represent false-positive samples. Underestimation of the sensitivity, as it compares with that determined for samples characterized using an imperfect existing standard, is therefore possible. It is correct that, for individual clinicians, the clinically relevant NPV and PPV require local prevalence data in specific patient cohorts. Although calculating these numbers in cohorts, given the knowledge about the individual prevalence, is not difficult, calculating prevalence and NPV or PPV across all the different cohorts, particularly the sensitivity-enriched cohorts, used in the study by Enders et al. and comparing the results to CDC statistics from the United States (as exemplified in the introduction of the article) is in our opinion very adventurous. Instead of extrapolating and supposing prevalence values, we accurately present sensitivity and specificity, calculated using the methods suggested by the FDA and European Union guidelines. If the prevalence of disease in each cohort must be calculated before the effectiveness of a new diagnostic assay is accepted, evaluation of new assays will become highly complex. REFERENCES 1. Zhang Q, Zhang Y-F, Chen F-Y, Liu L, Yang T-C, Niu J-J, Liu L-L. 2015. Better method for evaluating a new laboratory test for syphilis. Clin Vaccine Immunol 22:606. http://dx.doi.org/10.1128/CVI.00014-15. 2. Enders M, Hunjet A, Gleich M, Imdahl R, Mühlbacher A, Schennach H, Chaiwong K, Sakuldamrongpanich T, Turhan A, Sertöz R, Wolf E,
Citation Enders M, Gleich M, Mühlbacher A, Sakuldamrongpanich T, Turhan A, Sertöz R, Semprini S, Sambri V. 2015. Reply to “Better method for evaluating a new laboratory test for syphilis.” Clin Vaccine Immunol 22:607– 608. doi:10.1128/CVI.00109-15. Editor: T. S. Alexander Address correspondence to Vittorio Sambri, [email protected]. This is a response to a letter by Zhang et al. (doi:10.1128/CVI.00014-15). Copyright © 2015, American Society for Microbiology. All Rights Reserved. doi:10.1128/CVI.00109-15
Clinical and Vaccine Immunology
cvi.asm.org
607
Downloaded from http://cvi.asm.org/ on May 27, 2015 by NANYANG TECHNOLOGICAL UNIVERSITY
Martin Enders,a Michael Gleich,b Annelies Mühlbacher,c Tasanee Sakuldamrongpanich,d Ajda Turhan,e Rüchan Sertöz,e Simona Semprini,f Vittorio Sambrif
Author Reply
Mayer W, Tao C, Wang LL, Semprini S, Sambri V. 2015. Performance evaluation of the Elecsys syphilis assay for the detection of total antibodies to Treponema pallidum. Clin Vaccine Immunol 22:17–26. http://dx .doi.org/10.1128/CVI.00505-14. 3. Official Journal of the European Union. 2009. Commission decision of 27 November 2009 amending Decision 2002/364/EC on common technical specifications for in vitro diagnostic medical devices.
http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri⫽OJ:L:2009 :318:0025:0040:EN:PDF. Accessed 17 February 2015. 4. U.S. Food and Drug Administration. Statistical guidance on reporting results from studies evaluating diagnostic tests (guidance for industry and FDA staff—version 13 March 2007). http://www.fda.gov/MedicalDevices /DeviceRegulationandGuidance/GuidanceDocuments/ucm071148.htm. Accessed 17 February 2015.
Downloaded from http://cvi.asm.org/ on May 27, 2015 by NANYANG TECHNOLOGICAL UNIVERSITY
608
cvi.asm.org
Clinical and Vaccine Immunology
May 2015 Volume 22 Number 5
