LETTER TO THE EDITOR
Better Method for Evaluating a New Laboratory Test for Syphilis Qiao Zhang,a Ya-Feng Zhang,a Fu-Yi Chen,d Long Liu,e Tian-Ci Yang,a,b,c Jian-Jun Niu,a Li-Li Liua,c Zhongshan Hospital, Medical College of Xiamen University, Xiamen, Chinaa; Shenzhen Research Institute of Xiamen University, Shenzhen, Chinab; Xiamen Zhongshan Hospital, Fujian University of Traditional Chinese Medicine, Xiamen, Chinac; Department of Physiology and Neurobiology, University of Connecticut, Storrs, Connecticut, USAd; Department of Chemistry and Biology, College of Science, National University of Defense and Technology, Changsha, Chinae
ecently, we read the article by Enders et al. in Clinical and Vaccine Immunology with utmost interest (1). For that article, the authors evaluated the treponemal Elecsys syphilis assay by using 8,063 syphilis-positive samples to assess its sensitivity and 928 syphilis-negative samples to assess its specificity. The investigators calculated an overall sensitivity of 99.57% to 100% among the patients and an overall specificity of 99.88% among the controls for the Elecsys syphilis assay. Therefore, it was concluded that the Elecsys syphilis assay was both highly sensitive and highly specific, allowing it to play a major role in the diagnosis of syphilis. Do these data suffice to establish clinical usefulness? The answer is that they probably do not. With the development of clinical medicine and diagnostics, increasing numbers of diagnostic options are becoming available. However, methods for effectively evaluating diagnostic procedures and defining appropriate parameters to evaluate a new diagnostic test are becoming important issues. The article by Enders et al. (1) also faces the same problem. Currently, more and more researchers are using the Yerushalmy model (2), which mainly consists of a 2-by-2 table used to evaluate a new diagnostic assay (3–5). The analytic methods adopted by Enders et al. (1) are also dependent on the Yerushalmy model. However, not every researcher really understands the Yerushalmy model, as is the case for Enders et al. In the article by Enders et al., the investigation suffers from certain flaws, which are important aspects that must be considered. First, the researchers should not have directly and separately chosen syphilis-positive samples to calculate sensitivity and blood from subjects without syphilis to calculate specificity. As a result of their choices, both values were likely grossly overestimated relative to practical situations, in which diseased and healthy subjects cannot be clinically distinguished before testing (6). In addition, known positive and negative results could have created spectrum bias in the inspector, especially when the machine-read test results were indeterminate. Second, the test population for the new diagnostic test should have been relevant to practical situations. In other words, the new diagnostic test should have been performed in a population that has a disease prevalence similar to that observed in clinical practice (7). Other important considerations are as follows. Given a patient with a positive test result, what is the likelihood that the target disease is present? Given a patient with a negative test result, what is the likelihood that the target disease is absent? Thus, the positive predictive value (PPV) and negative predictive value (NPV) in the 2-by-2 table of the Yerushalmy model are critical in the assessment of clinical practice. In fact, the PPV and NPV depend on the prevalence of the target disease (4, 8). Instead, although the sensitivity and specificity remain the same, the PPV and NPV can change according to the prevalence of the disease in the test population. In the article, Enders et al. stated that the prevalence of syphilis was 0.05 cases per 1,000 (0.05‰) in the main text (1). However, they used 8,063 syphilisnegative samples and 928 syphilis-positive samples to evaluate the
606
cvi.asm.org
new diagnostic treponemal Elecsys syphilis assay. In their research population, the assumed “prevalence” of syphilis was 10.32% (928/ 8,991), which was not consistent with the rate of 0.05‰ mentioned above. Therefore, the PPV and NPV calculated from such a high rate of syphilis cannot represent the true PPV and NPV of the treponemal Elecsys syphilis assay in actual clinical practice. The purpose of this correspondence is to stimulate meaningful discussions on how to effectively evaluate new diagnostic procedures and to illuminate the importance of appropriate evaluation procedures. We hope that readers will benefit from this correspondence, and we welcome future discussions. REFERENCES 1. Enders M, Hunjet A, Gleich M, Imdahl R, Mühlbacher A, Schennach H, Chaiwong K, Sakuldamrongpanich T, Turhan A, Sertöz R, Wolf E, Mayer W, Tao C, Wang LL, Semprini S, Sambri V. 2015. Performance evaluation of the Elecsys syphilis assay for the detection of total antibodies to Treponema pallidum. Clin Vaccine Immunol 22:17–26. http://dx.doi.org /10.1128/CVI.00505-14. 2. Yerushalmy J. 1947. Statistical problems in assessing methods of medical diagnosis, with special reference to X-ray techniques. Public Health Rep 62:1432–1449. http://dx.doi.org/10.2307/4586294. 3. Chalmers RM, Campbell BM, Crouch N, Charlett A, Davies AP. 2011. Comparison of diagnostic sensitivity and specificity of seven Cryptosporidium assays used in the UK. J Med Microbiol 60:1598 –1604. http://dx.doi .org/10.1099/jmm.0.034181-0. 4. Tong M-L, Lin L-R, Liu L-L, Zhang H-L, Huang S-J, Chen Y-Y, Guo X-J, Xi Y, Liu L, Chen F-Y. 2014. Analysis of 3 algorithms for syphilis serodiagnosis and implications for clinical management. Clin Infect Dis 58:1116 – 1124. http://dx.doi.org/10.1093/cid/ciu087. 5. Yin Y-P, Chen X-S, Wei W-H, Gong K-L, Cao W-L, Yong G, Feng L, Huang S-J, Wang D-M, Han Y. 2013. A dual point-of-care test shows good performance in simultaneously detecting nontreponemal and treponemal antibodies in patients with syphilis: a multisite evaluation study in China. Clin Infect Dis 56:659 – 665. http://dx.doi.org/10.1093/cid/cis928. 6. Castro R, Lopes Â, da Luz Martins Pereira F. 2014. Evaluation of an immunochromatographic point-of-care test for the simultaneous detection of nontreponemal and treponemal antibodies in patients with syphilis. Sex Transm Dis 41:467– 469. http://dx.doi.org/10.1097/OLQ.0000000000000161. 7. Sheps SB, Schechter MT. 1984. The assessment of diagnostic tests. A survey of current medical research. JAMA 252:2418 –2422. 8. Maitournam A, Simon R. 2005. On the efficiency of targeted clinical trials. Stat Med 24:329 –339. http://dx.doi.org/10.1002/sim.1975.
Citation Zhang Q, Zhang Y-F, Chen F-Y, Liu L, Yang T-C, Niu J-J, Liu L-L. 2015. Better method for evaluating a new laboratory test for syphilis. Clin Vaccine Immunol 22:606. doi:10.1128/CVI.00014-15. Editor: T. S. Alexander Address correspondence to Jian-Jun Niu, [email protected], or Li-Li Liu, [email protected]. Q.Z. and Y.-F.Z. contributed equally to this article. For the author reply, see doi:10.1128/CVI.00109-15. Copyright © 2015, American Society for Microbiology. All Rights Reserved. doi:10.1128/CVI.00014-15
Clinical and Vaccine Immunology
May 2015 Volume 22 Number 5
Downloaded from http://cvi.asm.org/ on May 27, 2015 by NYU MEDICAL CENTER LIBRARY
R
Better Method for Evaluating a New Laboratory Test for Syphilis Qiao Zhang,a Ya-Feng Zhang,a Fu-Yi Chen,d Long Liu,e Tian-Ci Yang,a,b,c Jian-Jun Niu,a Li-Li Liua,c Zhongshan Hospital, Medical College of Xiamen University, Xiamen, Chinaa; Shenzhen Research Institute of Xiamen University, Shenzhen, Chinab; Xiamen Zhongshan Hospital, Fujian University of Traditional Chinese Medicine, Xiamen, Chinac; Department of Physiology and Neurobiology, University of Connecticut, Storrs, Connecticut, USAd; Department of Chemistry and Biology, College of Science, National University of Defense and Technology, Changsha, Chinae
ecently, we read the article by Enders et al. in Clinical and Vaccine Immunology with utmost interest (1). For that article, the authors evaluated the treponemal Elecsys syphilis assay by using 8,063 syphilis-positive samples to assess its sensitivity and 928 syphilis-negative samples to assess its specificity. The investigators calculated an overall sensitivity of 99.57% to 100% among the patients and an overall specificity of 99.88% among the controls for the Elecsys syphilis assay. Therefore, it was concluded that the Elecsys syphilis assay was both highly sensitive and highly specific, allowing it to play a major role in the diagnosis of syphilis. Do these data suffice to establish clinical usefulness? The answer is that they probably do not. With the development of clinical medicine and diagnostics, increasing numbers of diagnostic options are becoming available. However, methods for effectively evaluating diagnostic procedures and defining appropriate parameters to evaluate a new diagnostic test are becoming important issues. The article by Enders et al. (1) also faces the same problem. Currently, more and more researchers are using the Yerushalmy model (2), which mainly consists of a 2-by-2 table used to evaluate a new diagnostic assay (3–5). The analytic methods adopted by Enders et al. (1) are also dependent on the Yerushalmy model. However, not every researcher really understands the Yerushalmy model, as is the case for Enders et al. In the article by Enders et al., the investigation suffers from certain flaws, which are important aspects that must be considered. First, the researchers should not have directly and separately chosen syphilis-positive samples to calculate sensitivity and blood from subjects without syphilis to calculate specificity. As a result of their choices, both values were likely grossly overestimated relative to practical situations, in which diseased and healthy subjects cannot be clinically distinguished before testing (6). In addition, known positive and negative results could have created spectrum bias in the inspector, especially when the machine-read test results were indeterminate. Second, the test population for the new diagnostic test should have been relevant to practical situations. In other words, the new diagnostic test should have been performed in a population that has a disease prevalence similar to that observed in clinical practice (7). Other important considerations are as follows. Given a patient with a positive test result, what is the likelihood that the target disease is present? Given a patient with a negative test result, what is the likelihood that the target disease is absent? Thus, the positive predictive value (PPV) and negative predictive value (NPV) in the 2-by-2 table of the Yerushalmy model are critical in the assessment of clinical practice. In fact, the PPV and NPV depend on the prevalence of the target disease (4, 8). Instead, although the sensitivity and specificity remain the same, the PPV and NPV can change according to the prevalence of the disease in the test population. In the article, Enders et al. stated that the prevalence of syphilis was 0.05 cases per 1,000 (0.05‰) in the main text (1). However, they used 8,063 syphilisnegative samples and 928 syphilis-positive samples to evaluate the
606
cvi.asm.org
new diagnostic treponemal Elecsys syphilis assay. In their research population, the assumed “prevalence” of syphilis was 10.32% (928/ 8,991), which was not consistent with the rate of 0.05‰ mentioned above. Therefore, the PPV and NPV calculated from such a high rate of syphilis cannot represent the true PPV and NPV of the treponemal Elecsys syphilis assay in actual clinical practice. The purpose of this correspondence is to stimulate meaningful discussions on how to effectively evaluate new diagnostic procedures and to illuminate the importance of appropriate evaluation procedures. We hope that readers will benefit from this correspondence, and we welcome future discussions. REFERENCES 1. Enders M, Hunjet A, Gleich M, Imdahl R, Mühlbacher A, Schennach H, Chaiwong K, Sakuldamrongpanich T, Turhan A, Sertöz R, Wolf E, Mayer W, Tao C, Wang LL, Semprini S, Sambri V. 2015. Performance evaluation of the Elecsys syphilis assay for the detection of total antibodies to Treponema pallidum. Clin Vaccine Immunol 22:17–26. http://dx.doi.org /10.1128/CVI.00505-14. 2. Yerushalmy J. 1947. Statistical problems in assessing methods of medical diagnosis, with special reference to X-ray techniques. Public Health Rep 62:1432–1449. http://dx.doi.org/10.2307/4586294. 3. Chalmers RM, Campbell BM, Crouch N, Charlett A, Davies AP. 2011. Comparison of diagnostic sensitivity and specificity of seven Cryptosporidium assays used in the UK. J Med Microbiol 60:1598 –1604. http://dx.doi .org/10.1099/jmm.0.034181-0. 4. Tong M-L, Lin L-R, Liu L-L, Zhang H-L, Huang S-J, Chen Y-Y, Guo X-J, Xi Y, Liu L, Chen F-Y. 2014. Analysis of 3 algorithms for syphilis serodiagnosis and implications for clinical management. Clin Infect Dis 58:1116 – 1124. http://dx.doi.org/10.1093/cid/ciu087. 5. Yin Y-P, Chen X-S, Wei W-H, Gong K-L, Cao W-L, Yong G, Feng L, Huang S-J, Wang D-M, Han Y. 2013. A dual point-of-care test shows good performance in simultaneously detecting nontreponemal and treponemal antibodies in patients with syphilis: a multisite evaluation study in China. Clin Infect Dis 56:659 – 665. http://dx.doi.org/10.1093/cid/cis928. 6. Castro R, Lopes Â, da Luz Martins Pereira F. 2014. Evaluation of an immunochromatographic point-of-care test for the simultaneous detection of nontreponemal and treponemal antibodies in patients with syphilis. Sex Transm Dis 41:467– 469. http://dx.doi.org/10.1097/OLQ.0000000000000161. 7. Sheps SB, Schechter MT. 1984. The assessment of diagnostic tests. A survey of current medical research. JAMA 252:2418 –2422. 8. Maitournam A, Simon R. 2005. On the efficiency of targeted clinical trials. Stat Med 24:329 –339. http://dx.doi.org/10.1002/sim.1975.
Citation Zhang Q, Zhang Y-F, Chen F-Y, Liu L, Yang T-C, Niu J-J, Liu L-L. 2015. Better method for evaluating a new laboratory test for syphilis. Clin Vaccine Immunol 22:606. doi:10.1128/CVI.00014-15. Editor: T. S. Alexander Address correspondence to Jian-Jun Niu, [email protected], or Li-Li Liu, [email protected]. Q.Z. and Y.-F.Z. contributed equally to this article. For the author reply, see doi:10.1128/CVI.00109-15. Copyright © 2015, American Society for Microbiology. All Rights Reserved. doi:10.1128/CVI.00014-15
Clinical and Vaccine Immunology
May 2015 Volume 22 Number 5
Downloaded from http://cvi.asm.org/ on May 27, 2015 by NYU MEDICAL CENTER LIBRARY
R
