Journal of Perinatology (2014) 34, 802 © 2014 Nature America, Inc. All rights reserved 0743-8346/14 www.nature.com/jp
LETTER TO THE EDITOR
Predictive models for severe intraventricular hemorrhage in preterm infants Journal of Perinatology (2014) 34, 802; doi:10.1038/jp.2014.152 It was with great interest that we read the recent article by Luque et al.1 on a prediction model for severe intraventricular hemorrhage (SIVH) and noticed that it is quite consistent with our similar publication on a predictive model for SIVH.2 Both models demonstrate the utility of routinely-collected clinical data in predicting SIVH among vulnerable preterm infants and show good discrimination between infants with SIVH and those without (0.79 for Luque et al.1 and 0.84 for Singh et al.2). Further, the Singh model showed good calibration as indicated by a comparison of the observed and expected risk. Calibration of the Luque model appears more marginal, as indicated by the discrepancy between the observed SIVH risk and the SIVH risk quintile in Figure 2. In both the models, the benefits of indomethacin for decreasing SIVH risk are apparent, more so as the risk decile increases. The differences between the models can be explained by differences in sample definition, as well as differences in variable selection. Singh et al.2 enrolled all infants born less than 32 weeks (and ⩾ 500 g). This sample definition yielded an SIVH incidence of about 5%. Luque et al.1 restricted their sample to more vulnerable infants, that is, those with birth weights between 500 and 1249 g. This yielded an SIVH incidence of about 15%. We wish to address two interpretations by the authors. One concerns the investigators’ reference to the cut point of 17%, which they indicate maximizes sensitivity and specificity. There are several issues regarding this cut point and their interpretation that require clarification. First, maximizing sensitivity and specificity is a model-specific determination that minimizes classification errors (that is, false positives (FP) and false negatives (FN)) more or less equally. For clinical practice, it may not represent the optimal cut point. Maximizing sensitivity and specificity does not account for the relative cost of the classification errors, nor the fact that indomethacin use may have detrimental side effects, particularly on this vulnerable population. If the cost of FN errors (for example, clinically, financially or as perceived by parents) is considered greater than FP errors, then one may wish to select a cut point that enhances sensitivity (that is, reduce FN) and tolerate a greater FP fraction. If the reverse perception is true, then the appropriate cut point would favor specificity over sensitivity. Thus, the optimal cut point for clinical practice should be based on the relative costs of the errors, as well as the estimated prevalence of the condition.3 Second, the investigators state that ‘when the model is applied and a probability of 17% or greater is obtained, the chance of
developing a severe IVH is statistically significant, so it is considered as a positive test.’ (pg. 44). This statement reflects an inappropriate application and interpretation of the P-value. What is the null hypothesis for this test? Given the sample size of the study, there are likely to be many cut points that are ‘statistically significant’, including some that are clinically meaningless. Further, the interpretation may lead practitioners to several erroneous conclusions: (a) the model is only significant at 17%, (b) the only meaningful cut point is 17% or (c) that 17% is the optimal cut point for their clinical situation (see above). Indeed, it is unfortunate that the authors would suggest reducing their informationrich prediction model to a single cut point. Can a patient with a 45% estimate risk really be approached as a patient with a 17% estimated risk? The utility of the investigators work is the model — not a single cut point.4 The recent prediction models of Luque et al.1 and Singh et al.2 provide clinicians with valid and objective assessments of SIVH risk and will help guide them on therapeutic approaches. It is important to understand how they should be applied in clinical decision-making, as well as the strengths and limitations of each model. CONFLICT OF INTEREST The authors declare no conflict of interest.
R Singh1 and PF Visintainer2 Pediatric Residency Research, Division of Neonatology, Department of Pediatrics, Tufts University School of Medicine, Baystate Children’s Hospital, Springfield, MA, USA and 2 Department of Epidemiology, Baystate Medical Center, Tufts University School of Medicine, Springfield, MA, USA E-mail: [email protected] 1
REFERENCES 1 Luque MJ, Tapia JL, Villarroel L, Marshall G, Musante G, Carlo W et al. Neocosur Neonatal Network. A risk prediction model for severe intraventricular hemorrhage in very low birth weight infants and the effect of prophylactic indomethacin. J Perinatol 2014; 34(1): 43–48. 2 Singh R, Gorstein SV, Bednarek F, Chou JH, McGowan EC, Visintainer PF. A predictive model for SIVH risk in preterm infants and targeted indomethacin therapy for prevention. Sci Rep 2013; 3: 2539. 3 Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem 1993; 39(4): 561–577. 4 Sinclair JC. Weighing risks and benefits in treating the individual patient. Clin Perinatol 2003; 30: 251–268.
LETTER TO THE EDITOR
Predictive models for severe intraventricular hemorrhage in preterm infants Journal of Perinatology (2014) 34, 802; doi:10.1038/jp.2014.152 It was with great interest that we read the recent article by Luque et al.1 on a prediction model for severe intraventricular hemorrhage (SIVH) and noticed that it is quite consistent with our similar publication on a predictive model for SIVH.2 Both models demonstrate the utility of routinely-collected clinical data in predicting SIVH among vulnerable preterm infants and show good discrimination between infants with SIVH and those without (0.79 for Luque et al.1 and 0.84 for Singh et al.2). Further, the Singh model showed good calibration as indicated by a comparison of the observed and expected risk. Calibration of the Luque model appears more marginal, as indicated by the discrepancy between the observed SIVH risk and the SIVH risk quintile in Figure 2. In both the models, the benefits of indomethacin for decreasing SIVH risk are apparent, more so as the risk decile increases. The differences between the models can be explained by differences in sample definition, as well as differences in variable selection. Singh et al.2 enrolled all infants born less than 32 weeks (and ⩾ 500 g). This sample definition yielded an SIVH incidence of about 5%. Luque et al.1 restricted their sample to more vulnerable infants, that is, those with birth weights between 500 and 1249 g. This yielded an SIVH incidence of about 15%. We wish to address two interpretations by the authors. One concerns the investigators’ reference to the cut point of 17%, which they indicate maximizes sensitivity and specificity. There are several issues regarding this cut point and their interpretation that require clarification. First, maximizing sensitivity and specificity is a model-specific determination that minimizes classification errors (that is, false positives (FP) and false negatives (FN)) more or less equally. For clinical practice, it may not represent the optimal cut point. Maximizing sensitivity and specificity does not account for the relative cost of the classification errors, nor the fact that indomethacin use may have detrimental side effects, particularly on this vulnerable population. If the cost of FN errors (for example, clinically, financially or as perceived by parents) is considered greater than FP errors, then one may wish to select a cut point that enhances sensitivity (that is, reduce FN) and tolerate a greater FP fraction. If the reverse perception is true, then the appropriate cut point would favor specificity over sensitivity. Thus, the optimal cut point for clinical practice should be based on the relative costs of the errors, as well as the estimated prevalence of the condition.3 Second, the investigators state that ‘when the model is applied and a probability of 17% or greater is obtained, the chance of
developing a severe IVH is statistically significant, so it is considered as a positive test.’ (pg. 44). This statement reflects an inappropriate application and interpretation of the P-value. What is the null hypothesis for this test? Given the sample size of the study, there are likely to be many cut points that are ‘statistically significant’, including some that are clinically meaningless. Further, the interpretation may lead practitioners to several erroneous conclusions: (a) the model is only significant at 17%, (b) the only meaningful cut point is 17% or (c) that 17% is the optimal cut point for their clinical situation (see above). Indeed, it is unfortunate that the authors would suggest reducing their informationrich prediction model to a single cut point. Can a patient with a 45% estimate risk really be approached as a patient with a 17% estimated risk? The utility of the investigators work is the model — not a single cut point.4 The recent prediction models of Luque et al.1 and Singh et al.2 provide clinicians with valid and objective assessments of SIVH risk and will help guide them on therapeutic approaches. It is important to understand how they should be applied in clinical decision-making, as well as the strengths and limitations of each model. CONFLICT OF INTEREST The authors declare no conflict of interest.
R Singh1 and PF Visintainer2 Pediatric Residency Research, Division of Neonatology, Department of Pediatrics, Tufts University School of Medicine, Baystate Children’s Hospital, Springfield, MA, USA and 2 Department of Epidemiology, Baystate Medical Center, Tufts University School of Medicine, Springfield, MA, USA E-mail: [email protected] 1
REFERENCES 1 Luque MJ, Tapia JL, Villarroel L, Marshall G, Musante G, Carlo W et al. Neocosur Neonatal Network. A risk prediction model for severe intraventricular hemorrhage in very low birth weight infants and the effect of prophylactic indomethacin. J Perinatol 2014; 34(1): 43–48. 2 Singh R, Gorstein SV, Bednarek F, Chou JH, McGowan EC, Visintainer PF. A predictive model for SIVH risk in preterm infants and targeted indomethacin therapy for prevention. Sci Rep 2013; 3: 2539. 3 Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem 1993; 39(4): 561–577. 4 Sinclair JC. Weighing risks and benefits in treating the individual patient. Clin Perinatol 2003; 30: 251–268.
