JNCI J Natl Cancer Inst (2015) 107(6): djv127 doi: 10.1093/jnci/djv127 First published online May 08, 2015 Correspondence
correspondence RE: Combined Associations of Genetic and Environmental Risk Factors: Implications for Prevention of Breast Cancer Stuart G. Baker Affiliation of author: Division of Cancer Prevention, National Cancer Institute, Bethesda, MD (SGB). Correspondence to: Stuart G. Baker, ScD, National Cancer Institute, 9609 Medical Center Dr, 5E638, MSC 9789, Bethesda, MD 20892–9789 (e-mail: [email protected]).
colleagues (1) opined that AUC is not a good metric because it “cannot capture the degree of stratification of absolute risk that a new risk factor can add to the model” and concluded that the change in RR distribution indicated “substantial improvement in risk stratification.” However, there is a fundamental limitation that applies to both of the aforementioned metrics; namely, they do not incorporate harms and benefits associated with clinical decision-making. Without incorporating harms and benefits, it is not possible to choose between different statistical methods for evaluating risk prediction (2). Approaches that account for costs and benefit include decision curves (3) and relative utility curves (2,4,5). These approaches involve a sensitivity analysis for the risk threshold, the risk at which persons are
Figure 1. Receiver operating characteristic (ROC) and relative utility curves for the constant odds ratio ROC curve. FPR = false-positive rate; OR = odds ratio; ROC = receiver operating characteristic curve; TPR = true-positive rate.
Received: December 18, 2014; Accepted: April 9, 2015 Published by Oxford University Press 2015. This work is written by (a) US Government employee(s) and is in the public domain in the US.
1 of 2
correspondence
In a recent commentary, Garcia-Closas and colleagues (1) discussed two metrics to evaluate the addition of genetic information to established risk factors in models to predict the risk of invasive breast cancer in asymptomatic women: 1) change in the area under the receiver operating characteristic curve (AUC) and 2) change in the population distribution of relative risk (RR) of invasive breast cancer, where RR is defined with respect to average risk. Comparing Model 1, with established risk factors, and Model 7, which added a polygenic risk score to established risk factors, Garcia-Closas and colleagues (1) found that the AUC increased from 0.618 to 0.703 and the percent in the population (percent in cases) with RR between 2 and 3 increased from 3.8% (8.8%) to 6.0% (14.4%) and the percent with RR greater than 3 increased from 1.0% (3.7%) to 3.8% (16.0%). Garcia-Closas and
2 of 2 | JNCI J Natl Cancer Inst, 2015, Vol. 107, No. 6
indifferent between treatment (here chemoprevention) and no treatment (no chemoprevention). A full decision curve or relative utility curve analysis requires data not available in the article. Nevertheless the following back-of-the-envelope calculation provides perspective. Suppose the receiver operating characteristic (ROC) curve is based on a constant odds ratio (OR) model, namely OR = {true-positive rate [TPR] (1 – false positive rate [FPR])} / {(1–TPR) FPR}, as shown in Figure 1. For this ROC curve, AUC = OR(OR – 1– log[OR]) / (OR– 1)2. Relative utility is the utility of risk prediction divided by the utility of perfect prediction evaluated at a risk threshold. As derived in the Appendix of Baker et al. (5), for this constant odds ratio ROC curve, the maximum relative utility (maxRU) is (√OR– 1) / (√OR+ 1), which, incidentally, equals Yule’s 1912 coefficient of colligation (6), although they have very different rationales. For an OR = 2, the AUC = 0.62 and maxRU = 0.17. For an OR = 4, the AUC = 0.72 and maxRU = 0.33. If the probability of breast cancer in the target population is 0.001, the test tradeoff (5) comparing models with OR = 2 vs OR = 4 equals 1/{(0.33 – 0.17) .001} = 6250. This test tradeoff implies that genetic information needs to be collected on at least 6250
persons for every correct prediction of invasive breast in order to increase the net benefit.
Notes The author has no conflict of interest to declare. The opinions presented should not be viewed as official opinions or positions of the National Cancer Institute.
References 1. Garcia-Closas M, Gunsoy NB, Chatterjee N. Combined associations of genetic and environmental risk factors: implications for prevention of breast cancer. J Natl Cancer Inst. 2014;106(11): dju305 doi:10.1093/jnci/dju305. 2. Baker SG, Schuit E, Steyerberg EW, et al. How to interpret a small increase in AUC with an additional risk prediction marker: Decision analysis comes through. Stat Med. 2014;33(2):3946–3959. Correction 33(2):3960. 3. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Dec Making. 2006;26(6):565–574. 4. Baker SG. Putting risk prediction in perspective: relative utility curves. J Natl Cancer Inst. 2009;101:1538–1542. 5. Baker SG, Cook NR, Vickers A, Kramer BS. Using relative utility curves to evaluate risk prediction. J Roy Stat Soc Ser A. 2009;172:729–748. 6. Yule GU. On the methods of measuring association between two attributes. J Roy Stat Soc. 1912;75(6):579–652.
correspondence
correspondence RE: Combined Associations of Genetic and Environmental Risk Factors: Implications for Prevention of Breast Cancer Stuart G. Baker Affiliation of author: Division of Cancer Prevention, National Cancer Institute, Bethesda, MD (SGB). Correspondence to: Stuart G. Baker, ScD, National Cancer Institute, 9609 Medical Center Dr, 5E638, MSC 9789, Bethesda, MD 20892–9789 (e-mail: [email protected]).
colleagues (1) opined that AUC is not a good metric because it “cannot capture the degree of stratification of absolute risk that a new risk factor can add to the model” and concluded that the change in RR distribution indicated “substantial improvement in risk stratification.” However, there is a fundamental limitation that applies to both of the aforementioned metrics; namely, they do not incorporate harms and benefits associated with clinical decision-making. Without incorporating harms and benefits, it is not possible to choose between different statistical methods for evaluating risk prediction (2). Approaches that account for costs and benefit include decision curves (3) and relative utility curves (2,4,5). These approaches involve a sensitivity analysis for the risk threshold, the risk at which persons are
Figure 1. Receiver operating characteristic (ROC) and relative utility curves for the constant odds ratio ROC curve. FPR = false-positive rate; OR = odds ratio; ROC = receiver operating characteristic curve; TPR = true-positive rate.
Received: December 18, 2014; Accepted: April 9, 2015 Published by Oxford University Press 2015. This work is written by (a) US Government employee(s) and is in the public domain in the US.
1 of 2
correspondence
In a recent commentary, Garcia-Closas and colleagues (1) discussed two metrics to evaluate the addition of genetic information to established risk factors in models to predict the risk of invasive breast cancer in asymptomatic women: 1) change in the area under the receiver operating characteristic curve (AUC) and 2) change in the population distribution of relative risk (RR) of invasive breast cancer, where RR is defined with respect to average risk. Comparing Model 1, with established risk factors, and Model 7, which added a polygenic risk score to established risk factors, Garcia-Closas and colleagues (1) found that the AUC increased from 0.618 to 0.703 and the percent in the population (percent in cases) with RR between 2 and 3 increased from 3.8% (8.8%) to 6.0% (14.4%) and the percent with RR greater than 3 increased from 1.0% (3.7%) to 3.8% (16.0%). Garcia-Closas and
2 of 2 | JNCI J Natl Cancer Inst, 2015, Vol. 107, No. 6
indifferent between treatment (here chemoprevention) and no treatment (no chemoprevention). A full decision curve or relative utility curve analysis requires data not available in the article. Nevertheless the following back-of-the-envelope calculation provides perspective. Suppose the receiver operating characteristic (ROC) curve is based on a constant odds ratio (OR) model, namely OR = {true-positive rate [TPR] (1 – false positive rate [FPR])} / {(1–TPR) FPR}, as shown in Figure 1. For this ROC curve, AUC = OR(OR – 1– log[OR]) / (OR– 1)2. Relative utility is the utility of risk prediction divided by the utility of perfect prediction evaluated at a risk threshold. As derived in the Appendix of Baker et al. (5), for this constant odds ratio ROC curve, the maximum relative utility (maxRU) is (√OR– 1) / (√OR+ 1), which, incidentally, equals Yule’s 1912 coefficient of colligation (6), although they have very different rationales. For an OR = 2, the AUC = 0.62 and maxRU = 0.17. For an OR = 4, the AUC = 0.72 and maxRU = 0.33. If the probability of breast cancer in the target population is 0.001, the test tradeoff (5) comparing models with OR = 2 vs OR = 4 equals 1/{(0.33 – 0.17) .001} = 6250. This test tradeoff implies that genetic information needs to be collected on at least 6250
persons for every correct prediction of invasive breast in order to increase the net benefit.
Notes The author has no conflict of interest to declare. The opinions presented should not be viewed as official opinions or positions of the National Cancer Institute.
References 1. Garcia-Closas M, Gunsoy NB, Chatterjee N. Combined associations of genetic and environmental risk factors: implications for prevention of breast cancer. J Natl Cancer Inst. 2014;106(11): dju305 doi:10.1093/jnci/dju305. 2. Baker SG, Schuit E, Steyerberg EW, et al. How to interpret a small increase in AUC with an additional risk prediction marker: Decision analysis comes through. Stat Med. 2014;33(2):3946–3959. Correction 33(2):3960. 3. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Dec Making. 2006;26(6):565–574. 4. Baker SG. Putting risk prediction in perspective: relative utility curves. J Natl Cancer Inst. 2009;101:1538–1542. 5. Baker SG, Cook NR, Vickers A, Kramer BS. Using relative utility curves to evaluate risk prediction. J Roy Stat Soc Ser A. 2009;172:729–748. 6. Yule GU. On the methods of measuring association between two attributes. J Roy Stat Soc. 1912;75(6):579–652.
correspondence
