VOLUME 32 䡠 NUMBER 32 䡠 NOVEMBER 10 2014
JOURNAL OF CLINICAL ONCOLOGY
TO THE EDITOR: The study by Daniels et al1 focused on women with high-grade serous ovarian cancer and showed that carrier probabilities provided by risk prediction model BRCAPRO2 are too low among low-risk patients. This important observation agrees with earlier reports, including probands with both ovarian and breast cancers,2 in which the ratio of observed to expected cases (O/E) in low-risk groups was substantially greater than one. As genetic testing becomes more affordable and appropriately broadens to segments of the population who are at lower prior risk, this limitation deserves serious consideration. We are constantly working on improvements to make BRCAPRO more accurate and clinically helpful. We note with interest the suggestion to incorporate ovarian cancer histology, and we will consider it for future versions. Although Daniels et al1 report on the version of BRCAPRO included in BayesMendel 2-0.5 (October 2010), the latest version is 2-0.9 (March 2014). An array of improvements has been made since 2010,3 including updated penetrance estimates for contralateral breast cancer, more flexible incorporation of ethnicity, consideration of mastectomy in the proband and relatives, updated sensitivity parameters for BRCA1/2 testing, updated marker parameters, inclusion of HER2 status, and identical twins. It would be very interesting to evaluate the impact of the last 4 years of improvements on the calibration issues reported by Daniels et al.1 An additional issue deserving close scrutiny is the possibility that underestimation of risk in the low quintiles may be driven in large part by misreporting of family history. Family history collected by Daniels et al1 was obtained by genetic counselors, and although this is not fully clarified in the paper, may be self-reported by the proband, rather than validated through medical records, cancer registries, pathology reports, or death certificates. Self-reporting is the standard in clinical environments, and we consider it an appropriate approach for this type of model validation. At the same time, various studies evaluated misreporting of family history comparing self-reported with validated histories and showed that sensitivity and specificity for reported disease status can be serious. For example, in first-degree relatives, sensitivity estimates for breast cancer vary from 65% to 95% and for ovarian cancer from 67% to 84%. Sensitivity decreases further with the degree of the relative. Specificity estimates are approximately 98% to 99%.4-6 The effects of misreported family history on Mendelian risk prediction models, and BRCAPRO specifically, have been examined by Katki,7 who considered both underreporting of disease status and rounding of age, and showed that misreporting of family history, especially in disease status, leads to inaccurate calibration. To better understand the results reported by Daniels et al,1 we performed new analyses to evaluate whether misreporting could account for a significant portion of the observed miscalibration. We mimic the data collected by Daniels et al using data 3682
© 2014 by American Society of Clinical Oncology
Ratio of Observed to Expected Cases, in Log Scale
Misreported Family Histories and Underestimation of Risk
C O R R E S P O N D E N C E
4
Unadjusted Adjusted
3
2
1
0
-1
-2 0 to < 1
1 to < 3
3 to < 10
10 to < 40
≥ 40
Quintiles of BRCAPRO Probabilities (%) Fig 1. Log of observed over expected (O/E) ratios and 95% CIs for being a BRCA carrier for the subset of Cancer Genetics Network families (n ⫽ 157 families) with ovarian cancer. Results are stratified by BRCAPRO risk quintiles. In the last quintile, the adjusted O/E almost completely superimposes the unadjusted O/E.
from the Cancer Genetics Network Model Validation Study (described in detail elsewhere8). We focus on the 157 families of probands affected with ovarian cancer, ran BRCAPRO, and calculate the O/E ratio for each of the risk quintiles defined by Daniels et al (shown in gold in Fig 1). The overall O/E ratio is 1.02. Although we do not have any families in the first quintile, in the second quintile (1% to ⬍ 3% BRCAPRO probability) the O/E ratio is 16, similar to that observed by Daniels et al in the low-risk quintile. We then used a novel technique developed by Braun et al9 to adjust for measurement error. The measurement error adjustment involves averaging across all possible combinations of true disease status for relatives, each time weighing by the positive and negative predictive value of the reported history. We implemented it using estimates from Ziogas and Anton-Culver.6 This calculation, shown in blue in Figure 1, reduces the ratio in the second quintile from 16 to 5, whereas the overall O/E ratio remains 1.01. In conclusion, misreporting of family history is likely to play an important role in model calibration for low-risk probands. Using verified information in genetic counseling would likely lead to both more accuracy and better calibration of predictions for these women. However, verified information is impractical in many clinical settings. Therefore future versions of BRCAPRO will use the approach described by Braun et al to ameliorate calibration. In the interim, the results from Daniels et al1 can provide informal guidance for using lower BRCAPRO thresholds in settings where collection of verified information is impractical. Journal of Clinical Oncology, Vol 32, No 32 (November 10), 2014: pp 3682-3683
Information downloaded from jco.ascopubs.org and provided by at Charité - Med. Bibliothek on July 8, 2015 from Copyright © 2014 American Society of Clinical Oncology. All rights reserved. 130.133.8.114
Correspondence
Danielle Braun Harvard School of Public Health; Dana-Farber Cancer Institute, Boston, MA
Malka Gorfine Technion-Israel Institute of Technology, Haifa, Israel
Giovanni Parmigiani Harvard School of Public Health; Dana-Farber Cancer Institute, Boston, MA
AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST
Disclosures provided by the authors are available with this article at www.jco.org. REFERENCES 1. Daniels MS, Babb SA, King RH, et al: Underestimation of risk of a BRCA1 or BRCA2 mutation in women with high-grade serous ovarian cancer by BRCAPRO: A multi-institution study. J Clin Oncol 32:1249-1255, 2014 2. Berry DA, Iversen ES Jr, Gudbjartsson DF, et al: BRCAPRO validation, sensitivity of genetic testing of BRCA1/BRCA2, and prevalence of other breast cancer susceptibility genes. J Clin Oncol 20:2701-2712, 2002
3. Biswas S, Tankhiwale N, Blackford A, et al: Assessing the added value of breast tumor markers in genetic risk prediction model BRCAPRO. Breast Cancer Res Treat 133:347-355, 2012 4. Mai PL, Garceau AO, Graubard BI, et al: Confirmation of family cancer history reported in a population-based survey. J Natl Cancer Inst 103:788-797, 2011 5. Verkooijen HM, Fioretta G, Chappuis PO, et al: Set-up of a population-based familial breast cancer registry in Geneva, Switzerland: Validation of first results. Ann Oncol 15:350-353, 2004 6. Ziogas A, Anton-Culver H: Validation of family history data in cancer family registries. Am J Prev Med 24:190-198, 2003 7. Katki HA: Effect of misreported family history on Mendelian mutation prediction models. Biometrics 62:478-487, 2006 8. Parmigiani G, Chen S, Iversen ES Jr, et al: Validity of models for predicting BRCA1 and BRCA2 mutations. Ann Intern Med 147:441-450, 2007 9. Braun D, Gorfine M, Katki H, et al: Extending Mendelian risk prediction models to handle misreported family history. Boston, MA, Harvard University Biostatistics Working Paper Series, 2014
DOI: 10.1200/JCO.2014.57.4848; published online ahead of print at www.jco.org on September 8, 2014
■ ■ ■
www.jco.org
© 2014 by American Society of Clinical Oncology
Information downloaded from jco.ascopubs.org and provided by at Charité - Med. Bibliothek on July 8, 2015 from Copyright © 2014 American Society of Clinical Oncology. All rights reserved. 130.133.8.114
3683
Correspondence
AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST
Misreported Family Histories and Underestimation of Risk The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated. Relationships are self-held unless noted. I ⫽ Immediate Family Member, Inst ⫽ My Institution. For a detailed description of the disclosure categories, or for more information about ASCO’s conflict of interest policy, please refer to the Author Disclosure Declaration and the Disclosures of Potential Conflicts of Interest section in Information for Contributors. Danielle Braun No relationship to disclose
Patents, Royalties, Other Intellectual Property: Web risk service for BayesMendel Calculations
Malka Gorfine No relationship to disclose
ACKNOWLEDGMENT
Giovanni Parmigiani Stock or Other Ownership: Counsyl Consulting or Advisory Role: Counsyl
© 2014 by American Society of Clinical Oncology
Supported by funding from the National Cancer Institute at the National Institutes of Health (Grants No. 5T32CA009337-32 to G.P. and No. 5P30CA006516-46 to E.B.) and from the Israel Science Foundation (Grant No. 2012898 to M.G.).
JOURNAL OF CLINICAL ONCOLOGY
Information downloaded from jco.ascopubs.org and provided by at Charité - Med. Bibliothek on July 8, 2015 from Copyright © 2014 American Society of Clinical Oncology. All rights reserved. 130.133.8.114
JOURNAL OF CLINICAL ONCOLOGY
TO THE EDITOR: The study by Daniels et al1 focused on women with high-grade serous ovarian cancer and showed that carrier probabilities provided by risk prediction model BRCAPRO2 are too low among low-risk patients. This important observation agrees with earlier reports, including probands with both ovarian and breast cancers,2 in which the ratio of observed to expected cases (O/E) in low-risk groups was substantially greater than one. As genetic testing becomes more affordable and appropriately broadens to segments of the population who are at lower prior risk, this limitation deserves serious consideration. We are constantly working on improvements to make BRCAPRO more accurate and clinically helpful. We note with interest the suggestion to incorporate ovarian cancer histology, and we will consider it for future versions. Although Daniels et al1 report on the version of BRCAPRO included in BayesMendel 2-0.5 (October 2010), the latest version is 2-0.9 (March 2014). An array of improvements has been made since 2010,3 including updated penetrance estimates for contralateral breast cancer, more flexible incorporation of ethnicity, consideration of mastectomy in the proband and relatives, updated sensitivity parameters for BRCA1/2 testing, updated marker parameters, inclusion of HER2 status, and identical twins. It would be very interesting to evaluate the impact of the last 4 years of improvements on the calibration issues reported by Daniels et al.1 An additional issue deserving close scrutiny is the possibility that underestimation of risk in the low quintiles may be driven in large part by misreporting of family history. Family history collected by Daniels et al1 was obtained by genetic counselors, and although this is not fully clarified in the paper, may be self-reported by the proband, rather than validated through medical records, cancer registries, pathology reports, or death certificates. Self-reporting is the standard in clinical environments, and we consider it an appropriate approach for this type of model validation. At the same time, various studies evaluated misreporting of family history comparing self-reported with validated histories and showed that sensitivity and specificity for reported disease status can be serious. For example, in first-degree relatives, sensitivity estimates for breast cancer vary from 65% to 95% and for ovarian cancer from 67% to 84%. Sensitivity decreases further with the degree of the relative. Specificity estimates are approximately 98% to 99%.4-6 The effects of misreported family history on Mendelian risk prediction models, and BRCAPRO specifically, have been examined by Katki,7 who considered both underreporting of disease status and rounding of age, and showed that misreporting of family history, especially in disease status, leads to inaccurate calibration. To better understand the results reported by Daniels et al,1 we performed new analyses to evaluate whether misreporting could account for a significant portion of the observed miscalibration. We mimic the data collected by Daniels et al using data 3682
© 2014 by American Society of Clinical Oncology
Ratio of Observed to Expected Cases, in Log Scale
Misreported Family Histories and Underestimation of Risk
C O R R E S P O N D E N C E
4
Unadjusted Adjusted
3
2
1
0
-1
-2 0 to < 1
1 to < 3
3 to < 10
10 to < 40
≥ 40
Quintiles of BRCAPRO Probabilities (%) Fig 1. Log of observed over expected (O/E) ratios and 95% CIs for being a BRCA carrier for the subset of Cancer Genetics Network families (n ⫽ 157 families) with ovarian cancer. Results are stratified by BRCAPRO risk quintiles. In the last quintile, the adjusted O/E almost completely superimposes the unadjusted O/E.
from the Cancer Genetics Network Model Validation Study (described in detail elsewhere8). We focus on the 157 families of probands affected with ovarian cancer, ran BRCAPRO, and calculate the O/E ratio for each of the risk quintiles defined by Daniels et al (shown in gold in Fig 1). The overall O/E ratio is 1.02. Although we do not have any families in the first quintile, in the second quintile (1% to ⬍ 3% BRCAPRO probability) the O/E ratio is 16, similar to that observed by Daniels et al in the low-risk quintile. We then used a novel technique developed by Braun et al9 to adjust for measurement error. The measurement error adjustment involves averaging across all possible combinations of true disease status for relatives, each time weighing by the positive and negative predictive value of the reported history. We implemented it using estimates from Ziogas and Anton-Culver.6 This calculation, shown in blue in Figure 1, reduces the ratio in the second quintile from 16 to 5, whereas the overall O/E ratio remains 1.01. In conclusion, misreporting of family history is likely to play an important role in model calibration for low-risk probands. Using verified information in genetic counseling would likely lead to both more accuracy and better calibration of predictions for these women. However, verified information is impractical in many clinical settings. Therefore future versions of BRCAPRO will use the approach described by Braun et al to ameliorate calibration. In the interim, the results from Daniels et al1 can provide informal guidance for using lower BRCAPRO thresholds in settings where collection of verified information is impractical. Journal of Clinical Oncology, Vol 32, No 32 (November 10), 2014: pp 3682-3683
Information downloaded from jco.ascopubs.org and provided by at Charité - Med. Bibliothek on July 8, 2015 from Copyright © 2014 American Society of Clinical Oncology. All rights reserved. 130.133.8.114
Correspondence
Danielle Braun Harvard School of Public Health; Dana-Farber Cancer Institute, Boston, MA
Malka Gorfine Technion-Israel Institute of Technology, Haifa, Israel
Giovanni Parmigiani Harvard School of Public Health; Dana-Farber Cancer Institute, Boston, MA
AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST
Disclosures provided by the authors are available with this article at www.jco.org. REFERENCES 1. Daniels MS, Babb SA, King RH, et al: Underestimation of risk of a BRCA1 or BRCA2 mutation in women with high-grade serous ovarian cancer by BRCAPRO: A multi-institution study. J Clin Oncol 32:1249-1255, 2014 2. Berry DA, Iversen ES Jr, Gudbjartsson DF, et al: BRCAPRO validation, sensitivity of genetic testing of BRCA1/BRCA2, and prevalence of other breast cancer susceptibility genes. J Clin Oncol 20:2701-2712, 2002
3. Biswas S, Tankhiwale N, Blackford A, et al: Assessing the added value of breast tumor markers in genetic risk prediction model BRCAPRO. Breast Cancer Res Treat 133:347-355, 2012 4. Mai PL, Garceau AO, Graubard BI, et al: Confirmation of family cancer history reported in a population-based survey. J Natl Cancer Inst 103:788-797, 2011 5. Verkooijen HM, Fioretta G, Chappuis PO, et al: Set-up of a population-based familial breast cancer registry in Geneva, Switzerland: Validation of first results. Ann Oncol 15:350-353, 2004 6. Ziogas A, Anton-Culver H: Validation of family history data in cancer family registries. Am J Prev Med 24:190-198, 2003 7. Katki HA: Effect of misreported family history on Mendelian mutation prediction models. Biometrics 62:478-487, 2006 8. Parmigiani G, Chen S, Iversen ES Jr, et al: Validity of models for predicting BRCA1 and BRCA2 mutations. Ann Intern Med 147:441-450, 2007 9. Braun D, Gorfine M, Katki H, et al: Extending Mendelian risk prediction models to handle misreported family history. Boston, MA, Harvard University Biostatistics Working Paper Series, 2014
DOI: 10.1200/JCO.2014.57.4848; published online ahead of print at www.jco.org on September 8, 2014
■ ■ ■
www.jco.org
© 2014 by American Society of Clinical Oncology
Information downloaded from jco.ascopubs.org and provided by at Charité - Med. Bibliothek on July 8, 2015 from Copyright © 2014 American Society of Clinical Oncology. All rights reserved. 130.133.8.114
3683
Correspondence
AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST
Misreported Family Histories and Underestimation of Risk The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated. Relationships are self-held unless noted. I ⫽ Immediate Family Member, Inst ⫽ My Institution. For a detailed description of the disclosure categories, or for more information about ASCO’s conflict of interest policy, please refer to the Author Disclosure Declaration and the Disclosures of Potential Conflicts of Interest section in Information for Contributors. Danielle Braun No relationship to disclose
Patents, Royalties, Other Intellectual Property: Web risk service for BayesMendel Calculations
Malka Gorfine No relationship to disclose
ACKNOWLEDGMENT
Giovanni Parmigiani Stock or Other Ownership: Counsyl Consulting or Advisory Role: Counsyl
© 2014 by American Society of Clinical Oncology
Supported by funding from the National Cancer Institute at the National Institutes of Health (Grants No. 5T32CA009337-32 to G.P. and No. 5P30CA006516-46 to E.B.) and from the Israel Science Foundation (Grant No. 2012898 to M.G.).
JOURNAL OF CLINICAL ONCOLOGY
Information downloaded from jco.ascopubs.org and provided by at Charité - Med. Bibliothek on July 8, 2015 from Copyright © 2014 American Society of Clinical Oncology. All rights reserved. 130.133.8.114
