Downloaded from http://heart.bmj.com/ on February 20, 2015 - Published by group.bmj.com
Editorial
Predicting the risks of pregnancy in congenital heart disease: the importance of external validation Gerhard-Paul Diller,1,2 Anselm Uebing2,3 With growing numbers of patients with congenital heart disease (CHD) surviving to adulthood, prepregnancy counselling is increasingly required for women with this heterogeneous condition.1 This represents a major challenge for clinicians, especially in female patients with medium or highcomplexity CHD. While individual risk assessment should be advocated based on the patient’s underlying condition, previous operations/interventions, history of complications and the outcome of previous pregnancies, it is desirable to have standardised tools for estimating the risk of serious complications during pregnancy. These tools have the advantage of providing a quantitative (numeric) risk estimate, thus potentially offering a better, more accurate basis for prepregnancy counselling to allow the woman an informed decision on whether or not she wants to embark on pregnancy. Additionally, such prediction models may eliminate some of the subjective elements of risk assessment, thus making recommendations between clinicians more reproducible. These potential advantages are achieved at the cost of some informational loss (ie, generalisation), ignoring specific aspects of the patient’s condition that are not implemented in the risk score employed. While the desire to quantify matters is understandable and is, indeed, deep-seated in our scientific culture as exemplified by Galileo Galilei’s demand to ‘[m]easure what is measurable, and make measurable what is not so’, one should not underestimate the problems introduced by risk prediction models in the setting of CHD. This is compounded
by the fact that humans (including experts) tend to overestimate the accuracy of their predictions,2 providing a false sense of correctness to themselves and patients. Therefore, such scoring tools do not obviate the need for expert assessment, a thorough understanding of the underlying pathophysiology and an individualised approach. Beyond these psychological and philosophical issues, technical aspects of risk scores have not been fully resolved, and this is well illustrated by the current study by Balci et al.3 The authors set out to identify the optimal risk prediction model for estimating the risk of pregnancy in women with CHD. To this end, they assessed the external validity of three commonly used risk scores. This included the ZAHARA,4 the CARPREG5 and the WHO classification systems.6 Based on a sample of 213 pregnancies in 203 women, the area under curve (AUC) was quantified for maternal and offspring risk. The WHO classification system was found to have the highest AUC for predicting maternal cardiovascular risk (0.77 (95% CI 0.67 to 0.87)), followed by the ZAHARA classification (AUC 0.71 (95% CI 0.59 to 0.83)) and the
CARPREG system (AUC 0.57 (95% CI 0.43 to 0.70)). All three systems performed poorly when attempting to predict complications in the offspring. The authors are to be commended for performing this much needed study. The results of this analysis should be useful for clinicians in search of an appropriate risk prediction model but, more importantly, they illustrate the pitfalls of an uncritical application of a published risk model to one’s own patients. In addition to providing data on the discriminatory ability of the three risk models, the authors provide some information on the quality of the calibration. The AUC mentioned above is a measure of how well a model discriminates between subjects who will or will not develop a particular complication (ie, it tests whether the model will assign a higher event probability to the patient with an event compared to the no-event subject).7 Calibration, in turn, measures how well the predicted event probabilities agree with the real, observed event rates. The authors chose to present calibration data in a bar plot, while conventionally a calibration plot is used to this end. Figure 1 shows the calibration data extracted from the manuscript (figure 2 in the paper by Balci et al) in a calibration plot. This plot exposes the poor agreement between predicted and actual outcomes for both the CARPREG and the ZAHARA model in patients with an actual risk of cardiovascular complications greater than approximately 20%. In fact, both models overestimate the risk of
1
Division of Adult Congenital and Valvular Heart Disease, Department of Cardiovascular Medicine, University Hospital Muenster, Muenster, Germany; 2 Imperial College of Science and Medicine, London, UK; 3NIHR Cardiovascular and Respiratory Biomedical Research Unit, Adult Congenital Heart Centre and Centre for Pulmonary Hypertension, Royal Brompton Hospital, London, UK Correspondence to Dr Gerhard-Paul Diller, Division of Adult Congenital and Valvular Heart Disease, Department of Cardiovascular Medicine University Hospital of Münster, Albert-Schweitzer-Str. 33, Münster 48149, Germany; [email protected]
Figure 1 Calibration plot according to the data provided in the publication by Balci et al. The plot illustrates the modest calibration, especially in high-risk patients, where the actual risk is significantly overestimated. The size of the dot corresponds to the number of patients included in each subgroup.
Diller G-P, et al. Heart September 2014 Vol 100 No 17
1311
Downloaded from http://heart.bmj.com/ on February 20, 2015 - Published by group.bmj.com
Editorial roughly guide the clinician when counselling women with CHD of childbearing age about their individual risk associated with pregnancy. The estimated risk obtained by any score must be adjusted to factors that are not included in the scoring systems. It also emphasises that it is an important part of prepregnancy counselling to make women aware of the limitations and inaccuracies of any risk assessment, and the considerable amount of uncertainty related to pregnancy with their condition. Contributors both authors contributed equally to the manuscript, wrote and critically reviewed the final product. Competing interests None.
Figure 2 The difference between apparent and true performance: while apparent performance measures how well the model fits the original (derivation) sample, true performance provides a measure on the performance of the model when applied to an external cohort. higher-risk patients by as much as twofold (eg, patients with a predicted risk of >70% according to the ZAHARA score had an actual event rate of 35%). This result is particularly noteworthy, as the internal validity of the CARPREG was previously reported to be acceptable.5 8 As a consequence, the paper by Baci et al shows that currently available pregnancy risk scores suffer, both, from an only modest discriminatory ability and a poor calibration, especially in higher-risk patients. This limits the generalisability of the risk models to clinical cohorts of CHD patients outside the centres involved in the derivation of the respective risk models. It should be emphasised that the number of women at risk in some of the subgroups were rather small, therefore reducing the accuracy of the predictions. Interestingly, the presence of a mechanical prosthesis was associated with a high risk for complications (6/11), and this is particularly noteworthy as this parameter is not universally included as part of risk prediction scores. Formally, before a risk prediction model should be applied routinely to a large population, one should ensure that the correct statistical model has been chosen, that the derivation sample is representative of the underlying population, that the model performs adequately in the derivation sample and, ultimately, the validity should be assessed in an external population. Especially, the last step is often avoided in CHD—but it is crucial. This is because even if the predication model reflects the underlying dataset quite
1312
well, it may be sensitive to idiosyncrasies of this derivation sample and may perform poorly in another sample. This fact is illustrated in figure 2, showing the difference between apparent and true performance. Clinicians are interested in the true performance of a model as they try to predict the outcome of patients not included in the initial (derivation) study. The difference between apparent and true performance is also illustrated in the current study by Balci et al showing that one of the risk models failed to statistically deviate significantly from a ‘random guess’ in their patient sample. The lack of external validation is not a specific problem of CHD studies: Perel et al have studied 102 prognostic models in traumatic brain injury and found that only 11% of these provided data on the external validity of the data.9 The importance of the current study goes beyond estimation of pregnancy risk. It illustrates the problems with riskprediction models in CHD in general. Given the heterogeneous nature of CHD and the limited number of patients with some conditions at single centres, clinicians will have to continue to rely on unvalidated prognostic models for the foreseeable future. Nevertheless, in light of theoretical considerations and the results of the current study by Balci et al, every attempt should be made to externally validate and, if required, update available risk models to the benefit of CHD patients. The study by Balci et al also stresses that all available risk scores can only
Provenance and peer review Commissioned; internally peer reviewed.
To cite Diller G-P, Uebing A. Heart 2014;100:1311– 1312.
▸ http://dx.doi.org/10.1136/heartjnl-2014-305597 Heart 2014;100:1311–1312. doi:10.1136/heartjnl-2014-306060
REFERENCES 1 2
3
4
5
6
7 8
9
Uebing A, Steer PJ, Yentis SM, et al. Pregnancy and congenital heart disease. BMJ 2006;332:401. Klayman J, Soll JB, Gonzalez-Vallejo C, et al. Overconfidence: it depends on how, what, and whom you ask. Organ Behav Hum Decis Process 1999;79:216–47. Balci A, Sollie-Szarynska KM, van der Bijl AGL, et al. Prospective validation and assessment of cardiovascular and offspring risk models for pregnant women with congenital heart disease. Heart 2014;100:1373–81. Drenthen W, Boersma E, Balci A, et al. Predictors of pregnancy complications in women with congenital heart disease. Eur Heart J 2010;31:2124–32. Siu SC, Sermer M, Colman JM, et al. Prospective multicenter study of pregnancy outcomes in women with heart disease. Circulation 2001;104:515–21. Regitz-Zagrosek V, Lundqvist CB, Borghi C, et al. ESC Guidelines on the management of cardiovascular diseases during pregnancy The Task Force on the Management of Cardiovascular Diseases during Pregnancy of the European Society of Cardiology (ESC). Eur Heart J 2011;32:3147–97. Steyerberg EW. Clinical prediction models. Springer, 2009. Khairy P, Ouyang DW, Fernandes SM, et al. Pregnancy outcomes in women with congenital heart disease. Circulation 2006;113:517–24. Perel P, Edwards P, Wentz R, et al. Systematic review of prognostic models in traumatic brain injury. BMC Med Inform Decis Mak 2006;6:38.
Diller G-P, et al. Heart September 2014 Vol 100 No 17
Downloaded from http://heart.bmj.com/ on February 20, 2015 - Published by group.bmj.com
Predicting the risks of pregnancy in congenital heart disease: the importance of external validation Gerhard-Paul Diller and Anselm Uebing Heart 2014 100: 1311-1312
doi: 10.1136/heartjnl-2014-306060 Updated information and services can be found at: http://heart.bmj.com/content/100/17/1311
These include:
References Email alerting service
Topic Collections
This article cites 8 articles, 6 of which you can access for free at: http://heart.bmj.com/content/100/17/1311#BIBL Receive free email alerts when new articles cite this article. Sign up in the box at the top right corner of the online article.
Articles on similar topics can be found in the following collections Drugs: cardiovascular system (8099) Hypertension (2745) Congenital heart disease (672)
Notes
To request permissions go to: http://group.bmj.com/group/rights-licensing/permissions To order reprints go to: http://journals.bmj.com/cgi/reprintform To subscribe to BMJ go to: http://group.bmj.com/subscribe/
Editorial
Predicting the risks of pregnancy in congenital heart disease: the importance of external validation Gerhard-Paul Diller,1,2 Anselm Uebing2,3 With growing numbers of patients with congenital heart disease (CHD) surviving to adulthood, prepregnancy counselling is increasingly required for women with this heterogeneous condition.1 This represents a major challenge for clinicians, especially in female patients with medium or highcomplexity CHD. While individual risk assessment should be advocated based on the patient’s underlying condition, previous operations/interventions, history of complications and the outcome of previous pregnancies, it is desirable to have standardised tools for estimating the risk of serious complications during pregnancy. These tools have the advantage of providing a quantitative (numeric) risk estimate, thus potentially offering a better, more accurate basis for prepregnancy counselling to allow the woman an informed decision on whether or not she wants to embark on pregnancy. Additionally, such prediction models may eliminate some of the subjective elements of risk assessment, thus making recommendations between clinicians more reproducible. These potential advantages are achieved at the cost of some informational loss (ie, generalisation), ignoring specific aspects of the patient’s condition that are not implemented in the risk score employed. While the desire to quantify matters is understandable and is, indeed, deep-seated in our scientific culture as exemplified by Galileo Galilei’s demand to ‘[m]easure what is measurable, and make measurable what is not so’, one should not underestimate the problems introduced by risk prediction models in the setting of CHD. This is compounded
by the fact that humans (including experts) tend to overestimate the accuracy of their predictions,2 providing a false sense of correctness to themselves and patients. Therefore, such scoring tools do not obviate the need for expert assessment, a thorough understanding of the underlying pathophysiology and an individualised approach. Beyond these psychological and philosophical issues, technical aspects of risk scores have not been fully resolved, and this is well illustrated by the current study by Balci et al.3 The authors set out to identify the optimal risk prediction model for estimating the risk of pregnancy in women with CHD. To this end, they assessed the external validity of three commonly used risk scores. This included the ZAHARA,4 the CARPREG5 and the WHO classification systems.6 Based on a sample of 213 pregnancies in 203 women, the area under curve (AUC) was quantified for maternal and offspring risk. The WHO classification system was found to have the highest AUC for predicting maternal cardiovascular risk (0.77 (95% CI 0.67 to 0.87)), followed by the ZAHARA classification (AUC 0.71 (95% CI 0.59 to 0.83)) and the
CARPREG system (AUC 0.57 (95% CI 0.43 to 0.70)). All three systems performed poorly when attempting to predict complications in the offspring. The authors are to be commended for performing this much needed study. The results of this analysis should be useful for clinicians in search of an appropriate risk prediction model but, more importantly, they illustrate the pitfalls of an uncritical application of a published risk model to one’s own patients. In addition to providing data on the discriminatory ability of the three risk models, the authors provide some information on the quality of the calibration. The AUC mentioned above is a measure of how well a model discriminates between subjects who will or will not develop a particular complication (ie, it tests whether the model will assign a higher event probability to the patient with an event compared to the no-event subject).7 Calibration, in turn, measures how well the predicted event probabilities agree with the real, observed event rates. The authors chose to present calibration data in a bar plot, while conventionally a calibration plot is used to this end. Figure 1 shows the calibration data extracted from the manuscript (figure 2 in the paper by Balci et al) in a calibration plot. This plot exposes the poor agreement between predicted and actual outcomes for both the CARPREG and the ZAHARA model in patients with an actual risk of cardiovascular complications greater than approximately 20%. In fact, both models overestimate the risk of
1
Division of Adult Congenital and Valvular Heart Disease, Department of Cardiovascular Medicine, University Hospital Muenster, Muenster, Germany; 2 Imperial College of Science and Medicine, London, UK; 3NIHR Cardiovascular and Respiratory Biomedical Research Unit, Adult Congenital Heart Centre and Centre for Pulmonary Hypertension, Royal Brompton Hospital, London, UK Correspondence to Dr Gerhard-Paul Diller, Division of Adult Congenital and Valvular Heart Disease, Department of Cardiovascular Medicine University Hospital of Münster, Albert-Schweitzer-Str. 33, Münster 48149, Germany; [email protected]
Figure 1 Calibration plot according to the data provided in the publication by Balci et al. The plot illustrates the modest calibration, especially in high-risk patients, where the actual risk is significantly overestimated. The size of the dot corresponds to the number of patients included in each subgroup.
Diller G-P, et al. Heart September 2014 Vol 100 No 17
1311
Downloaded from http://heart.bmj.com/ on February 20, 2015 - Published by group.bmj.com
Editorial roughly guide the clinician when counselling women with CHD of childbearing age about their individual risk associated with pregnancy. The estimated risk obtained by any score must be adjusted to factors that are not included in the scoring systems. It also emphasises that it is an important part of prepregnancy counselling to make women aware of the limitations and inaccuracies of any risk assessment, and the considerable amount of uncertainty related to pregnancy with their condition. Contributors both authors contributed equally to the manuscript, wrote and critically reviewed the final product. Competing interests None.
Figure 2 The difference between apparent and true performance: while apparent performance measures how well the model fits the original (derivation) sample, true performance provides a measure on the performance of the model when applied to an external cohort. higher-risk patients by as much as twofold (eg, patients with a predicted risk of >70% according to the ZAHARA score had an actual event rate of 35%). This result is particularly noteworthy, as the internal validity of the CARPREG was previously reported to be acceptable.5 8 As a consequence, the paper by Baci et al shows that currently available pregnancy risk scores suffer, both, from an only modest discriminatory ability and a poor calibration, especially in higher-risk patients. This limits the generalisability of the risk models to clinical cohorts of CHD patients outside the centres involved in the derivation of the respective risk models. It should be emphasised that the number of women at risk in some of the subgroups were rather small, therefore reducing the accuracy of the predictions. Interestingly, the presence of a mechanical prosthesis was associated with a high risk for complications (6/11), and this is particularly noteworthy as this parameter is not universally included as part of risk prediction scores. Formally, before a risk prediction model should be applied routinely to a large population, one should ensure that the correct statistical model has been chosen, that the derivation sample is representative of the underlying population, that the model performs adequately in the derivation sample and, ultimately, the validity should be assessed in an external population. Especially, the last step is often avoided in CHD—but it is crucial. This is because even if the predication model reflects the underlying dataset quite
1312
well, it may be sensitive to idiosyncrasies of this derivation sample and may perform poorly in another sample. This fact is illustrated in figure 2, showing the difference between apparent and true performance. Clinicians are interested in the true performance of a model as they try to predict the outcome of patients not included in the initial (derivation) study. The difference between apparent and true performance is also illustrated in the current study by Balci et al showing that one of the risk models failed to statistically deviate significantly from a ‘random guess’ in their patient sample. The lack of external validation is not a specific problem of CHD studies: Perel et al have studied 102 prognostic models in traumatic brain injury and found that only 11% of these provided data on the external validity of the data.9 The importance of the current study goes beyond estimation of pregnancy risk. It illustrates the problems with riskprediction models in CHD in general. Given the heterogeneous nature of CHD and the limited number of patients with some conditions at single centres, clinicians will have to continue to rely on unvalidated prognostic models for the foreseeable future. Nevertheless, in light of theoretical considerations and the results of the current study by Balci et al, every attempt should be made to externally validate and, if required, update available risk models to the benefit of CHD patients. The study by Balci et al also stresses that all available risk scores can only
Provenance and peer review Commissioned; internally peer reviewed.
To cite Diller G-P, Uebing A. Heart 2014;100:1311– 1312.
▸ http://dx.doi.org/10.1136/heartjnl-2014-305597 Heart 2014;100:1311–1312. doi:10.1136/heartjnl-2014-306060
REFERENCES 1 2
3
4
5
6
7 8
9
Uebing A, Steer PJ, Yentis SM, et al. Pregnancy and congenital heart disease. BMJ 2006;332:401. Klayman J, Soll JB, Gonzalez-Vallejo C, et al. Overconfidence: it depends on how, what, and whom you ask. Organ Behav Hum Decis Process 1999;79:216–47. Balci A, Sollie-Szarynska KM, van der Bijl AGL, et al. Prospective validation and assessment of cardiovascular and offspring risk models for pregnant women with congenital heart disease. Heart 2014;100:1373–81. Drenthen W, Boersma E, Balci A, et al. Predictors of pregnancy complications in women with congenital heart disease. Eur Heart J 2010;31:2124–32. Siu SC, Sermer M, Colman JM, et al. Prospective multicenter study of pregnancy outcomes in women with heart disease. Circulation 2001;104:515–21. Regitz-Zagrosek V, Lundqvist CB, Borghi C, et al. ESC Guidelines on the management of cardiovascular diseases during pregnancy The Task Force on the Management of Cardiovascular Diseases during Pregnancy of the European Society of Cardiology (ESC). Eur Heart J 2011;32:3147–97. Steyerberg EW. Clinical prediction models. Springer, 2009. Khairy P, Ouyang DW, Fernandes SM, et al. Pregnancy outcomes in women with congenital heart disease. Circulation 2006;113:517–24. Perel P, Edwards P, Wentz R, et al. Systematic review of prognostic models in traumatic brain injury. BMC Med Inform Decis Mak 2006;6:38.
Diller G-P, et al. Heart September 2014 Vol 100 No 17
Downloaded from http://heart.bmj.com/ on February 20, 2015 - Published by group.bmj.com
Predicting the risks of pregnancy in congenital heart disease: the importance of external validation Gerhard-Paul Diller and Anselm Uebing Heart 2014 100: 1311-1312
doi: 10.1136/heartjnl-2014-306060 Updated information and services can be found at: http://heart.bmj.com/content/100/17/1311
These include:
References Email alerting service
Topic Collections
This article cites 8 articles, 6 of which you can access for free at: http://heart.bmj.com/content/100/17/1311#BIBL Receive free email alerts when new articles cite this article. Sign up in the box at the top right corner of the online article.
Articles on similar topics can be found in the following collections Drugs: cardiovascular system (8099) Hypertension (2745) Congenital heart disease (672)
Notes
To request permissions go to: http://group.bmj.com/group/rights-licensing/permissions To order reprints go to: http://journals.bmj.com/cgi/reprintform To subscribe to BMJ go to: http://group.bmj.com/subscribe/
