European Journal of Heart Failure (2014) 16, 173–179 doi:10.1111/ejhf.32
Predicting survival in heart failure: validation of the MAGGIC heart failure risk score in 51 043 patients from the Swedish Heart Failure Registry Ulrik Sartipy1,2*, Ulf Dahlström3, Magnus Edner4,5, and Lars H. Lund4,6 1 Department
of Cardiothoracic Surgery and Anesthesiology, Karolinska University Hospital, Stockholm, Sweden; 2 Department of Molecular Medicine and Surgery, Karolinska Institutet, Stockholm, Sweden; 3 Division of Cardiovascular Medicine, Department of Medicine and Health Sciences, Faculty of Health Sciences, Linköping University, Department of Cardiology UHL, County Council of Östergötland, Linköping, Sweden; 4 Department of Medicine, Karolinska Institutet, Stockholm, Sweden; 5 Cardiology Research Unit, Karolinska University Hospital, Stockholm, Sweden; and 6 Department of Cardiology, Karolinska University Hospital, Stockholm, Sweden Received 24 April 2013; revised 9 July 2013; accepted 12 July 2013 ; online publish-ahead-of-print 14 December 2013
Aims
The aim of this study was to evaluate the performance of a recently developed risk score for mortality in heart failure by external validation in a national heart failure registry. ..................................................................................................................................................................... Methods and From 13 routinely available patient characteristics, the Meta-analysis Global Group in Chronic Heart Failure results (MAGGIC) constructed a risk score for prediction of mortality in heart failure. We included 51 043 patients from the national Swedish Heart Failure Registry and calculated the MAGGIC risk score for each patient. The outcome measure was 3-year mortality. The predicted probability of death obtained from the calculated risk score was compared with the observed 3-year mortality, and model discrimination and calibration were assessed by formal tests and graphical means. The overall 3-year mortality in the study population was 39.4% and the MAGGIC project heart failure risk score predicted mortality was 36.4% (observed to expected ratio: 1.08). Discrimination was excellent overall (C index = 0.741). The difference between the model-predicted and the observed 3-year mortality in the six risk groups varied between 5% and −12%. Calibration plots demonstrated slight overprediction for the lowest risk patients, and underprediction in high risk patients. ..................................................................................................................................................................... Conclusion The MAGGIC project heart failure risk score demonstrated an excellent ability to categorize patients in separate risk strata. Although the predicted 3-year mortality risk was higher in low risk groups and lower in high risk groups compared with the observed 3-year mortality in the Swedish Heart Failure Registry, the MAGGIC project heart failure risk score performed well in a large nationwide contemporary external validation cohort.
.......................................................................................................... Heart failure • Survival • Prognostic risk models • External validation
Introduction Clinical prediction models are often used as an adjunct to clinical reasoning and decision-making in cardiology1,2 and cardiac surgery,3 and have the potential to aid professionals and patients through better understanding of prognosis. Prediction models use several predictors to estimate the risk that a specified event will
....................
Keywords
happen within a specific time period in an individual. The Metaanalysis Global Group in Chronic Heart Failure (MAGGIC) performed a literature-based meta-analysis4 and extracted individual patient data from 30 studies regarding demographics, medical history, medical treatment, symptom status, clinical variables, laboratory variables, and outcome. From 13 routinely available patient characteristics, they constructed a risk score for mortality in
*Corresponding author. Department of Cardiothoracic Surgery and Anesthesiology, Karolinska University Hospital, SE-171 76 Stockholm, Sweden. Tel: +46 8 517 728 94, Fax: +46 8 33 19 31, Email: [email protected]
© 2013 The Authors European Journal of Heart Failure © 2013 European Society of Cardiology
174
Methods Study population The Swedish Heart Failure Registry (RiksSvikt) has been previously described.12,13 The inclusion criterion is clinician-judged heart failure. Eighty variables are recorded at discharge from hospital or after clinic visit, and entered into an electronic database managed by the Uppsala Clinical Research Center (Uppsala, Sweden). The database is continuously updated regarding vital status by linkage to the Total Population Register at Statistics Sweden. The protocol, registration form, and annual report are available at http://www.rikssvikt.se. Establishment of the registry and registration and analysis of data were approved by a multisite Ethics Committee. The registry and this study conform to the Declaration of Helsinki. Individual patient consent is not required or obtained, but patients are informed of being entered into national registries and are allowed to opt out. Between 11 May 2000 and 1 November 2012, there were 78 692 registrations in the Swedish Heart Failure Registry from 66 of 77 hospitals and 115 of 1011 primary care outpatient clinics in Sweden, representing 51 064 unique patients. We excluded 13 patients with missing/unknown entry date, and 8 patients with missing/unknown birth date, leaving a final study population of 51 043 patients.
Data preparation and missing data Data were missing for some variables in the data set. We used multiple imputation by chained equations to impute missing values.14 The event indicator and the Nelson–Aalen estimator of the cumulative baseline hazard were included in the imputation model.15 Based on our original data set, we generated 25 multiply imputed data sets, and estimates from these data sets were combined using standard methods provided in Stata version 12.1.14 All analyses except for descriptive statistics were performed on the imputed data set. The 13 variables required for calculation of the MAGGIC project heart failure risk score were: age, EF, NYHA class, serum creatinine, diabetes, systolic blood pressure, body mass index, heart failure
..................................................................................................................................................................................
patients with heart failure5 and made an easy to use calculator available online at the website www.heartfailurerisk.org. In addition to the risk score, MAGGIC has also explored individual risk factors such as serum sodium,6 the role of gender,7 and survival of patients with heart failure with preserved or reduced LVEF.8 The MAGGIC project heart failure risk score was found to have excellent discrimination and calibration, and the authors argued that their findings would be generalizable to current and future patients globally due to the variety of different studies included in their database. However, it has been shown that the real-world performance of prediction models is generally lower in new patient populations compared with the original developmental study population.9 Even if the prediction model was internally validated, it is therefore necessary to test the performance in a new set of patients before the model can be recommended for use in clinical practice.10,11 The concept of validating a prognostic model means establishing that it works satisfactorily for patients other than those from whose data it was derived.9 The objective of this study was to evaluate the performance of the MAGICC project heart failure risk score by external validation in the Swedish Heart Failure Registry population.
U. Sartipy et al.
duration, current smoker, COPD, male gender, not prescribed a betablocker, and not prescribed an ACE inhibitor or ARBs. Eleven variables were available in the Swedish Heart Failure Registry and could be used in their original form without special considerations or approximations. However, EF and heart failure duration needed to be re-categorized.
Ejection fraction and heart failure duration The LVEF was available from the registry in the following categories: ≥50% (n = 9746), 40–49% (n = 9076), 30–39% (n =11 532), and
Predicting survival in heart failure: validation of the MAGGIC heart failure risk score in 51 043 patients from the Swedish Heart Failure Registry Ulrik Sartipy1,2*, Ulf Dahlström3, Magnus Edner4,5, and Lars H. Lund4,6 1 Department
of Cardiothoracic Surgery and Anesthesiology, Karolinska University Hospital, Stockholm, Sweden; 2 Department of Molecular Medicine and Surgery, Karolinska Institutet, Stockholm, Sweden; 3 Division of Cardiovascular Medicine, Department of Medicine and Health Sciences, Faculty of Health Sciences, Linköping University, Department of Cardiology UHL, County Council of Östergötland, Linköping, Sweden; 4 Department of Medicine, Karolinska Institutet, Stockholm, Sweden; 5 Cardiology Research Unit, Karolinska University Hospital, Stockholm, Sweden; and 6 Department of Cardiology, Karolinska University Hospital, Stockholm, Sweden Received 24 April 2013; revised 9 July 2013; accepted 12 July 2013 ; online publish-ahead-of-print 14 December 2013
Aims
The aim of this study was to evaluate the performance of a recently developed risk score for mortality in heart failure by external validation in a national heart failure registry. ..................................................................................................................................................................... Methods and From 13 routinely available patient characteristics, the Meta-analysis Global Group in Chronic Heart Failure results (MAGGIC) constructed a risk score for prediction of mortality in heart failure. We included 51 043 patients from the national Swedish Heart Failure Registry and calculated the MAGGIC risk score for each patient. The outcome measure was 3-year mortality. The predicted probability of death obtained from the calculated risk score was compared with the observed 3-year mortality, and model discrimination and calibration were assessed by formal tests and graphical means. The overall 3-year mortality in the study population was 39.4% and the MAGGIC project heart failure risk score predicted mortality was 36.4% (observed to expected ratio: 1.08). Discrimination was excellent overall (C index = 0.741). The difference between the model-predicted and the observed 3-year mortality in the six risk groups varied between 5% and −12%. Calibration plots demonstrated slight overprediction for the lowest risk patients, and underprediction in high risk patients. ..................................................................................................................................................................... Conclusion The MAGGIC project heart failure risk score demonstrated an excellent ability to categorize patients in separate risk strata. Although the predicted 3-year mortality risk was higher in low risk groups and lower in high risk groups compared with the observed 3-year mortality in the Swedish Heart Failure Registry, the MAGGIC project heart failure risk score performed well in a large nationwide contemporary external validation cohort.
.......................................................................................................... Heart failure • Survival • Prognostic risk models • External validation
Introduction Clinical prediction models are often used as an adjunct to clinical reasoning and decision-making in cardiology1,2 and cardiac surgery,3 and have the potential to aid professionals and patients through better understanding of prognosis. Prediction models use several predictors to estimate the risk that a specified event will
....................
Keywords
happen within a specific time period in an individual. The Metaanalysis Global Group in Chronic Heart Failure (MAGGIC) performed a literature-based meta-analysis4 and extracted individual patient data from 30 studies regarding demographics, medical history, medical treatment, symptom status, clinical variables, laboratory variables, and outcome. From 13 routinely available patient characteristics, they constructed a risk score for mortality in
*Corresponding author. Department of Cardiothoracic Surgery and Anesthesiology, Karolinska University Hospital, SE-171 76 Stockholm, Sweden. Tel: +46 8 517 728 94, Fax: +46 8 33 19 31, Email: [email protected]
© 2013 The Authors European Journal of Heart Failure © 2013 European Society of Cardiology
174
Methods Study population The Swedish Heart Failure Registry (RiksSvikt) has been previously described.12,13 The inclusion criterion is clinician-judged heart failure. Eighty variables are recorded at discharge from hospital or after clinic visit, and entered into an electronic database managed by the Uppsala Clinical Research Center (Uppsala, Sweden). The database is continuously updated regarding vital status by linkage to the Total Population Register at Statistics Sweden. The protocol, registration form, and annual report are available at http://www.rikssvikt.se. Establishment of the registry and registration and analysis of data were approved by a multisite Ethics Committee. The registry and this study conform to the Declaration of Helsinki. Individual patient consent is not required or obtained, but patients are informed of being entered into national registries and are allowed to opt out. Between 11 May 2000 and 1 November 2012, there were 78 692 registrations in the Swedish Heart Failure Registry from 66 of 77 hospitals and 115 of 1011 primary care outpatient clinics in Sweden, representing 51 064 unique patients. We excluded 13 patients with missing/unknown entry date, and 8 patients with missing/unknown birth date, leaving a final study population of 51 043 patients.
Data preparation and missing data Data were missing for some variables in the data set. We used multiple imputation by chained equations to impute missing values.14 The event indicator and the Nelson–Aalen estimator of the cumulative baseline hazard were included in the imputation model.15 Based on our original data set, we generated 25 multiply imputed data sets, and estimates from these data sets were combined using standard methods provided in Stata version 12.1.14 All analyses except for descriptive statistics were performed on the imputed data set. The 13 variables required for calculation of the MAGGIC project heart failure risk score were: age, EF, NYHA class, serum creatinine, diabetes, systolic blood pressure, body mass index, heart failure
..................................................................................................................................................................................
patients with heart failure5 and made an easy to use calculator available online at the website www.heartfailurerisk.org. In addition to the risk score, MAGGIC has also explored individual risk factors such as serum sodium,6 the role of gender,7 and survival of patients with heart failure with preserved or reduced LVEF.8 The MAGGIC project heart failure risk score was found to have excellent discrimination and calibration, and the authors argued that their findings would be generalizable to current and future patients globally due to the variety of different studies included in their database. However, it has been shown that the real-world performance of prediction models is generally lower in new patient populations compared with the original developmental study population.9 Even if the prediction model was internally validated, it is therefore necessary to test the performance in a new set of patients before the model can be recommended for use in clinical practice.10,11 The concept of validating a prognostic model means establishing that it works satisfactorily for patients other than those from whose data it was derived.9 The objective of this study was to evaluate the performance of the MAGICC project heart failure risk score by external validation in the Swedish Heart Failure Registry population.
U. Sartipy et al.
duration, current smoker, COPD, male gender, not prescribed a betablocker, and not prescribed an ACE inhibitor or ARBs. Eleven variables were available in the Swedish Heart Failure Registry and could be used in their original form without special considerations or approximations. However, EF and heart failure duration needed to be re-categorized.
Ejection fraction and heart failure duration The LVEF was available from the registry in the following categories: ≥50% (n = 9746), 40–49% (n = 9076), 30–39% (n =11 532), and
