JACC: HEART FAILURE
VOL.
ª 2014 BY THE AMERICAN COLLEGE OF CARDIOLOGY FOUNDATION PUBLISHED BY ELSEVIER INC.
-, NO. -, 2014
ISSN 2213-1779/$36.00 http://dx.doi.org/10.1016/j.jchf.2014.04.006
STATE-OF-THE-ART PAPER
Factors Influencing the Predictive Power of Models for Predicting Mortality and/or Heart-Failure Hospitalization in Patients With Heart Failure Wouter Ouwerkerk, MSC,* Adriaan A. Voors, MD, PHD,y Aeilko H. Zwinderman, PHD*
ABSTRACT The present paper systematically reviews and compares existing prediction models in order to establish the strongest variables, models, and model characteristics in patients with heart failure predicting outcome. To improve decision making accurately predicting mortality and heart-failure hospitalization in patients with heart failure can be important for selecting patients with a poorer prognosis or nonresponders to current therapy, to improve decision making. MEDLINE/ PubMed was searched for papers dealing with heart failure prediction models. To identify similar models on the basis of their variables hierarchical cluster analysis was performed. Meta-analysis was used to estimate the mean predictive value of the variables and models; meta-regression was used to find characteristics that explain variation in discriminating values between models. We identified 117 models in 55 papers. These models used 249 different variables. The strongest predictors were blood urea nitrogen and sodium. Four subgroups of models were identified. Mortality was most accurately predicted by prospective registry-type studies using a large number of clinical predictor variables. Mean C-statistic of all models was 0.66 0.0005, with 0.71 0.001, 0.68 0.001 and 0.63 0.001 for models predicting mortality, heart failure hospitalization, or both, respectively. There was no significant difference in discriminating value of models between patients with chronic and acute heart failure. Prediction of mortality and in particular heart failure hospitalization in patients with heart failure remains only moderately successful. The strongest predictors were blood urea nitrogen and sodium. The highest C-statistic values were achieved in a clinical setting, predicting short-term mortality with the use of models derived from prospective cohort/registry studies with a large number of predictor variables. (J Am Coll Cardiol HF 2014;-:-–-) © 2014 by the American College of Cardiology Foundation.
H
eart failure (HF) is a major cause of cardio-
Accurately predicting prognosis can be of benefit
vascular mortality and morbidity (1), and
for patients with heart failure. First, patients with a
its prevalence and incidence is increasing
poorer prognosis might benefit more from aggressive
(2). Despite wider use of evidence-based medical
treatment and a closer follow-up (5). Prediction rules
therapy and preventive device therapy, the prognosis
like the CHA 2DS2-VASc (Congestive heart failure [or
remains poor (3,4).
left ventricular systolic dysfunction]; Hypertension:
From the *Department of Clinical Epidemiology, Biostatistics, and Bioinformatics, Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands; and the yDepartment of Cardiology, University of Groningen, University Medical Center, Groningen, the Netherlands. Prof. Voors received consultancy fees and/or research grants from Alere, Bayer HealthCare, Cardio3Biosciences, Celladon, Novartis, Servier, Torrent, Trevena Vifor Pharmaceuticals; is supported by a grant (#FP7-242209-BIOSTAT-CHF) from the European Commission; and is a clinical established investigator (2006T37) of the Dutch Heart Foundation. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose. Manuscript received December 3, 2013; revised manuscript received April 14, 2014, accepted April 15, 2014.
2
Ouwerkerk et al.
JACC: HEART FAILURE VOL.
-, NO. -, 2014 - 2014:-–-
Discriminating Power of Models Predicting Outcome in HF
ABBREVIATIONS
blood pressure consistently above 140/90
and “(re-)hospitalization.” These terms had to be used
AND ACRONYMS
mm Hg [or treated hypertension on medica-
in either title or abstract. We drew a distinction
tion]; Age $75 years; Diabetes mellitus; pre-
between papers including patients diagnosed with
vious stroke or transient ischemic attack or
CHF and those with ADHF.
ADHF = acute decompensated heart failure
CHF = chronic heart failure HF = heart failure HR = hazard ratio OR = odds ratio
thromboembolism; Vascular disease [e.g., peripheral artery disease, myocardial infarction, aortic plaque]; Age 65 to 74 years; Sex category [i.e., female sex]) and the TIMI (Thrombolysis In Myocardial Infarction) risk
score are widely used in clinical practice to identify high-risk patients who require medical/surgical treatment (6,7). These prediction rules are used to justify medical treatment (like antithrombotic therapy for patients with atrial fibrillation). However, the clinical value of the HF risk predictors remains limited. Second, accurately predicting mortality and morbidity might help patients in their decision making (8). Third, identifying patients who are at risk and do not respond to currently recommended therapies for HF might lead to personalized medicine aimed at targeted treatments for patients with HF. Finally, improved prognostic
risk models may help
in
designing trials by choosing population characteristics with higher event rates. Many studies on predictive markers for outcome in
DATA EXTRACTION. Papers were only included in
the present analysis when the predictor variables were reported and the predicted value was quantified using the C-statistics or receiver-operating characteristic curve value. Of these papers, all reported models were included, even if only variable or C-statistics were available. We collected the predicted outcome variable of each model and grouped these according to the following: 1) mortality; 2) mortality or HF hospitalization; or 3) HF hospitalization. We documented all predictor variables and, when published, their predictive power (odds ratio [OR] and hazard ratio [HR]). We reduced the multitude of different variables to generalized variables. (See the Online Appendix for details on the data extraction, data reduction, and statistical analysis.) For each model—derivation or validation (whether the paper described a new model or validated an existing one)— we collected the following information: Total number of variables in the model
patients with HF and several reviews on prediction
Time of prediction in days (the period for which the
models have been published. Most of these models
model makes its prediction [e.g., in 1-year mortal-
focused on prediction of HF hospitalization and were found to perform poorly or only averagely on a specific patient population (9–12). Nutter et al. (13) evaluated
ity, the time of prediction would be 365 days]) Study design (randomized controlled trial, cohort study or registry)
6 prognostic models predicting mortality. They
Retrospective or prospective data collection
concluded that the prediction models used were
Type of prediction model (subjective prediction, a
adequate in discriminating patients, but they might
risk score– or regression-based model)
underestimate absolute risk of mortality in elderly
The form of statistical analysis used to derive the
patients (14). All these reviews focused on descriptive
model (classification and regression tree analysis,
analyses to explain the predictive power of the models.
Cox proportional hazards regression, generalized
The goal of this paper is to provide an overview of different prediction models developed in the recent years. We compared models with respect to the number, type, and predictive power of predictor
linear model or hierarchical modified Poisson regression) Source of data (medical records or administrative registry data)
variables used. We performed a meta-analysis to
Total number of patients studied in the paper
detect the predictive and discriminating value of
Mean age
variables and models and analyzed which model
Percent of male patients
characteristics were associated with the highest C-statistic value.
STATISTICAL ANALYSIS. First, models were com-
pared with respect to the predictor variables. We
METHODS
therefore performed hierarchical cluster analysis to identify those subgroups of models that were com-
SEARCH STRATEGY. MEDLINE/PubMed was searched
parable (15–17). We counted the models in each sub-
for relevant English-language papers using established
group incorporating more than 5 models. Next, the
methods. Search terms used were: “heart failure,”
predictive weights of the variables for predicting
“chronic heart failure” (CHF), “acute decompensated
mortality
heart failure” (ADHF), “risk scores,” “prediction,”
analyzed, using fixed and random effect models. We
“prognosis,” “models,” “mortality,” “(re-)admission,”
separately analyzed the z-scores, which are the OR/SE
and
HF
hospitalization
were
meta-
JACC: HEART FAILURE VOL.
-, NO. -, 2014
Ouwerkerk et al.
- 2014:-–-
Discriminating Power of Models Predicting Outcome in HF
(OR) and the HR/SE (HR), of the OR and HR of pre-
of the papers there were 10 additional models for
diction variables depending on whether they were
which no C-statistics were provided. These models
obtained from case-control or cohort studies.
were included in the next section comparing models
Third, we estimated the mean C-statistics by fixedand random-effects meta-analysis and used meta-
on their predictor variables, but were excluded from the meta-analysis and meta-regression.
regression to estimate the association between
We excluded 5 models from the variables analysis
study and model characteristics and C-statistics (18).
because these constituted models developed for measuring activities of daily life (19,20) or used only
RESULTS
subjective predictions by the physician and nurse (21).
We identified 117 different models in 55 papers, all
PREDICTORS OF OUTCOME. The prediction models
published between 1994 and 2012 (details are pre-
showed a great variety in number and type of vari-
sented in the Online Appendix). These models were
ables. The numbers varied from 1 (22–24) to 65 (25). A
also divided into CHF and ADHF groups, consisting of
total of 249 different variables were used in 117
10 and 111 models, respectively. Four models were
models. An OR/HR was for 140 variables; no OR/HR,
used in predicting events in both patient groups. In 8
therefore, was mentioned for 109 variables. Most
T A B L E 1 The Most Frequently Used Variables, Along With the Number of Times Used in the Different Models, and Their Predictive Scores
Odds Ratio
Hazard Ratio
n
z-Score
Mean
95% CI
z-Score
Mean
95% CI
Age, yrs
78
17,656
1.04
1.03 to 1.04
20,728
1.06
1.06 to 1.07
Sex
53
15,542
1.12
1.11 to 1.14
10,813
1.22
1.19 to 1.26
Systolic blood pressure
51
31,799
1.30
1.29 to 1.32
19,353
1.16
1.15 to 1.18
Sodium
46
33,502
1.41
1.39 to 1.43
30,343
1.09
1.08 to 1.09
Diabetes
41
15,849
1.14
1.12 to 1.15
21,227
1.44
1.41 to 1.47
Creatinine
38
34,191
1.12
1.11 to 1.13
10,411
1.07
1.05 to 1.08
CHF and ADHF
New York Heart Association functional class
35
15,109
4.14
3.96 to 4.33
17,683
1.41
1.37 to 1.45
Blood urea nitrogen
28
60,698
2.28
2.26 to 2.31
12,495
1.07
1.06 to 1.09
12,843
1.08
1.07 to 1.10
15,346
1.12
1.11 to 1.14
8,526
1.08
1.06 to 1.10
15,327
1.44
1.39 to 1.49
Ejection fraction
23
Hemoglobin
23
(N-terminal pro) B-type natriuretic peptide
23
CHF Age, yrs
75
17,566
1.04
1.03 to 1.04
20,650
1.06
1.06 to 1.07
Sex
53
15,542
1.12
1.11 to 1.14
10,813
1.22
1.19 to 1.26
Systolic blood pressure
46
54,716
1.16
1.15 to 1.16
18,647
1.18
1.16 to 1.19
Sodium
44
33,502
1.41
1.39 to 1.43
36,527
1.07
1.06 to 1.07
Diabetes
41
15,849
1.14
1.12 to 1.15
21,227
1.44
1.41 to 1.47
Creatinine
37
34,191
1.12
1.11 to 1.13
10,411
1.07
1.05 to 1.08
New York Heart Association functional class
34
15,109
4.14
3.96 to 4.33
17,506
1.41
1.37 to 1.44
Blood urea nitrogen
25
61,497
2.37
2.34 to 2.39
12,014
1.07
1.06 to 1.08
Ejection fraction
23
12,843
1.08
1.07 to 1.10
(N-terminal pro) B-type natriuretic peptide
23
27,196
1.11
1.1 to 1.12
Etiology
22
6,795
1.23
1.17 to 1.29 -0.01 to 2.57
ADHF Systolic blood pressure
7
2,181
1.22
1.04 to 1.41
0.378
1.28
Age, yrs
6
8,454
1.46
1.37 to 1.55
0.254
1.27
-0.57 to 3.11
Blood urea nitrogen
4
0.250
1.26
-0.55 to 3.07
0.339
1.33
-0.32 to 2.98
Heart failure admissions
4
12,443
1.44
1.39 to 1.5
0.737
2.00
Sodium
4
0.530
1.53
-0.04 to 3.1
1,663
1.33
Dementia/Alzheimer disease or senility
3
6,306
2.42
2.15 to 2.70
Cancer
2
0.827
1.86
0.39 to 3.33
Cerebrovascular disease
2
0.410
1.43
-0.28 to 3.14
Creatinine
2
Sex
2
0.636
1.63
0.13 to 3.13
0.438
1.33
ADHF ¼ acute decompensated heart failure; CHF ¼ chronic heart failure; CI ¼ confidence interval.
0.99 to 1.67
3
4
Ouwerkerk et al.
JACC: HEART FAILURE VOL.
T A B L E 2 The 10 Variables With the Highest z-Score Used >5 in Models
n
z-Score
-, NO. -, 2014 - 2014:-–-
Discriminating Power of Models Predicting Outcome in HF
Mean
95% CI
10 highest standardized OR
different models, and their predictive power, are shown in Table 1. Table 2 summarizes the value of the strongest predictor variables used most frequently measured by
Blood urea nitrogen
28
60.698
2.28
2.26-2.31
Cancer
10
37,997
1.84
1.81-1.88
Troponin
8
34,585
1.73
1.69-1.76
the z-score of OR and HR. The predictive value of predictor variables for CHF; ADHF; and for mortality, mortality or HF hospitalization, or HF hospitalization
Creatinine
38
34,191
1.12
1.11-1.13
Sodium
46
33,502
1.41
1.39-1.43
Systolic blood pressure
51
31,799
1.3
1.29-1.32
The most frequently used variables with the high-
Heart failure
9
27,924
1.28
1.26-1.30
est predictive values were blood urea nitrogen and
models are presented in the Online Appendix.
Arterial pH
6
27,753
1.87
1.83-1.92
sodium. There were 3 variables with a high predictive
Diastolic blood pressure
11
22,013
1.15
1.14-1.16
value of both OR and HR: sodium; blood urea nitro-
Renal failure
8
21,113
1.26
1.24-1.28
Sodium
46
30.343
1.09
1.08-1.09
Race
10
27,027
1.11
1.1-1.12
control studies, but not in prognostic cohort studies
Diabetes
41
21,227
1.44
1.41-1.47
(high OR, but low or no HR). The opposite situation
Age, yrs
77
20,728
1.06
1.06-1.07
was seen with ejection fraction and (N-terminal pro)
Systolic blood pressure
51
19,353
1.16
1.15-1.18
B-type natriuretic peptide, which were found to be
New York Heart Association functional class
35
17,683
1.41
1.37-1.45
highly prognostic in cohort studies but not in case-
(N-terminal pro) B-type natriuretic peptide
23
15,327
1.44
1.39-1.49
Angiotensin-converting enzyme inhibitor/angiotensin II receptor blockers
8
14,008
1.19
1.17-1.21
10 highest standardized HR
gen; and systolic blood pressure. Cancer, arterial pH, and renal failure were highly predictive in case-
control studies. In the ADHF models, the strongest predictor variable was HF admissions. Hierarchical clustering was performed on the prediction variables of the various models to identify
Ejection fraction
23
12,843
1.08
1.07-1.1
subgroups of similar models. The subgroups created
Blood urea nitrogen
28
12,495
1.07
1.06-1.09
by the hierarchical clustering are presented in Figure 1. There were 4 subgroups of models that
CI ¼ confidence interval; HR ¼ hazard ratio; OR ¼ odds ratio.
incorporated more than 5 models. The largest subgroup (purple) consisted of models using relatively few predictor variables. Variables used vary between
models used a combination of demographic, clinical,
these models; age, sodium, and systolic blood pres-
and easily obtainable data to achieve the highest
sure were the 3 most common variables used.
predictive power. The most frequently used vari-
A second subgroup (red) consisted of models on the
ables, along with the number of times used in the
basis of the Seattle Heart Failure Model. These models used the Seattle Heart Failure Model with addition of 1 or 2 predictor variables. The green subgroup contained
models
using
medication
(beta-blocking
agents), glomerular filtration rate, left ventricular ejection fraction, and New York Heart Association Class as predictor variables. The blue subgroup contained models that used such clinical variables as renal failure, weight, blood pressure, and body mass index, as distinct prediction variables different from the green and the other clusters. META-ANALYSIS OF THE PREDICTIVE VALUE FOR MORTALITY AND HF HOSPITALIZATION. There were
260 C-statistic values reported for 103 models: 181 comprised models predicting mortality; 32 were for HF hospitalization; and 47 for mortality or HF hospitalization. F I G U R E 1 Hierarchical Clustering Dendrogram
The models were ordered according to their similarity as a result of hierar-
There were more C-statistic values reported than there were models; 14 models were validated more
chical clustering. There were 4 subgroups (red, blue, green, and purple)
than once (up to 15). Eight models to predict mor-
found incorporated more than 5 models.
tality were also validated in predicting HF hospitalization. Although models were validated multiple
JACC: HEART FAILURE VOL.
-, NO. -, 2014
Ouwerkerk et al.
- 2014:-–-
Discriminating Power of Models Predicting Outcome in HF
times, and 79% of the C-statistic values were the
had larger weights in the overall mean. This resulted
result of derivation models, 69 models were as yet not
in the mean of 0.71 0.0010, whereas the raw un-
validated in separate patient cohorts.
corrected C-index mean was 0.74.
The highest C-statistic value (0.9) was achieved by
There
was
a
highly
significant
difference
Selker et al. (26) who used multivariable logistic
(p < 0.0001) in C-statistics for models predicting
regression to predict in-hospital mortality in patients
mortality, mortality or HF hospitalization, and HF
with CHF. The lowest C-statistic value (0.52) was
hospitalization: 0.71 0.001, 0.63 0.001, and 0.68
found in the multivariate model of Yamokoski et al.
0.001.
(21) (model 41 (21)) predicting 6-month HF hospitali-
The green cluster, with mean C-statistic values
zation. This value was lower than the C-statistic
of 0.83 0.003, had the highest predictive power of
values of the subjective prediction for HF hospitali-
all 4 clusters (blue: 0.82 0.018, red: 0.74 0.004,
zation by physicians and nurses (0.579 and 0.566,
and purple: 0.61 0.001). META-REGRESSION. Table 3 shows the results of
respectively) in the same paper. In Figure 2, we illustrate the C-statistics of models
multivariable meta-regression analysis of the relation
that reported a standard error or confidence interval.
between C-statistic and model/study characteristics.
The mean C-statistic, the black triangle in Figure 2,
Models predicting mortality had significantly higher
was 0.66 0.0005 for the models reporting a standard
C-statistic values than models predicting HF hospi-
error. Standard errors were reported in 3 models (17,
talization (D C-statistic ¼ 0.03, SE ¼ 0.001).
105, and 111) predicting mortality in ADHF. These
Papers describing the derivation of models re-
models had a mean C-statistic value of 0.71 0.0154,
ported lower C-statistic values than papers vali-
which was not statistically different from the mean
dating results ( D C-statistic ¼ -0.01, SE ¼ 0.0084).
statistic value for predicting mortality in CHF
Prospective cohort/registry studies yielded models
patients (0.71 0.0010). Keep in mind that these
with higher C-statistic values than models on the basis of data of randomized trials ( D C-statistic ¼
model, weighted for the SE-squared. All mortality
0.03, SE ¼ 0.037, DC-statistic ¼ 0.11, SE ¼ 0.042) or
models with a C-statistic value of |t|)
0.7035
0.1676
4.20
0.0000
Mortality
0.0296
0.0091
3.27
0.0012
181
Mortality or readmission
-0.0165
0.0106
-1.56
0.1207
47
in the characteristics of prediction models. The risk
0
-
-
-
32
factors found in these reviews (Ross et al. [9], Gia-
Validation
0
-
-
-
115
similar to the predictor variables found in this paper.
Derivation
-0.0095
0.0084
-1.14
0.2565
145
Ross et al. (9), Giamouzis et al. (10), and Betihavas
Number of variables
0.0036
0.0005
7.71
0.0000
256
et al. (12) also found that the C-statistic values of
Time of prediction
-0.0001
0.0000
-1.42
0.0003
260
(Intercept)
n
Outcome
Readmission
hospital HF hospitalization and compared differences
mouzis et al. [10], and Betihavas et al. [12]) were
Derivation or validation study
Study design Randomized controlled trial
prediction variables to the model. The previously published reviews focused only on
models predicting mortality was higher than that of models predicting HF hospitalization. They suggest
0
-
-
-
50
Cohort
0.0337
0.0374
0.90
0.3682
119
that either important predictors of HF hospitalization
Registry
0.1145
0.0422
2.71
0.0072
91
are lacking in the relevant models or that non-
0
-
-
-
124
talization risk. Developing a model using a systems
0.1025
0.0312
3.29
0.0012
136
biology approach, by incorporating information from
0
-
-
-
4
Point-based additive risk score
-0.0717
0.0376
-1.91
0.0579
44
Regression model
-0.0174
0.0321
-0.54
0.5885
209
medical factors may play a larger role in HF hospi-
Prospective or retrospective study Retrospective Prospective Type prediction model Subjective prediction
demographic, biomarker, genomic, proteomic, and the initial response to therapy might create a more effective prediction model and hopefully aid in understanding HF prognosis, as described by Giamouzis
Statistical method Generalized linear model
0
-
-
-
17
et al. (10). An additional advantage of this approach is
Cox proportional hazards
0.1078
0.0131
8.20
0.0000
134
that it will at least identify patients with a poor
Hierarchical modified Poisson
0.0940
0.0905
1.04
0.3000
4
outcome on currently recommended therapy, which
Multivariate logistic regression
0.1399
0.0205
6.82
0.0000
98
might lead to further development of targeted ther-
0
-
-
-
185
-0.1358
0.0198
-6.85
0.0000
75
-0.0000
0.0000
-3.54
0.0005
260
outcome for patients with HF. Although mortality models had the best discrimi-
Data source Medical records Claims data Sample size
apies,
eventually
leading
to
improvements
in
Age, yrs
-0.0013
0.0017
-0.78
0.4384
252
nating values, these models also have important
Male
-0.0010
0.0005
-1.92
0.0557
258
limitations. Nutter et al. (13) demonstrated that the
0
-
-
-
243
elderly cohort at or approaching the end of life.
-0.0014
0.0127
-0.11
0.9104
17
Nutter et al. (13) compared mortality prediction from
models underestimated the mortality risk in an
Patients’ diagnoses CHF ADHF Abbreviations as in Table 1.
6 prediction models in a retrospective cohort with a mean age of 82.7 8.2 and in-hospital death of 28.8%. The differences in predicted 1-year mortality between
Models using data from medical records had significantly better C-statistic values than models
the models was very high; the predicted mortality varied between 11.1 8.5% and 55.3 17.6% (13).
using claims data. Also models using more predictor
In addition to previous reviews (9,11), which
variables had better predictive values; C-statistic
enumerate the variables in prediction models, we
increased 0.0036 (SE ¼ 0.0005) with each added
quantified the predictive capabilities of each variable.
predictor variable.
We also explained variations in C-statistic values by
There was no significant difference in C-statistic values between patients diagnosed with either CHF or ADHF.
DISCUSSION
meta-analyzing model characteristics mentioned in the published reviews. PREDICTORS OF OUTCOME. The 10 models predict-
ing events in ADHF patients are not grouped into 1 subgroup in the dendrogram, as might be expected in
The present review shows that risk prediction in pa-
the event of the use of identical variables, but are
tients with CHF remains difficult. The best predictors
spread throughout the entire dendrogram. Similar to
of outcome were sodium and blood urea nitrogen. In
models developed for CHF patients, the models
addition, some characteristics of the prediction
developed for ADHF patients are inconclusive on
JACC: HEART FAILURE VOL.
-, NO. -, 2014
Ouwerkerk et al.
- 2014:-–-
Discriminating Power of Models Predicting Outcome in HF
which variables to use. There is no consensus as to
These models want to measure quality-of-care in
which variables are to be used to achieve the highest
contrast to patient disease prognosis. Krumholtz
predictive values.
et al. (27) shows that using claims data has its limi-
DISCRIMINATIVE POWER OF MODELS. Sixty-nine
tations but can be used to compare hospital-specific
derivation models were not yet validated. These
rehospitalization rates. Despite the reduced discrim-
models probably overestimate the prediction capa-
inating power, the risk of rehospitalization may be
bilities. Most of these models used internal valida-
more dependent on quality-of-care and system char-
tion, utilizing a bootstrap method for validation.
acteristics. It is important, therefore, to keep the
This, however, does not account for varying patient
objective in mind when creating a prediction model.
populations.
STUDY LIMITATIONS. Our meta-analysis was limited
In the meta-analysis, we only used the C-statistics
to published data only. From the 113 prediction
values of the models reporting standard errors. This
models found only 103 reported C-statistics, of which
might result in an underestimation of the mean
only 50 models in turn incorporated standard errors
values in the analysis; the raw mean C-statistic value
with their C-statistic values.
was, after all, higher than the mean C-statistic in the
We could not include all variations mentioned by
meta-analysis. Nevertheless, mean C-statistics indi-
Ross et al. (9) and Giamouzis et al. (10) in the meta-
cate only moderate predictive capacity.
regression. These variations might result in higher
PREDICTION MODELS. It is less difficult to predict
DC-statistic values than variations currently included
mortality, which had significantly higher C-statistic
in the meta-regression.
values, than to predict HF hospitalization. As expected, models developed in a derivation set reported
CONCLUSIONS
higher C-statistics than papers validating these models in a different patient population. In addition,
There are still difficulties associated with predicting
we found that cohort and prospective studies pro-
mortality and/or HF hospitalization in HF patients
duced higher C-statistics than models on the basis of
Prediction models need to be improved before they
data of randomized trials or retrospective data. Ran-
can be helpful to physicians and patients. Devel-
domized controlled trial studies had lower C-statistics
oping a model using a systems biology approach,
because randomized controlled trials were not pri-
incorporating information from demographic, bio-
marily created for model development, and the pop-
marker, genomic, proteomic, and initial responses to
ulation is more homogeneous (with therefore less
therapy might create a more effective model. An
discriminating capacity), relatively healthy (less
additional advantage of this approach is that it may
comorbidities), and highly controlled.
serve to identify patients with a poor outcome with
Models using data from medical records had
currently recommended therapy, thereby leading to
significantly better C-statistic values than models
the further development of targeted therapies and
using claims data did. This would suggest that pre-
eventually to improvements in outcome for patients
diction models are most accurate when created with
with HF.
data from patients followed prospectively in a cohort study using data from medical records. Models pre-
REPRINT REQUESTS AND CORRESPONDENCE: Dr.
dicting rehospitalization and mortality rates, how-
Wouter Ouwerkerk, Department of Clinical Epidemi-
ever, often use claims data instead of data from
ology, Biostatistics and Bioinformatics, Academic
medical records. Models predicting rehospitalization
Medical Center, P.O. 22660, room J1B-207, Meiberg-
and mortality rates are developed for purposes
dreef 9, 1105 AZ Amsterdam, the Netherlands. E-mail:
different from those for predicting disease prognosis.
[email protected].
REFERENCES 1. McCullough PA, Philbin EF, Spertus JA, et al., for the REACH Study Investigators. Confirmation of a
Force for the Diagnosis and Treatment of Acute and Chronic Heart Failure 2012 of the European
4. Thom T, Haase N, Rosamond W, et al. Heart disease and stroke statistics–2006 update: a
heart failure epidemic: findings from the Resource Utilization Among Congestive Heart Failure (REACH) study. J Am Coll Cardiol 2002;39:60–9.
Society of Cardiology. Eur Heart J 2012;33: 1787–847.
report from the American Heart Association Statistics Committee and Stroke Statistics Subcommittee. Circulation 2006;113:e85–151.
2. McMurray JJ, Adamopoulos S, Anker SD, et al. ESC guidelines for the diagnosis and treatment of acute and chronic heart failure 2012: the Task
3. Roger VL, Weston SA, Redfield MM, et al. Trends in heart failure incidence and survival in a community-based population. JAMA 2004;292: 344–50.
5. Hunt SA, Abraham WT, Chin MH, et al. 2009 focused update incorporated into the ACC/AHA 2005 Guidelines for the Diagnosis and
7
8
Ouwerkerk et al.
JACC: HEART FAILURE VOL.
Management of Heart Failure in Adults: a report of the American College of Cardiology Foundation/ American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol 2009;53:e1–90. 6. Lip GY, Nieuwlaat R, Pisters R, Lane DA, Crijns HJ. Refining clinical risk stratification for predicting stroke and thromboembolism in atrial fibrillation using a novel risk factor-based
mortality rates in elderly heart failure patients with a fatal heart failure admission. Congest Heart Fail 2010;16:196–201. 14. Kalogeropoulos AP, Georgiopoulou VV, Giamouzis G, et al. Utility of the Seattle Heart Failure Model in patients with advanced heart failure. J Am Coll Cardiol 2009;53:334–42.
approach: the Euro Heart Survey on Atrial Fibrillation. Chest 2010;137:263–72.
15. Hartigan JA. Clustering Algorithms. New York,
7. Antman EM, Cohen M, Bernink PJ, et al. The TIMI risk score for unstable angina/non-ST eleva-
16. Everitt BS, Landau S, Leese M, Stahl D. Cluster Analysis. 5th ed. New York: Halsted, 1980.
tion MI: a method for prognostication and therapeutic decision making. JAMA 2000;284:835–42.
17. Ploner A. Heatplus: Heatmaps with row and/or
8. Steyerberg EW. Applications of prediction models. In: Clinical Prediction Models: A Practical Approach to Development Validation and Updating. New York, NY: Springer, 2009:500. 9. Ross JS, Mulvey GK, Stauffer B, et al. Statistical models and patient predictors of readmission for heart failure: a systemic review. Arch Intern Med 2008;168:1371–86.
NY: Wiley, 1975:365.
column covariates and colored clusters. 2012; R package version 2.6.0. Available at: http://www. bioconductor.org/packages/release/bioc/html/ Heatplus.html. 18. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods 2010;1:97–111. 19. Shah S, Vanclay F, Cooper B. Improving the
10. Giamouzis G, Kalogeropoulos A, Georgiopoulou V, et al. Hospitalization epidemic in patients with heart failure: risk factors, risk prediction, knowledge gaps, and future directions. J Card Fail 2011; 17:54–75. 11. Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission a systematic review. JAMA 2011;306:1688–98. 12. Betihavas V,
Davidson PM,
Newton PJ,
Frost SA, Macdonald PS, Stewart S. What are the factors in risk prediction models for rehospitalisation for adults with chronic heart failure? Aust Crit Care 2012;25:31–40. 13. Nutter AL, Tanawuttiwat T, Silver MA. Evaluation of 6 prognostic models used to calculate
-, NO. -, 2014 - 2014:-–-
Discriminating Power of Models Predicting Outcome in HF
sensitivity of the Barthel Index for stroke rehabilitation. J Clin Epidemiol 1989;42:703–9.
22. Tabak YP, Johannes RS, Silber JH. Using automated clinical data for risk adjustment: development and validation of six diseasespecific mortality predictive models for pay-forperformance. Med Care 2007;45:789–805. 23. Ky B, French B, McCloskey K, et al. Highsensitivity ST2 for prediction of adverse outcomes in chronic heart failure. Circ Heart Fail 2011;4: 180–7. 24. Ketchum ES, Jacobson AF, Caldwell JH, et al. Selective improvement in Seattle Heart Failure Model risk stratification using iodine-123 metaiodobenzylguanidine imaging. J Nucl Cardiol 2012; 19:1007–16. 25. Wang L, Porter B, Maynard C, et al. Predicting risk of hospitalization or death among patients with heart failure in the Veterans Health Administration. Am J Cardiol 2012;110:1342–9. 26. Selker HP, Griffith JL, D’Agostino RB. A TimeInsensitive Predictive Instrument for Acute Hospital mortality Due to Congestive Heart Failure: Development testing, and use for comparing hospitals: A multicenter study. Med Care 1994;32: 1040–52. 27. Krumholz HM, Wang Y, Mattera JA, et al. An
20. Martín-Sánchez FJ, Gil V, Llorens P, et al., for the Acute Heart Failure Working Group of the Spanish Society of Emergency Medicine Investigation Group. Barthel Index-Enhanced Feedback for Effective Cardiac Treatment (BI-EFFECT) study: contribution of the Barthel Index to the Heart Failure Risk Scoring System model in elderly adults with acute heart failure in the emergency department. J Am Geriatr Soc 2012;60:493–8.
administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with heart failure. Circulation 2006;113:1693–701.
KEY WORDS heart failure, outcome, prognosis, risk factor, risk prediction
21. Yamokoski LM, Hasselblad V, Moser DK, et al. Prediction of rehospitalization and death in severe
A PPE NDI X For supplemental data extraction
heart failure by physicians and nurses of the ESCAPE trial. J Card Fail 2007;13:8–13.
and reduction and statistical analysis information, please see the online version of this paper.
VOL.
ª 2014 BY THE AMERICAN COLLEGE OF CARDIOLOGY FOUNDATION PUBLISHED BY ELSEVIER INC.
-, NO. -, 2014
ISSN 2213-1779/$36.00 http://dx.doi.org/10.1016/j.jchf.2014.04.006
STATE-OF-THE-ART PAPER
Factors Influencing the Predictive Power of Models for Predicting Mortality and/or Heart-Failure Hospitalization in Patients With Heart Failure Wouter Ouwerkerk, MSC,* Adriaan A. Voors, MD, PHD,y Aeilko H. Zwinderman, PHD*
ABSTRACT The present paper systematically reviews and compares existing prediction models in order to establish the strongest variables, models, and model characteristics in patients with heart failure predicting outcome. To improve decision making accurately predicting mortality and heart-failure hospitalization in patients with heart failure can be important for selecting patients with a poorer prognosis or nonresponders to current therapy, to improve decision making. MEDLINE/ PubMed was searched for papers dealing with heart failure prediction models. To identify similar models on the basis of their variables hierarchical cluster analysis was performed. Meta-analysis was used to estimate the mean predictive value of the variables and models; meta-regression was used to find characteristics that explain variation in discriminating values between models. We identified 117 models in 55 papers. These models used 249 different variables. The strongest predictors were blood urea nitrogen and sodium. Four subgroups of models were identified. Mortality was most accurately predicted by prospective registry-type studies using a large number of clinical predictor variables. Mean C-statistic of all models was 0.66 0.0005, with 0.71 0.001, 0.68 0.001 and 0.63 0.001 for models predicting mortality, heart failure hospitalization, or both, respectively. There was no significant difference in discriminating value of models between patients with chronic and acute heart failure. Prediction of mortality and in particular heart failure hospitalization in patients with heart failure remains only moderately successful. The strongest predictors were blood urea nitrogen and sodium. The highest C-statistic values were achieved in a clinical setting, predicting short-term mortality with the use of models derived from prospective cohort/registry studies with a large number of predictor variables. (J Am Coll Cardiol HF 2014;-:-–-) © 2014 by the American College of Cardiology Foundation.
H
eart failure (HF) is a major cause of cardio-
Accurately predicting prognosis can be of benefit
vascular mortality and morbidity (1), and
for patients with heart failure. First, patients with a
its prevalence and incidence is increasing
poorer prognosis might benefit more from aggressive
(2). Despite wider use of evidence-based medical
treatment and a closer follow-up (5). Prediction rules
therapy and preventive device therapy, the prognosis
like the CHA 2DS2-VASc (Congestive heart failure [or
remains poor (3,4).
left ventricular systolic dysfunction]; Hypertension:
From the *Department of Clinical Epidemiology, Biostatistics, and Bioinformatics, Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands; and the yDepartment of Cardiology, University of Groningen, University Medical Center, Groningen, the Netherlands. Prof. Voors received consultancy fees and/or research grants from Alere, Bayer HealthCare, Cardio3Biosciences, Celladon, Novartis, Servier, Torrent, Trevena Vifor Pharmaceuticals; is supported by a grant (#FP7-242209-BIOSTAT-CHF) from the European Commission; and is a clinical established investigator (2006T37) of the Dutch Heart Foundation. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose. Manuscript received December 3, 2013; revised manuscript received April 14, 2014, accepted April 15, 2014.
2
Ouwerkerk et al.
JACC: HEART FAILURE VOL.
-, NO. -, 2014 - 2014:-–-
Discriminating Power of Models Predicting Outcome in HF
ABBREVIATIONS
blood pressure consistently above 140/90
and “(re-)hospitalization.” These terms had to be used
AND ACRONYMS
mm Hg [or treated hypertension on medica-
in either title or abstract. We drew a distinction
tion]; Age $75 years; Diabetes mellitus; pre-
between papers including patients diagnosed with
vious stroke or transient ischemic attack or
CHF and those with ADHF.
ADHF = acute decompensated heart failure
CHF = chronic heart failure HF = heart failure HR = hazard ratio OR = odds ratio
thromboembolism; Vascular disease [e.g., peripheral artery disease, myocardial infarction, aortic plaque]; Age 65 to 74 years; Sex category [i.e., female sex]) and the TIMI (Thrombolysis In Myocardial Infarction) risk
score are widely used in clinical practice to identify high-risk patients who require medical/surgical treatment (6,7). These prediction rules are used to justify medical treatment (like antithrombotic therapy for patients with atrial fibrillation). However, the clinical value of the HF risk predictors remains limited. Second, accurately predicting mortality and morbidity might help patients in their decision making (8). Third, identifying patients who are at risk and do not respond to currently recommended therapies for HF might lead to personalized medicine aimed at targeted treatments for patients with HF. Finally, improved prognostic
risk models may help
in
designing trials by choosing population characteristics with higher event rates. Many studies on predictive markers for outcome in
DATA EXTRACTION. Papers were only included in
the present analysis when the predictor variables were reported and the predicted value was quantified using the C-statistics or receiver-operating characteristic curve value. Of these papers, all reported models were included, even if only variable or C-statistics were available. We collected the predicted outcome variable of each model and grouped these according to the following: 1) mortality; 2) mortality or HF hospitalization; or 3) HF hospitalization. We documented all predictor variables and, when published, their predictive power (odds ratio [OR] and hazard ratio [HR]). We reduced the multitude of different variables to generalized variables. (See the Online Appendix for details on the data extraction, data reduction, and statistical analysis.) For each model—derivation or validation (whether the paper described a new model or validated an existing one)— we collected the following information: Total number of variables in the model
patients with HF and several reviews on prediction
Time of prediction in days (the period for which the
models have been published. Most of these models
model makes its prediction [e.g., in 1-year mortal-
focused on prediction of HF hospitalization and were found to perform poorly or only averagely on a specific patient population (9–12). Nutter et al. (13) evaluated
ity, the time of prediction would be 365 days]) Study design (randomized controlled trial, cohort study or registry)
6 prognostic models predicting mortality. They
Retrospective or prospective data collection
concluded that the prediction models used were
Type of prediction model (subjective prediction, a
adequate in discriminating patients, but they might
risk score– or regression-based model)
underestimate absolute risk of mortality in elderly
The form of statistical analysis used to derive the
patients (14). All these reviews focused on descriptive
model (classification and regression tree analysis,
analyses to explain the predictive power of the models.
Cox proportional hazards regression, generalized
The goal of this paper is to provide an overview of different prediction models developed in the recent years. We compared models with respect to the number, type, and predictive power of predictor
linear model or hierarchical modified Poisson regression) Source of data (medical records or administrative registry data)
variables used. We performed a meta-analysis to
Total number of patients studied in the paper
detect the predictive and discriminating value of
Mean age
variables and models and analyzed which model
Percent of male patients
characteristics were associated with the highest C-statistic value.
STATISTICAL ANALYSIS. First, models were com-
pared with respect to the predictor variables. We
METHODS
therefore performed hierarchical cluster analysis to identify those subgroups of models that were com-
SEARCH STRATEGY. MEDLINE/PubMed was searched
parable (15–17). We counted the models in each sub-
for relevant English-language papers using established
group incorporating more than 5 models. Next, the
methods. Search terms used were: “heart failure,”
predictive weights of the variables for predicting
“chronic heart failure” (CHF), “acute decompensated
mortality
heart failure” (ADHF), “risk scores,” “prediction,”
analyzed, using fixed and random effect models. We
“prognosis,” “models,” “mortality,” “(re-)admission,”
separately analyzed the z-scores, which are the OR/SE
and
HF
hospitalization
were
meta-
JACC: HEART FAILURE VOL.
-, NO. -, 2014
Ouwerkerk et al.
- 2014:-–-
Discriminating Power of Models Predicting Outcome in HF
(OR) and the HR/SE (HR), of the OR and HR of pre-
of the papers there were 10 additional models for
diction variables depending on whether they were
which no C-statistics were provided. These models
obtained from case-control or cohort studies.
were included in the next section comparing models
Third, we estimated the mean C-statistics by fixedand random-effects meta-analysis and used meta-
on their predictor variables, but were excluded from the meta-analysis and meta-regression.
regression to estimate the association between
We excluded 5 models from the variables analysis
study and model characteristics and C-statistics (18).
because these constituted models developed for measuring activities of daily life (19,20) or used only
RESULTS
subjective predictions by the physician and nurse (21).
We identified 117 different models in 55 papers, all
PREDICTORS OF OUTCOME. The prediction models
published between 1994 and 2012 (details are pre-
showed a great variety in number and type of vari-
sented in the Online Appendix). These models were
ables. The numbers varied from 1 (22–24) to 65 (25). A
also divided into CHF and ADHF groups, consisting of
total of 249 different variables were used in 117
10 and 111 models, respectively. Four models were
models. An OR/HR was for 140 variables; no OR/HR,
used in predicting events in both patient groups. In 8
therefore, was mentioned for 109 variables. Most
T A B L E 1 The Most Frequently Used Variables, Along With the Number of Times Used in the Different Models, and Their Predictive Scores
Odds Ratio
Hazard Ratio
n
z-Score
Mean
95% CI
z-Score
Mean
95% CI
Age, yrs
78
17,656
1.04
1.03 to 1.04
20,728
1.06
1.06 to 1.07
Sex
53
15,542
1.12
1.11 to 1.14
10,813
1.22
1.19 to 1.26
Systolic blood pressure
51
31,799
1.30
1.29 to 1.32
19,353
1.16
1.15 to 1.18
Sodium
46
33,502
1.41
1.39 to 1.43
30,343
1.09
1.08 to 1.09
Diabetes
41
15,849
1.14
1.12 to 1.15
21,227
1.44
1.41 to 1.47
Creatinine
38
34,191
1.12
1.11 to 1.13
10,411
1.07
1.05 to 1.08
CHF and ADHF
New York Heart Association functional class
35
15,109
4.14
3.96 to 4.33
17,683
1.41
1.37 to 1.45
Blood urea nitrogen
28
60,698
2.28
2.26 to 2.31
12,495
1.07
1.06 to 1.09
12,843
1.08
1.07 to 1.10
15,346
1.12
1.11 to 1.14
8,526
1.08
1.06 to 1.10
15,327
1.44
1.39 to 1.49
Ejection fraction
23
Hemoglobin
23
(N-terminal pro) B-type natriuretic peptide
23
CHF Age, yrs
75
17,566
1.04
1.03 to 1.04
20,650
1.06
1.06 to 1.07
Sex
53
15,542
1.12
1.11 to 1.14
10,813
1.22
1.19 to 1.26
Systolic blood pressure
46
54,716
1.16
1.15 to 1.16
18,647
1.18
1.16 to 1.19
Sodium
44
33,502
1.41
1.39 to 1.43
36,527
1.07
1.06 to 1.07
Diabetes
41
15,849
1.14
1.12 to 1.15
21,227
1.44
1.41 to 1.47
Creatinine
37
34,191
1.12
1.11 to 1.13
10,411
1.07
1.05 to 1.08
New York Heart Association functional class
34
15,109
4.14
3.96 to 4.33
17,506
1.41
1.37 to 1.44
Blood urea nitrogen
25
61,497
2.37
2.34 to 2.39
12,014
1.07
1.06 to 1.08
Ejection fraction
23
12,843
1.08
1.07 to 1.10
(N-terminal pro) B-type natriuretic peptide
23
27,196
1.11
1.1 to 1.12
Etiology
22
6,795
1.23
1.17 to 1.29 -0.01 to 2.57
ADHF Systolic blood pressure
7
2,181
1.22
1.04 to 1.41
0.378
1.28
Age, yrs
6
8,454
1.46
1.37 to 1.55
0.254
1.27
-0.57 to 3.11
Blood urea nitrogen
4
0.250
1.26
-0.55 to 3.07
0.339
1.33
-0.32 to 2.98
Heart failure admissions
4
12,443
1.44
1.39 to 1.5
0.737
2.00
Sodium
4
0.530
1.53
-0.04 to 3.1
1,663
1.33
Dementia/Alzheimer disease or senility
3
6,306
2.42
2.15 to 2.70
Cancer
2
0.827
1.86
0.39 to 3.33
Cerebrovascular disease
2
0.410
1.43
-0.28 to 3.14
Creatinine
2
Sex
2
0.636
1.63
0.13 to 3.13
0.438
1.33
ADHF ¼ acute decompensated heart failure; CHF ¼ chronic heart failure; CI ¼ confidence interval.
0.99 to 1.67
3
4
Ouwerkerk et al.
JACC: HEART FAILURE VOL.
T A B L E 2 The 10 Variables With the Highest z-Score Used >5 in Models
n
z-Score
-, NO. -, 2014 - 2014:-–-
Discriminating Power of Models Predicting Outcome in HF
Mean
95% CI
10 highest standardized OR
different models, and their predictive power, are shown in Table 1. Table 2 summarizes the value of the strongest predictor variables used most frequently measured by
Blood urea nitrogen
28
60.698
2.28
2.26-2.31
Cancer
10
37,997
1.84
1.81-1.88
Troponin
8
34,585
1.73
1.69-1.76
the z-score of OR and HR. The predictive value of predictor variables for CHF; ADHF; and for mortality, mortality or HF hospitalization, or HF hospitalization
Creatinine
38
34,191
1.12
1.11-1.13
Sodium
46
33,502
1.41
1.39-1.43
Systolic blood pressure
51
31,799
1.3
1.29-1.32
The most frequently used variables with the high-
Heart failure
9
27,924
1.28
1.26-1.30
est predictive values were blood urea nitrogen and
models are presented in the Online Appendix.
Arterial pH
6
27,753
1.87
1.83-1.92
sodium. There were 3 variables with a high predictive
Diastolic blood pressure
11
22,013
1.15
1.14-1.16
value of both OR and HR: sodium; blood urea nitro-
Renal failure
8
21,113
1.26
1.24-1.28
Sodium
46
30.343
1.09
1.08-1.09
Race
10
27,027
1.11
1.1-1.12
control studies, but not in prognostic cohort studies
Diabetes
41
21,227
1.44
1.41-1.47
(high OR, but low or no HR). The opposite situation
Age, yrs
77
20,728
1.06
1.06-1.07
was seen with ejection fraction and (N-terminal pro)
Systolic blood pressure
51
19,353
1.16
1.15-1.18
B-type natriuretic peptide, which were found to be
New York Heart Association functional class
35
17,683
1.41
1.37-1.45
highly prognostic in cohort studies but not in case-
(N-terminal pro) B-type natriuretic peptide
23
15,327
1.44
1.39-1.49
Angiotensin-converting enzyme inhibitor/angiotensin II receptor blockers
8
14,008
1.19
1.17-1.21
10 highest standardized HR
gen; and systolic blood pressure. Cancer, arterial pH, and renal failure were highly predictive in case-
control studies. In the ADHF models, the strongest predictor variable was HF admissions. Hierarchical clustering was performed on the prediction variables of the various models to identify
Ejection fraction
23
12,843
1.08
1.07-1.1
subgroups of similar models. The subgroups created
Blood urea nitrogen
28
12,495
1.07
1.06-1.09
by the hierarchical clustering are presented in Figure 1. There were 4 subgroups of models that
CI ¼ confidence interval; HR ¼ hazard ratio; OR ¼ odds ratio.
incorporated more than 5 models. The largest subgroup (purple) consisted of models using relatively few predictor variables. Variables used vary between
models used a combination of demographic, clinical,
these models; age, sodium, and systolic blood pres-
and easily obtainable data to achieve the highest
sure were the 3 most common variables used.
predictive power. The most frequently used vari-
A second subgroup (red) consisted of models on the
ables, along with the number of times used in the
basis of the Seattle Heart Failure Model. These models used the Seattle Heart Failure Model with addition of 1 or 2 predictor variables. The green subgroup contained
models
using
medication
(beta-blocking
agents), glomerular filtration rate, left ventricular ejection fraction, and New York Heart Association Class as predictor variables. The blue subgroup contained models that used such clinical variables as renal failure, weight, blood pressure, and body mass index, as distinct prediction variables different from the green and the other clusters. META-ANALYSIS OF THE PREDICTIVE VALUE FOR MORTALITY AND HF HOSPITALIZATION. There were
260 C-statistic values reported for 103 models: 181 comprised models predicting mortality; 32 were for HF hospitalization; and 47 for mortality or HF hospitalization. F I G U R E 1 Hierarchical Clustering Dendrogram
The models were ordered according to their similarity as a result of hierar-
There were more C-statistic values reported than there were models; 14 models were validated more
chical clustering. There were 4 subgroups (red, blue, green, and purple)
than once (up to 15). Eight models to predict mor-
found incorporated more than 5 models.
tality were also validated in predicting HF hospitalization. Although models were validated multiple
JACC: HEART FAILURE VOL.
-, NO. -, 2014
Ouwerkerk et al.
- 2014:-–-
Discriminating Power of Models Predicting Outcome in HF
times, and 79% of the C-statistic values were the
had larger weights in the overall mean. This resulted
result of derivation models, 69 models were as yet not
in the mean of 0.71 0.0010, whereas the raw un-
validated in separate patient cohorts.
corrected C-index mean was 0.74.
The highest C-statistic value (0.9) was achieved by
There
was
a
highly
significant
difference
Selker et al. (26) who used multivariable logistic
(p < 0.0001) in C-statistics for models predicting
regression to predict in-hospital mortality in patients
mortality, mortality or HF hospitalization, and HF
with CHF. The lowest C-statistic value (0.52) was
hospitalization: 0.71 0.001, 0.63 0.001, and 0.68
found in the multivariate model of Yamokoski et al.
0.001.
(21) (model 41 (21)) predicting 6-month HF hospitali-
The green cluster, with mean C-statistic values
zation. This value was lower than the C-statistic
of 0.83 0.003, had the highest predictive power of
values of the subjective prediction for HF hospitali-
all 4 clusters (blue: 0.82 0.018, red: 0.74 0.004,
zation by physicians and nurses (0.579 and 0.566,
and purple: 0.61 0.001). META-REGRESSION. Table 3 shows the results of
respectively) in the same paper. In Figure 2, we illustrate the C-statistics of models
multivariable meta-regression analysis of the relation
that reported a standard error or confidence interval.
between C-statistic and model/study characteristics.
The mean C-statistic, the black triangle in Figure 2,
Models predicting mortality had significantly higher
was 0.66 0.0005 for the models reporting a standard
C-statistic values than models predicting HF hospi-
error. Standard errors were reported in 3 models (17,
talization (D C-statistic ¼ 0.03, SE ¼ 0.001).
105, and 111) predicting mortality in ADHF. These
Papers describing the derivation of models re-
models had a mean C-statistic value of 0.71 0.0154,
ported lower C-statistic values than papers vali-
which was not statistically different from the mean
dating results ( D C-statistic ¼ -0.01, SE ¼ 0.0084).
statistic value for predicting mortality in CHF
Prospective cohort/registry studies yielded models
patients (0.71 0.0010). Keep in mind that these
with higher C-statistic values than models on the basis of data of randomized trials ( D C-statistic ¼
model, weighted for the SE-squared. All mortality
0.03, SE ¼ 0.037, DC-statistic ¼ 0.11, SE ¼ 0.042) or
models with a C-statistic value of |t|)
0.7035
0.1676
4.20
0.0000
Mortality
0.0296
0.0091
3.27
0.0012
181
Mortality or readmission
-0.0165
0.0106
-1.56
0.1207
47
in the characteristics of prediction models. The risk
0
-
-
-
32
factors found in these reviews (Ross et al. [9], Gia-
Validation
0
-
-
-
115
similar to the predictor variables found in this paper.
Derivation
-0.0095
0.0084
-1.14
0.2565
145
Ross et al. (9), Giamouzis et al. (10), and Betihavas
Number of variables
0.0036
0.0005
7.71
0.0000
256
et al. (12) also found that the C-statistic values of
Time of prediction
-0.0001
0.0000
-1.42
0.0003
260
(Intercept)
n
Outcome
Readmission
hospital HF hospitalization and compared differences
mouzis et al. [10], and Betihavas et al. [12]) were
Derivation or validation study
Study design Randomized controlled trial
prediction variables to the model. The previously published reviews focused only on
models predicting mortality was higher than that of models predicting HF hospitalization. They suggest
0
-
-
-
50
Cohort
0.0337
0.0374
0.90
0.3682
119
that either important predictors of HF hospitalization
Registry
0.1145
0.0422
2.71
0.0072
91
are lacking in the relevant models or that non-
0
-
-
-
124
talization risk. Developing a model using a systems
0.1025
0.0312
3.29
0.0012
136
biology approach, by incorporating information from
0
-
-
-
4
Point-based additive risk score
-0.0717
0.0376
-1.91
0.0579
44
Regression model
-0.0174
0.0321
-0.54
0.5885
209
medical factors may play a larger role in HF hospi-
Prospective or retrospective study Retrospective Prospective Type prediction model Subjective prediction
demographic, biomarker, genomic, proteomic, and the initial response to therapy might create a more effective prediction model and hopefully aid in understanding HF prognosis, as described by Giamouzis
Statistical method Generalized linear model
0
-
-
-
17
et al. (10). An additional advantage of this approach is
Cox proportional hazards
0.1078
0.0131
8.20
0.0000
134
that it will at least identify patients with a poor
Hierarchical modified Poisson
0.0940
0.0905
1.04
0.3000
4
outcome on currently recommended therapy, which
Multivariate logistic regression
0.1399
0.0205
6.82
0.0000
98
might lead to further development of targeted ther-
0
-
-
-
185
-0.1358
0.0198
-6.85
0.0000
75
-0.0000
0.0000
-3.54
0.0005
260
outcome for patients with HF. Although mortality models had the best discrimi-
Data source Medical records Claims data Sample size
apies,
eventually
leading
to
improvements
in
Age, yrs
-0.0013
0.0017
-0.78
0.4384
252
nating values, these models also have important
Male
-0.0010
0.0005
-1.92
0.0557
258
limitations. Nutter et al. (13) demonstrated that the
0
-
-
-
243
elderly cohort at or approaching the end of life.
-0.0014
0.0127
-0.11
0.9104
17
Nutter et al. (13) compared mortality prediction from
models underestimated the mortality risk in an
Patients’ diagnoses CHF ADHF Abbreviations as in Table 1.
6 prediction models in a retrospective cohort with a mean age of 82.7 8.2 and in-hospital death of 28.8%. The differences in predicted 1-year mortality between
Models using data from medical records had significantly better C-statistic values than models
the models was very high; the predicted mortality varied between 11.1 8.5% and 55.3 17.6% (13).
using claims data. Also models using more predictor
In addition to previous reviews (9,11), which
variables had better predictive values; C-statistic
enumerate the variables in prediction models, we
increased 0.0036 (SE ¼ 0.0005) with each added
quantified the predictive capabilities of each variable.
predictor variable.
We also explained variations in C-statistic values by
There was no significant difference in C-statistic values between patients diagnosed with either CHF or ADHF.
DISCUSSION
meta-analyzing model characteristics mentioned in the published reviews. PREDICTORS OF OUTCOME. The 10 models predict-
ing events in ADHF patients are not grouped into 1 subgroup in the dendrogram, as might be expected in
The present review shows that risk prediction in pa-
the event of the use of identical variables, but are
tients with CHF remains difficult. The best predictors
spread throughout the entire dendrogram. Similar to
of outcome were sodium and blood urea nitrogen. In
models developed for CHF patients, the models
addition, some characteristics of the prediction
developed for ADHF patients are inconclusive on
JACC: HEART FAILURE VOL.
-, NO. -, 2014
Ouwerkerk et al.
- 2014:-–-
Discriminating Power of Models Predicting Outcome in HF
which variables to use. There is no consensus as to
These models want to measure quality-of-care in
which variables are to be used to achieve the highest
contrast to patient disease prognosis. Krumholtz
predictive values.
et al. (27) shows that using claims data has its limi-
DISCRIMINATIVE POWER OF MODELS. Sixty-nine
tations but can be used to compare hospital-specific
derivation models were not yet validated. These
rehospitalization rates. Despite the reduced discrim-
models probably overestimate the prediction capa-
inating power, the risk of rehospitalization may be
bilities. Most of these models used internal valida-
more dependent on quality-of-care and system char-
tion, utilizing a bootstrap method for validation.
acteristics. It is important, therefore, to keep the
This, however, does not account for varying patient
objective in mind when creating a prediction model.
populations.
STUDY LIMITATIONS. Our meta-analysis was limited
In the meta-analysis, we only used the C-statistics
to published data only. From the 113 prediction
values of the models reporting standard errors. This
models found only 103 reported C-statistics, of which
might result in an underestimation of the mean
only 50 models in turn incorporated standard errors
values in the analysis; the raw mean C-statistic value
with their C-statistic values.
was, after all, higher than the mean C-statistic in the
We could not include all variations mentioned by
meta-analysis. Nevertheless, mean C-statistics indi-
Ross et al. (9) and Giamouzis et al. (10) in the meta-
cate only moderate predictive capacity.
regression. These variations might result in higher
PREDICTION MODELS. It is less difficult to predict
DC-statistic values than variations currently included
mortality, which had significantly higher C-statistic
in the meta-regression.
values, than to predict HF hospitalization. As expected, models developed in a derivation set reported
CONCLUSIONS
higher C-statistics than papers validating these models in a different patient population. In addition,
There are still difficulties associated with predicting
we found that cohort and prospective studies pro-
mortality and/or HF hospitalization in HF patients
duced higher C-statistics than models on the basis of
Prediction models need to be improved before they
data of randomized trials or retrospective data. Ran-
can be helpful to physicians and patients. Devel-
domized controlled trial studies had lower C-statistics
oping a model using a systems biology approach,
because randomized controlled trials were not pri-
incorporating information from demographic, bio-
marily created for model development, and the pop-
marker, genomic, proteomic, and initial responses to
ulation is more homogeneous (with therefore less
therapy might create a more effective model. An
discriminating capacity), relatively healthy (less
additional advantage of this approach is that it may
comorbidities), and highly controlled.
serve to identify patients with a poor outcome with
Models using data from medical records had
currently recommended therapy, thereby leading to
significantly better C-statistic values than models
the further development of targeted therapies and
using claims data did. This would suggest that pre-
eventually to improvements in outcome for patients
diction models are most accurate when created with
with HF.
data from patients followed prospectively in a cohort study using data from medical records. Models pre-
REPRINT REQUESTS AND CORRESPONDENCE: Dr.
dicting rehospitalization and mortality rates, how-
Wouter Ouwerkerk, Department of Clinical Epidemi-
ever, often use claims data instead of data from
ology, Biostatistics and Bioinformatics, Academic
medical records. Models predicting rehospitalization
Medical Center, P.O. 22660, room J1B-207, Meiberg-
and mortality rates are developed for purposes
dreef 9, 1105 AZ Amsterdam, the Netherlands. E-mail:
different from those for predicting disease prognosis.
[email protected].
REFERENCES 1. McCullough PA, Philbin EF, Spertus JA, et al., for the REACH Study Investigators. Confirmation of a
Force for the Diagnosis and Treatment of Acute and Chronic Heart Failure 2012 of the European
4. Thom T, Haase N, Rosamond W, et al. Heart disease and stroke statistics–2006 update: a
heart failure epidemic: findings from the Resource Utilization Among Congestive Heart Failure (REACH) study. J Am Coll Cardiol 2002;39:60–9.
Society of Cardiology. Eur Heart J 2012;33: 1787–847.
report from the American Heart Association Statistics Committee and Stroke Statistics Subcommittee. Circulation 2006;113:e85–151.
2. McMurray JJ, Adamopoulos S, Anker SD, et al. ESC guidelines for the diagnosis and treatment of acute and chronic heart failure 2012: the Task
3. Roger VL, Weston SA, Redfield MM, et al. Trends in heart failure incidence and survival in a community-based population. JAMA 2004;292: 344–50.
5. Hunt SA, Abraham WT, Chin MH, et al. 2009 focused update incorporated into the ACC/AHA 2005 Guidelines for the Diagnosis and
7
8
Ouwerkerk et al.
JACC: HEART FAILURE VOL.
Management of Heart Failure in Adults: a report of the American College of Cardiology Foundation/ American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol 2009;53:e1–90. 6. Lip GY, Nieuwlaat R, Pisters R, Lane DA, Crijns HJ. Refining clinical risk stratification for predicting stroke and thromboembolism in atrial fibrillation using a novel risk factor-based
mortality rates in elderly heart failure patients with a fatal heart failure admission. Congest Heart Fail 2010;16:196–201. 14. Kalogeropoulos AP, Georgiopoulou VV, Giamouzis G, et al. Utility of the Seattle Heart Failure Model in patients with advanced heart failure. J Am Coll Cardiol 2009;53:334–42.
approach: the Euro Heart Survey on Atrial Fibrillation. Chest 2010;137:263–72.
15. Hartigan JA. Clustering Algorithms. New York,
7. Antman EM, Cohen M, Bernink PJ, et al. The TIMI risk score for unstable angina/non-ST eleva-
16. Everitt BS, Landau S, Leese M, Stahl D. Cluster Analysis. 5th ed. New York: Halsted, 1980.
tion MI: a method for prognostication and therapeutic decision making. JAMA 2000;284:835–42.
17. Ploner A. Heatplus: Heatmaps with row and/or
8. Steyerberg EW. Applications of prediction models. In: Clinical Prediction Models: A Practical Approach to Development Validation and Updating. New York, NY: Springer, 2009:500. 9. Ross JS, Mulvey GK, Stauffer B, et al. Statistical models and patient predictors of readmission for heart failure: a systemic review. Arch Intern Med 2008;168:1371–86.
NY: Wiley, 1975:365.
column covariates and colored clusters. 2012; R package version 2.6.0. Available at: http://www. bioconductor.org/packages/release/bioc/html/ Heatplus.html. 18. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods 2010;1:97–111. 19. Shah S, Vanclay F, Cooper B. Improving the
10. Giamouzis G, Kalogeropoulos A, Georgiopoulou V, et al. Hospitalization epidemic in patients with heart failure: risk factors, risk prediction, knowledge gaps, and future directions. J Card Fail 2011; 17:54–75. 11. Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission a systematic review. JAMA 2011;306:1688–98. 12. Betihavas V,
Davidson PM,
Newton PJ,
Frost SA, Macdonald PS, Stewart S. What are the factors in risk prediction models for rehospitalisation for adults with chronic heart failure? Aust Crit Care 2012;25:31–40. 13. Nutter AL, Tanawuttiwat T, Silver MA. Evaluation of 6 prognostic models used to calculate
-, NO. -, 2014 - 2014:-–-
Discriminating Power of Models Predicting Outcome in HF
sensitivity of the Barthel Index for stroke rehabilitation. J Clin Epidemiol 1989;42:703–9.
22. Tabak YP, Johannes RS, Silber JH. Using automated clinical data for risk adjustment: development and validation of six diseasespecific mortality predictive models for pay-forperformance. Med Care 2007;45:789–805. 23. Ky B, French B, McCloskey K, et al. Highsensitivity ST2 for prediction of adverse outcomes in chronic heart failure. Circ Heart Fail 2011;4: 180–7. 24. Ketchum ES, Jacobson AF, Caldwell JH, et al. Selective improvement in Seattle Heart Failure Model risk stratification using iodine-123 metaiodobenzylguanidine imaging. J Nucl Cardiol 2012; 19:1007–16. 25. Wang L, Porter B, Maynard C, et al. Predicting risk of hospitalization or death among patients with heart failure in the Veterans Health Administration. Am J Cardiol 2012;110:1342–9. 26. Selker HP, Griffith JL, D’Agostino RB. A TimeInsensitive Predictive Instrument for Acute Hospital mortality Due to Congestive Heart Failure: Development testing, and use for comparing hospitals: A multicenter study. Med Care 1994;32: 1040–52. 27. Krumholz HM, Wang Y, Mattera JA, et al. An
20. Martín-Sánchez FJ, Gil V, Llorens P, et al., for the Acute Heart Failure Working Group of the Spanish Society of Emergency Medicine Investigation Group. Barthel Index-Enhanced Feedback for Effective Cardiac Treatment (BI-EFFECT) study: contribution of the Barthel Index to the Heart Failure Risk Scoring System model in elderly adults with acute heart failure in the emergency department. J Am Geriatr Soc 2012;60:493–8.
administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with heart failure. Circulation 2006;113:1693–701.
KEY WORDS heart failure, outcome, prognosis, risk factor, risk prediction
21. Yamokoski LM, Hasselblad V, Moser DK, et al. Prediction of rehospitalization and death in severe
A PPE NDI X For supplemental data extraction
heart failure by physicians and nurses of the ESCAPE trial. J Card Fail 2007;13:8–13.
and reduction and statistical analysis information, please see the online version of this paper.
