HHS Public Access Author manuscript Author Manuscript
Circ Heart Fail. Author manuscript; available in PMC 2017 August 01. Published in final edited form as: Circ Heart Fail. 2016 August ; 9(8): . doi:10.1161/CIRCHEARTFAILURE.115.002912.
Validation and Comparison of Seven Mortality Prediction Models for Hospitalized Patients With Acute Decompensated Heart Failure
Author Manuscript
Tara Lagu, MD, MPH1,2,3, Penelope S. Pekow, PhD1,4, Meng-Shiou Shieh, PhD1, Mihaela Stefan, MD, PhD1,2,3, Quinn R. Pack, MD, MSc1,3,5, Mohammed Amin Kashef, MD3,5, Auras R. Atreya, MD3,5, Gregory Valania, DO1,3,5, Mara T. Slawsky, MD, PhD1,3,5, and Peter K. Lindenauer, MD, MSc1,2,3 1Center
for Quality of Care Research, Baystate Medical Center, Springfield, MA
2Division
of Hospital Medicine, Department of Medicine, Baystate Medical Center, Springfield, MA
3Department 4School
of Medicine, Tufts University School of Medicine, Boston, MA
of Public Health and Health Sciences, University of Massachusetts-Amherst, Amherst,
MA 5Division
of Cardiology, Baystate Medical Center, Springfield, MA
Abstract Author Manuscript
Background—Heart failure (HF) inpatient mortality prediction models can help clinicians make treatment decisions and researchers conduct observational studies. Published models have not been validated in external populations, however.
Author Manuscript
Methods and Results—We compared the performance of seven models that predict inpatient mortality in patients hospitalized with acute decompensated heart failure (ADHF): Four HFspecific mortality prediction models developed from three clinical databases (Acute Decompensated HF National Registry [ADHERE], Enhanced Feedback for Effective Cardiac Treatment [EFFECT] Study, Get with the Guidelines-HF [GWTG-HF] Registry); two administrative HF mortality prediction models (Premier, Premier+); and a model that uses clinical data but is not specific for HF (Laboratory-Based Acute Physiology Score [LAPS2]). Using a multi-hospital electronic health record-derived (EHR) dataset (HealthFacts [Cerner Corp], 2010– 2012), we identified patients ≥18 years admitted with HF. Of 13,163 eligible patients, median age was 74 years; half were women; and 27% were black. In-hospital mortality was 4.3%. Model predicted mortality ranges varied: Premier+ (0.8–23.1%), LAPS2 (0.7–19.0%), ADHERE (1.2– 17.4%), EFFECT (1.0–12.8%), GWTC-Eapen (1.2–13.8%), and GWTG-Peterson (1.1–12.8%). The LAPS2 and Premier models outperformed the clinical models (c-statistics: LAPS2 0.80 [95% CI: 0.78–0.82], Premier models 0.81 [95% CI: 0.79–0.83]) and 0.76 [95% CI: 0.74–0.78]; clinical models 0.68–0.70). Correspondence to. Dr. Tara Lagu, 280 Chestnut St., 3rd Floor, Center for Quality of Care Research, Springfield, MA 01199, Phone: 413-505-9173, Fax: 413-794-8866, [email protected]. Disclosures None.
Lagu et al.
Page 2
Author Manuscript
Conclusions—Four clinically-derived inpatient HF mortality models exhibited similar performance, with c-statistics near 0.70. Three other models, one developed in EHR data and two developed in administrative data, also were predictive, with c-statistics from 0.76–0.80. Because every model performed acceptably, the decision to use a given model should depend on practical concerns and intended use. Keywords heart failure; mortality prediction; hospitalization
Author Manuscript
Heart failure (HF) is a leading cause of hospital admissions in patients aged 65 years and older.1,2 Patients hospitalized with acute decompensated heart failure (ADHF) have a high risk of mortality, with 30-day mortality rates approaching 10%.3,4 Because risk of mortality varies across patient populations, a mortality prediction model that estimates an individual patient’s risk can be a useful aid for making clinical decisions at the bedside. Additionally, researchers performing comparative effectiveness studies of treatments for ADHF need a validated method of risk adjustment to ensure that differences in outcomes are not simply the result of differences in patient case-mix.
Author Manuscript
Several published mortality prediction models and scoring systems have been developed using clinical data collected for research purposes (e.g., from registries or randomized trials). Each of these was designed with the goal of helping clinicians risk-stratify hospitalized patients with ADHF at the bedside.5–9 However, none of the models were validated in external populations, some are now more than a decade old, and none have been widely adopted in routine clinical settings. It is also not clear how the models perform relative to each other or relative to other risk adjustment methods. We therefore examined the performance of four published clinical HF inpatient mortality prediction models in a dataset derived from the comprehensive electronic health records (EHRs) of more than 50 U.S. hospitals. We compared clinical model performance to three other models used for risk adjustment, one that uses EHR data and two that use administrative data. Since the administrative and EHR models included many more variables, we hypothesized that they would outperform the clinical models. However, we also hypothesized that we would identify that one or more of the clinical models could be useful for risk stratification at the bedside.
Methods Data Source and Patient Population
Author Manuscript
We used HealthFacts (Cerner Corporation), a database that is derived from the EHRs of more than 50 geographically and structurally diverse hospitals throughout the U.S. HealthFacts contains time-stamped pharmacy, laboratory, vital sign (physiologic), and billing information for more than 84 million acute admissions and emergency and ambulatory patient visits.10–13 We first limited the dataset to hospitals that contributed laboratory and vital signs to the database. We then identified a cohort of patients who were 18 years or older, admitted to an included hospital between 1/1/2010 and 12/31/2012, and had a principal International Classification of Diseases, Ninth Revision (ICD-9-CM)
Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 3
Author Manuscript Author Manuscript
diagnosis of HF or a principal diagnosis of respiratory failure with a secondary diagnosis of HF (ICD-9-CM codes for HF: 402.01, 402.11, 402.91, 404.01, 404.03, 404.11, 404.13, 404.91, 404.93, 428.xx; for respiratory failure: 518.81, 518.82, 518.84). To ensure that patients were treated for ADHF during the hospitalization, we restricted the cohort to patients in whom at least one HF therapy (including loop diuretics, metolazone, inotropes, vasodilators, or intra-aortic balloon pump) was initiated within the first two days of hospitalization. We excluded patients who had a length of stay less than 24 hours, lacked vital signs or laboratory data, and patients that were transferred to or from another acute care facility (because we could not accurately determine the onset or subsequent course of their illness). Healthfacts includes demographics (patient age, gender, marital status, insurance status, and race/ethnicity), and we used these variables for some of the models. For models that included comorbid conditions, we used software provided by the Healthcare Costs and Utilization Project of the Agency for Healthcare Research and Quality to identify comorbidities included in the Elixhauser index.14,15 Additionally, for the administrative (Premier) models, we used ICD-9-CM codes to identify other acute conditions that are of concern in the setting of heart failure, including atrial fibrillation (427.3), acute myocardial infarction (410.x1, 410.x2), pneumonia (480–487), malnutrition (263, V77.2), and acute kidney injury (580.4, 580.0, 580.81, 580.89, 580.9,584.5, 584.6, 584.7,584.8, 584.9). The Institutional Review Board at Baystate Medical Center granted permission to conduct the study. Validation Methods by Mortality Prediction Method
Author Manuscript
For each mortality prediction model, we replicated the methods used by the original authors to calculate the predicted mortality for HF patients in the Healthfacts database. In some cases, due to lack of availability of variables or missing data, we had to slightly modify the original methods. These are described in detail in the sections below (see also the Appendix).
Author Manuscript
Premier Models (Administrative Models)—Using administrative billing data, we previously developed a model16,17 that is similar to the heart failure model developed by Krumholz et al. for the Centers for Medicare and Medicaid Services (CMS).18 We developed this model using data from the cost-accounting systems of 433 hospitals that participated in the Premier, Inc. Data Warehouse (PDW, a voluntary, fee-supported database) between January 1, 2009, and June 30, 2011. PDW contains all elements found in hospital claims derived from the uniform billing 04 form (UB-04). In addition, PDW contains an itemized, date-stamped log of all items and services charged to the patient or insurer, including medications, diagnostic and therapeutic services, and laboratory tests. PDW has been used extensively for research purposes.19,20 We used a generalized estimating equation logistic regression model (GEE), clustering on hospital, to predict each patient’s in-hospital mortality. We initially included all clinically relevant variables in the model: variables with a well-established association with mortality (such as age), all conditions listed in the Elixhauser comorbidity index,14 and selected comorbid acute illnesses (described above). Using backward selection, we retained variables in the final model (“The Premier Model”) with p CHF Risk Model [Internet]. [cited 2015 Sep 24] Available from: http:// www.ccort.ca/Research/CHFRiskModel.aspx.
Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 12
Author Manuscript
30. Escobar GJ, LaGuardia JC, Turk BJ, Ragins A, Kipnis P, Draper D. Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med. 2012; 7:388–395. [PubMed: 22447632] 31. Bueno H, Ross JS, Wang Y, Chen J, Vidán MT, Normand S-LT, Curtis JP, Drye EE, Lichtman JH, Keenan PS, Kosiborod M, Krumholz HM. Trends in length of stay and short-term outcomes among Medicare patients hospitalized for heart failure, 1993–2006. JAMA. 2010; 303:2141–2147. [PubMed: 20516414] 32. Minne L, Eslami S, de Keizer N, de Jonge E, de Rooij SE, Abu-Hanna A. Effect of changes over time in the performance of a customized SAPS-II model on the quality of care assessment. Intensive Care Med. 2012; 38:40–46. [PubMed: 22042520] 33. Seymour CW, Liu VX, Iwashyna TJ, Brunkhorst FM, Rea TD, Scherag A, Rubenfeld G, Kahn JM, Shankar-Hari M, Singer M, Deutschman CS, Escobar GJ, Angus DC. Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016; 315:762–774. [PubMed: 26903335]
Author Manuscript Author Manuscript Author Manuscript Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 13
Author Manuscript
Clinical Perspective
Author Manuscript
Heart failure (HF) inpatient mortality prediction models can help physicians to inform patients about the expected outcomes of an acute illness, know the probability of serious adverse events during hospitalization, and make initial management decisions. They are also useful for researchers conducting observational studies. Over the last decade, several mortality prediction models designed to stratify HF patients’ risk during a hospitalization were published, but none has been widely adopted. We compared the performance of seven models that predict inpatient mortality in patients hospitalized with acute decompensated heart failure (ADHF): four models developed from three clinical databases (Acute Decompensated HF National Registry [ADHERE], Enhanced Feedback for Effective Cardiac Treatment [EFFECT] Study, Get with the Guidelines-HF [GWTGHF] Registry), two administrative HF mortality prediction models (Premier, Premier+), and a model that uses clinical data but is not specific for HF (Laboratory-Based Acute Physiology Score [LAPS2]). We found that all models were predictive, with c-statistics ranging from 0.70–0.80. The decision to use a given model should therefore depend on intended use. To use any model in real-time, an online calculator is helpful so that clinicians can more easily incorporate the numerous variables. Of the clinical models, only EFFECT currently has an online calculator. An automated version of the LAPS2 was built into one health system’s EHR, allowing real-time use. Because the Premier models use discharge diagnosis codes, they cannot be used to calculate risk at the time of admission, but, like any of the models, could be used to conduct retrospective observational studies that compare treatment outcomes.
Author Manuscript Author Manuscript Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 14
Author Manuscript Author Manuscript Author Manuscript
Figure 1.
Area Under the Receiver Operating Curves for Seven Mortality Prediction Models with Lines Indicating Specificity at Sensitivity of 0.75
Author Manuscript Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 15
Author Manuscript Author Manuscript Author Manuscript
Figure 2.
Calibration Curves for Seven Mortality Prediction Models
Author Manuscript Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 16
Author Manuscript Author Manuscript Figure 3.
Author Manuscript
Bland-Altman Plots for Seven Mortality Prediction Models
Author Manuscript Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Author Manuscript ✓ ✓
Albumin (g/dL)
Hematocrit (%)
Circ Heart Fail. Author manuscript; available in PMC 2017 August 01. ✓ ✓ ✓ ✓ ✓ ✓ ✓
Lactate (mM/L)
Arterial PaCO2 (mm Hg)
Arterial PaO2 (mm Hg)
Glucose (mg/dL)
Bilirubin (mg/dL)
Troponin (ng/mL)
Estimated Lactate†
Hemoglobin (g/dL)
B-type Natriuretic Peptide (BNP) (pg/mL)
✓
Arterial pH
✓
✓
BUN/Creatinine
✓
✓
Creatinine (mg/dL)
White blood cell count (WBC)
✓
✓
(1000s/mm3)
✓
✓
✓
0.70
Blood urea nitrogen (BUN) (mg/dL)
✓
✓
0.68
EFFECT6
✓
✓
✓
✓
0.69
GWTG-HF Peterson8
✓
✓
✓
✓
✓
✓
✓
✓
0.70
GWTGHF Eapen7
✓ ✓
✓
✓
✓
0.76
Premier16 Without Treatment*
✓
✓
✓
0.81
Premier16 With Treatment*
MORTALITY PREDICTION MODELS
Sodium (mEq/L)
Laboratory Tests
Clinical Characteristics
Admitted to Emergency Department
Insurance ✓
✓
Sex
Race (% Black)
✓
0.80
Age (year)
Demographics
C-statistic
ADHERE5
Author Manuscript LAPS223
Author Manuscript
Variables for Heart Failure Mortality Models
Author Manuscript
Table 1 Lagu et al. Page 17
Author Manuscript
Author Manuscript ✓
✓ ✓ ✓ ✓
Systolic Blood Pressure (SBP) (mm Hg)
Shock Index
Oxygen Saturation (SpO2) (%)
Glasgow Coma Score
✓ ✓
✓ ✓
Liver Disease (%)
Cancer (%)
Formula for Estimated Lactate: (Anion Gap/Bicarbonate) ×100
†
✓
✓
✓
✓
✓
Premier16 Without Treatment*
Contains all Elixhauser comorbidities and other acute conditions not listed here; treatment model includes early critical care treatments (e.g. mechanical ventilation, inotropes), see Table 2
*
✓ Used in the model for ADHERE, EFFECT, GWTG, LAPS2, or Premier
✓
✓
Chronic Obstructive Pulmonary Disease (%)
✓
✓
✓
Dementia (%)
Premier16 With Treatment*
✓
✓
✓
✓
✓
GWTGHF Eapen7
✓ ✓
✓
✓
GWTG-HF Peterson8
Cerebrovascular Disease (%)
Comorbid Conditions
Weight (kg)
✓
✓
Respiratory Rate (breaths/min) ✓
✓
Heart Rate (beats/min)
✓
✓
Temperature (°F)
Vital Signs and Neurologic Score
EFFECT6
MORTALITY PREDICTION MODELS
Author Manuscript ADHERE5
Author Manuscript
LAPS223
Lagu et al. Page 18
Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 19
Table 2
Author Manuscript
Characteristics of Patients with Heart Failure in the HealthFacts Dataset HealthFacts data N (%) 13,163 (100%)
TOTAL Demographics Age, Mean (Median, Q1– Q3)* Female*
71.8 (74, 62–84) 6,775 (51.5)
Race/Ethnicity* White
8,605 (65.4)
Black
3,551 (27.0)
Hispanic
339 (2.6)
Other
668 (5.1)
Author Manuscript
Insurance Payer* Medicare
7,925 (60.2)
Medicaid
1,044 (7.9)
Private
1,257 (9.6)
Uninsured
495 (3.8)
Other/Unknown
2,442 (18.6)
Clinical Characteristics Elixhauser Comorbidities* Valvular Disease
539 (4.1)
Pulmonary Circulation Disease
494 (3.8)
Author Manuscript
Peripheral Vascular Disease
1,376 (10.5)
Hypertension
8,257 (62.7)
Paralysis
233 (1.8)
Other Neurological Disorders
884 (6.7)
Chronic Pulmonary Disease
4,967 (37.7)
Diabetes
5,567 (42.3)
Renal Failure
5,203 (39.5)
Liver Disease
348 (2.6)
Peptic Ulcer Disease with Bleeding Acquired Immunodeficiency Syndrome (AIDS)
3 (0.0) 17 (0.1)
Author Manuscript
Lymphoma
115 (0.9)
Metastatic Cancer
120 (0.9)
Solid Tumor without Metastasis
244 (1.9)
Rheumatoid Arthritis
359 (2.7)
Coagulopathy
784 (6.0)
Obesity
2,495 (19.0)
Weight Loss
538 (4.1)
Fluid and Electrolyte Disorders Chronic Blood Loss Anemia
3,934 (29.9) 107 (0.8)
Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 20
HealthFacts data N (%)
Author Manuscript
Deficiency Anemias
3,804 (28.9)
Alcohol Abuse
335 (2.6)
Drug Abuse
334 (2.5)
Psychoses
306 (2.3)
Depression
1,142 (8.7)
Comorbidity Score
4.6 (4, 3–6)
≤2
2,927 (22.2)
3–4
3,722 (28.3)
5–6
3,834 (29.1)
≥7
2,680 (20.4)
Additional Acute Comorbidities* Acute Myocardial Infarction
588 (4.5)
Author Manuscript
Acute Kidney Injury
2,922 (22.2)
Atrial Fibrillation
4,786 (36.4)
Coronary Artery Disease
6,536 (49.7)
Pneumonia
1,806 (13.7)
Treatment During First 48 Hours† Non-invasive Ventilation
1,220 (9.3)
Invasive Mechanical Ventilation
865 (6.6)
Vasopressors
386 (2.9)
Inotropes
836 (6.4)
Vasodilators
1,039 (7.9)
Prior Admission
Author Manuscript
None
9,525 (72.4)
1
2,246 (17.1)
≥2
1,392 (10.6)
Outcomes Mortality
560 (4.3)
Hospital Characteristics Urban (vs. Rural)
13,136 (99.8)
Teaching
10,414 (79.1)
Geographical Location Midwest
2,318 (17.6)
North
4,893 (37.2)
South
4,492 (34.1)
West
1,460 (11.1)
Author Manuscript
Number of Beds ≤199
1,358 (10.3)
200 – 499
8,295 (63.0)
≥ 500
3,510 (26.7)
*
Included in both Premier models
Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 21
†
Included in the Premier treatment model
Author Manuscript Author Manuscript Author Manuscript Author Manuscript Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 22
Table 3
Author Manuscript
Specificity at a Fixed Sensitivity ROC area
Specificity when Sensitivity=0.75
Premier +16
0.81
0.72
LAPS223
0.80
0.71
Premier16
0.76
0.63
EFFECT6
0.70
0.55
GWTG-HF-Eapen7
0.70
0.52
GWTG-HF-Peterson8
0.69
0.55
ADHERE5
0.68
0.50
Author Manuscript Author Manuscript Author Manuscript Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Circ Heart Fail. Author manuscript; available in PMC 2017 August 01. Published in final edited form as: Circ Heart Fail. 2016 August ; 9(8): . doi:10.1161/CIRCHEARTFAILURE.115.002912.
Validation and Comparison of Seven Mortality Prediction Models for Hospitalized Patients With Acute Decompensated Heart Failure
Author Manuscript
Tara Lagu, MD, MPH1,2,3, Penelope S. Pekow, PhD1,4, Meng-Shiou Shieh, PhD1, Mihaela Stefan, MD, PhD1,2,3, Quinn R. Pack, MD, MSc1,3,5, Mohammed Amin Kashef, MD3,5, Auras R. Atreya, MD3,5, Gregory Valania, DO1,3,5, Mara T. Slawsky, MD, PhD1,3,5, and Peter K. Lindenauer, MD, MSc1,2,3 1Center
for Quality of Care Research, Baystate Medical Center, Springfield, MA
2Division
of Hospital Medicine, Department of Medicine, Baystate Medical Center, Springfield, MA
3Department 4School
of Medicine, Tufts University School of Medicine, Boston, MA
of Public Health and Health Sciences, University of Massachusetts-Amherst, Amherst,
MA 5Division
of Cardiology, Baystate Medical Center, Springfield, MA
Abstract Author Manuscript
Background—Heart failure (HF) inpatient mortality prediction models can help clinicians make treatment decisions and researchers conduct observational studies. Published models have not been validated in external populations, however.
Author Manuscript
Methods and Results—We compared the performance of seven models that predict inpatient mortality in patients hospitalized with acute decompensated heart failure (ADHF): Four HFspecific mortality prediction models developed from three clinical databases (Acute Decompensated HF National Registry [ADHERE], Enhanced Feedback for Effective Cardiac Treatment [EFFECT] Study, Get with the Guidelines-HF [GWTG-HF] Registry); two administrative HF mortality prediction models (Premier, Premier+); and a model that uses clinical data but is not specific for HF (Laboratory-Based Acute Physiology Score [LAPS2]). Using a multi-hospital electronic health record-derived (EHR) dataset (HealthFacts [Cerner Corp], 2010– 2012), we identified patients ≥18 years admitted with HF. Of 13,163 eligible patients, median age was 74 years; half were women; and 27% were black. In-hospital mortality was 4.3%. Model predicted mortality ranges varied: Premier+ (0.8–23.1%), LAPS2 (0.7–19.0%), ADHERE (1.2– 17.4%), EFFECT (1.0–12.8%), GWTC-Eapen (1.2–13.8%), and GWTG-Peterson (1.1–12.8%). The LAPS2 and Premier models outperformed the clinical models (c-statistics: LAPS2 0.80 [95% CI: 0.78–0.82], Premier models 0.81 [95% CI: 0.79–0.83]) and 0.76 [95% CI: 0.74–0.78]; clinical models 0.68–0.70). Correspondence to. Dr. Tara Lagu, 280 Chestnut St., 3rd Floor, Center for Quality of Care Research, Springfield, MA 01199, Phone: 413-505-9173, Fax: 413-794-8866, [email protected]. Disclosures None.
Lagu et al.
Page 2
Author Manuscript
Conclusions—Four clinically-derived inpatient HF mortality models exhibited similar performance, with c-statistics near 0.70. Three other models, one developed in EHR data and two developed in administrative data, also were predictive, with c-statistics from 0.76–0.80. Because every model performed acceptably, the decision to use a given model should depend on practical concerns and intended use. Keywords heart failure; mortality prediction; hospitalization
Author Manuscript
Heart failure (HF) is a leading cause of hospital admissions in patients aged 65 years and older.1,2 Patients hospitalized with acute decompensated heart failure (ADHF) have a high risk of mortality, with 30-day mortality rates approaching 10%.3,4 Because risk of mortality varies across patient populations, a mortality prediction model that estimates an individual patient’s risk can be a useful aid for making clinical decisions at the bedside. Additionally, researchers performing comparative effectiveness studies of treatments for ADHF need a validated method of risk adjustment to ensure that differences in outcomes are not simply the result of differences in patient case-mix.
Author Manuscript
Several published mortality prediction models and scoring systems have been developed using clinical data collected for research purposes (e.g., from registries or randomized trials). Each of these was designed with the goal of helping clinicians risk-stratify hospitalized patients with ADHF at the bedside.5–9 However, none of the models were validated in external populations, some are now more than a decade old, and none have been widely adopted in routine clinical settings. It is also not clear how the models perform relative to each other or relative to other risk adjustment methods. We therefore examined the performance of four published clinical HF inpatient mortality prediction models in a dataset derived from the comprehensive electronic health records (EHRs) of more than 50 U.S. hospitals. We compared clinical model performance to three other models used for risk adjustment, one that uses EHR data and two that use administrative data. Since the administrative and EHR models included many more variables, we hypothesized that they would outperform the clinical models. However, we also hypothesized that we would identify that one or more of the clinical models could be useful for risk stratification at the bedside.
Methods Data Source and Patient Population
Author Manuscript
We used HealthFacts (Cerner Corporation), a database that is derived from the EHRs of more than 50 geographically and structurally diverse hospitals throughout the U.S. HealthFacts contains time-stamped pharmacy, laboratory, vital sign (physiologic), and billing information for more than 84 million acute admissions and emergency and ambulatory patient visits.10–13 We first limited the dataset to hospitals that contributed laboratory and vital signs to the database. We then identified a cohort of patients who were 18 years or older, admitted to an included hospital between 1/1/2010 and 12/31/2012, and had a principal International Classification of Diseases, Ninth Revision (ICD-9-CM)
Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 3
Author Manuscript Author Manuscript
diagnosis of HF or a principal diagnosis of respiratory failure with a secondary diagnosis of HF (ICD-9-CM codes for HF: 402.01, 402.11, 402.91, 404.01, 404.03, 404.11, 404.13, 404.91, 404.93, 428.xx; for respiratory failure: 518.81, 518.82, 518.84). To ensure that patients were treated for ADHF during the hospitalization, we restricted the cohort to patients in whom at least one HF therapy (including loop diuretics, metolazone, inotropes, vasodilators, or intra-aortic balloon pump) was initiated within the first two days of hospitalization. We excluded patients who had a length of stay less than 24 hours, lacked vital signs or laboratory data, and patients that were transferred to or from another acute care facility (because we could not accurately determine the onset or subsequent course of their illness). Healthfacts includes demographics (patient age, gender, marital status, insurance status, and race/ethnicity), and we used these variables for some of the models. For models that included comorbid conditions, we used software provided by the Healthcare Costs and Utilization Project of the Agency for Healthcare Research and Quality to identify comorbidities included in the Elixhauser index.14,15 Additionally, for the administrative (Premier) models, we used ICD-9-CM codes to identify other acute conditions that are of concern in the setting of heart failure, including atrial fibrillation (427.3), acute myocardial infarction (410.x1, 410.x2), pneumonia (480–487), malnutrition (263, V77.2), and acute kidney injury (580.4, 580.0, 580.81, 580.89, 580.9,584.5, 584.6, 584.7,584.8, 584.9). The Institutional Review Board at Baystate Medical Center granted permission to conduct the study. Validation Methods by Mortality Prediction Method
Author Manuscript
For each mortality prediction model, we replicated the methods used by the original authors to calculate the predicted mortality for HF patients in the Healthfacts database. In some cases, due to lack of availability of variables or missing data, we had to slightly modify the original methods. These are described in detail in the sections below (see also the Appendix).
Author Manuscript
Premier Models (Administrative Models)—Using administrative billing data, we previously developed a model16,17 that is similar to the heart failure model developed by Krumholz et al. for the Centers for Medicare and Medicaid Services (CMS).18 We developed this model using data from the cost-accounting systems of 433 hospitals that participated in the Premier, Inc. Data Warehouse (PDW, a voluntary, fee-supported database) between January 1, 2009, and June 30, 2011. PDW contains all elements found in hospital claims derived from the uniform billing 04 form (UB-04). In addition, PDW contains an itemized, date-stamped log of all items and services charged to the patient or insurer, including medications, diagnostic and therapeutic services, and laboratory tests. PDW has been used extensively for research purposes.19,20 We used a generalized estimating equation logistic regression model (GEE), clustering on hospital, to predict each patient’s in-hospital mortality. We initially included all clinically relevant variables in the model: variables with a well-established association with mortality (such as age), all conditions listed in the Elixhauser comorbidity index,14 and selected comorbid acute illnesses (described above). Using backward selection, we retained variables in the final model (“The Premier Model”) with p CHF Risk Model [Internet]. [cited 2015 Sep 24] Available from: http:// www.ccort.ca/Research/CHFRiskModel.aspx.
Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 12
Author Manuscript
30. Escobar GJ, LaGuardia JC, Turk BJ, Ragins A, Kipnis P, Draper D. Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med. 2012; 7:388–395. [PubMed: 22447632] 31. Bueno H, Ross JS, Wang Y, Chen J, Vidán MT, Normand S-LT, Curtis JP, Drye EE, Lichtman JH, Keenan PS, Kosiborod M, Krumholz HM. Trends in length of stay and short-term outcomes among Medicare patients hospitalized for heart failure, 1993–2006. JAMA. 2010; 303:2141–2147. [PubMed: 20516414] 32. Minne L, Eslami S, de Keizer N, de Jonge E, de Rooij SE, Abu-Hanna A. Effect of changes over time in the performance of a customized SAPS-II model on the quality of care assessment. Intensive Care Med. 2012; 38:40–46. [PubMed: 22042520] 33. Seymour CW, Liu VX, Iwashyna TJ, Brunkhorst FM, Rea TD, Scherag A, Rubenfeld G, Kahn JM, Shankar-Hari M, Singer M, Deutschman CS, Escobar GJ, Angus DC. Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016; 315:762–774. [PubMed: 26903335]
Author Manuscript Author Manuscript Author Manuscript Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 13
Author Manuscript
Clinical Perspective
Author Manuscript
Heart failure (HF) inpatient mortality prediction models can help physicians to inform patients about the expected outcomes of an acute illness, know the probability of serious adverse events during hospitalization, and make initial management decisions. They are also useful for researchers conducting observational studies. Over the last decade, several mortality prediction models designed to stratify HF patients’ risk during a hospitalization were published, but none has been widely adopted. We compared the performance of seven models that predict inpatient mortality in patients hospitalized with acute decompensated heart failure (ADHF): four models developed from three clinical databases (Acute Decompensated HF National Registry [ADHERE], Enhanced Feedback for Effective Cardiac Treatment [EFFECT] Study, Get with the Guidelines-HF [GWTGHF] Registry), two administrative HF mortality prediction models (Premier, Premier+), and a model that uses clinical data but is not specific for HF (Laboratory-Based Acute Physiology Score [LAPS2]). We found that all models were predictive, with c-statistics ranging from 0.70–0.80. The decision to use a given model should therefore depend on intended use. To use any model in real-time, an online calculator is helpful so that clinicians can more easily incorporate the numerous variables. Of the clinical models, only EFFECT currently has an online calculator. An automated version of the LAPS2 was built into one health system’s EHR, allowing real-time use. Because the Premier models use discharge diagnosis codes, they cannot be used to calculate risk at the time of admission, but, like any of the models, could be used to conduct retrospective observational studies that compare treatment outcomes.
Author Manuscript Author Manuscript Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 14
Author Manuscript Author Manuscript Author Manuscript
Figure 1.
Area Under the Receiver Operating Curves for Seven Mortality Prediction Models with Lines Indicating Specificity at Sensitivity of 0.75
Author Manuscript Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 15
Author Manuscript Author Manuscript Author Manuscript
Figure 2.
Calibration Curves for Seven Mortality Prediction Models
Author Manuscript Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 16
Author Manuscript Author Manuscript Figure 3.
Author Manuscript
Bland-Altman Plots for Seven Mortality Prediction Models
Author Manuscript Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Author Manuscript ✓ ✓
Albumin (g/dL)
Hematocrit (%)
Circ Heart Fail. Author manuscript; available in PMC 2017 August 01. ✓ ✓ ✓ ✓ ✓ ✓ ✓
Lactate (mM/L)
Arterial PaCO2 (mm Hg)
Arterial PaO2 (mm Hg)
Glucose (mg/dL)
Bilirubin (mg/dL)
Troponin (ng/mL)
Estimated Lactate†
Hemoglobin (g/dL)
B-type Natriuretic Peptide (BNP) (pg/mL)
✓
Arterial pH
✓
✓
BUN/Creatinine
✓
✓
Creatinine (mg/dL)
White blood cell count (WBC)
✓
✓
(1000s/mm3)
✓
✓
✓
0.70
Blood urea nitrogen (BUN) (mg/dL)
✓
✓
0.68
EFFECT6
✓
✓
✓
✓
0.69
GWTG-HF Peterson8
✓
✓
✓
✓
✓
✓
✓
✓
0.70
GWTGHF Eapen7
✓ ✓
✓
✓
✓
0.76
Premier16 Without Treatment*
✓
✓
✓
0.81
Premier16 With Treatment*
MORTALITY PREDICTION MODELS
Sodium (mEq/L)
Laboratory Tests
Clinical Characteristics
Admitted to Emergency Department
Insurance ✓
✓
Sex
Race (% Black)
✓
0.80
Age (year)
Demographics
C-statistic
ADHERE5
Author Manuscript LAPS223
Author Manuscript
Variables for Heart Failure Mortality Models
Author Manuscript
Table 1 Lagu et al. Page 17
Author Manuscript
Author Manuscript ✓
✓ ✓ ✓ ✓
Systolic Blood Pressure (SBP) (mm Hg)
Shock Index
Oxygen Saturation (SpO2) (%)
Glasgow Coma Score
✓ ✓
✓ ✓
Liver Disease (%)
Cancer (%)
Formula for Estimated Lactate: (Anion Gap/Bicarbonate) ×100
†
✓
✓
✓
✓
✓
Premier16 Without Treatment*
Contains all Elixhauser comorbidities and other acute conditions not listed here; treatment model includes early critical care treatments (e.g. mechanical ventilation, inotropes), see Table 2
*
✓ Used in the model for ADHERE, EFFECT, GWTG, LAPS2, or Premier
✓
✓
Chronic Obstructive Pulmonary Disease (%)
✓
✓
✓
Dementia (%)
Premier16 With Treatment*
✓
✓
✓
✓
✓
GWTGHF Eapen7
✓ ✓
✓
✓
GWTG-HF Peterson8
Cerebrovascular Disease (%)
Comorbid Conditions
Weight (kg)
✓
✓
Respiratory Rate (breaths/min) ✓
✓
Heart Rate (beats/min)
✓
✓
Temperature (°F)
Vital Signs and Neurologic Score
EFFECT6
MORTALITY PREDICTION MODELS
Author Manuscript ADHERE5
Author Manuscript
LAPS223
Lagu et al. Page 18
Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 19
Table 2
Author Manuscript
Characteristics of Patients with Heart Failure in the HealthFacts Dataset HealthFacts data N (%) 13,163 (100%)
TOTAL Demographics Age, Mean (Median, Q1– Q3)* Female*
71.8 (74, 62–84) 6,775 (51.5)
Race/Ethnicity* White
8,605 (65.4)
Black
3,551 (27.0)
Hispanic
339 (2.6)
Other
668 (5.1)
Author Manuscript
Insurance Payer* Medicare
7,925 (60.2)
Medicaid
1,044 (7.9)
Private
1,257 (9.6)
Uninsured
495 (3.8)
Other/Unknown
2,442 (18.6)
Clinical Characteristics Elixhauser Comorbidities* Valvular Disease
539 (4.1)
Pulmonary Circulation Disease
494 (3.8)
Author Manuscript
Peripheral Vascular Disease
1,376 (10.5)
Hypertension
8,257 (62.7)
Paralysis
233 (1.8)
Other Neurological Disorders
884 (6.7)
Chronic Pulmonary Disease
4,967 (37.7)
Diabetes
5,567 (42.3)
Renal Failure
5,203 (39.5)
Liver Disease
348 (2.6)
Peptic Ulcer Disease with Bleeding Acquired Immunodeficiency Syndrome (AIDS)
3 (0.0) 17 (0.1)
Author Manuscript
Lymphoma
115 (0.9)
Metastatic Cancer
120 (0.9)
Solid Tumor without Metastasis
244 (1.9)
Rheumatoid Arthritis
359 (2.7)
Coagulopathy
784 (6.0)
Obesity
2,495 (19.0)
Weight Loss
538 (4.1)
Fluid and Electrolyte Disorders Chronic Blood Loss Anemia
3,934 (29.9) 107 (0.8)
Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 20
HealthFacts data N (%)
Author Manuscript
Deficiency Anemias
3,804 (28.9)
Alcohol Abuse
335 (2.6)
Drug Abuse
334 (2.5)
Psychoses
306 (2.3)
Depression
1,142 (8.7)
Comorbidity Score
4.6 (4, 3–6)
≤2
2,927 (22.2)
3–4
3,722 (28.3)
5–6
3,834 (29.1)
≥7
2,680 (20.4)
Additional Acute Comorbidities* Acute Myocardial Infarction
588 (4.5)
Author Manuscript
Acute Kidney Injury
2,922 (22.2)
Atrial Fibrillation
4,786 (36.4)
Coronary Artery Disease
6,536 (49.7)
Pneumonia
1,806 (13.7)
Treatment During First 48 Hours† Non-invasive Ventilation
1,220 (9.3)
Invasive Mechanical Ventilation
865 (6.6)
Vasopressors
386 (2.9)
Inotropes
836 (6.4)
Vasodilators
1,039 (7.9)
Prior Admission
Author Manuscript
None
9,525 (72.4)
1
2,246 (17.1)
≥2
1,392 (10.6)
Outcomes Mortality
560 (4.3)
Hospital Characteristics Urban (vs. Rural)
13,136 (99.8)
Teaching
10,414 (79.1)
Geographical Location Midwest
2,318 (17.6)
North
4,893 (37.2)
South
4,492 (34.1)
West
1,460 (11.1)
Author Manuscript
Number of Beds ≤199
1,358 (10.3)
200 – 499
8,295 (63.0)
≥ 500
3,510 (26.7)
*
Included in both Premier models
Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 21
†
Included in the Premier treatment model
Author Manuscript Author Manuscript Author Manuscript Author Manuscript Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
Lagu et al.
Page 22
Table 3
Author Manuscript
Specificity at a Fixed Sensitivity ROC area
Specificity when Sensitivity=0.75
Premier +16
0.81
0.72
LAPS223
0.80
0.71
Premier16
0.76
0.63
EFFECT6
0.70
0.55
GWTG-HF-Eapen7
0.70
0.52
GWTG-HF-Peterson8
0.69
0.55
ADHERE5
0.68
0.50
Author Manuscript Author Manuscript Author Manuscript Circ Heart Fail. Author manuscript; available in PMC 2017 August 01.
