J Med Syst (2014) 38:29 DOI 10.1007/s10916-014-0029-x
SYSTEMS-LEVEL QUALITY IMPROVEMENT
Classifying Hospitals as Mortality Outliers: Logistic Versus Hierarchical Logistic Models Roxana Alexandrescu & Alex Bottle & Brian Jarman & Paul Aylin
Received: 29 November 2013 / Accepted: 10 March 2014 / Published online: 8 April 2014 # Springer Science+Business Media New York 2014
Abstract The use of hierarchical logistic regression for provider profiling has been recommended due to the clustering of patients within hospitals, but has some associated difficulties. We assess changes in hospital outlier status based on standard logistic versus hierarchical logistic modelling of mortality. The study population consisted of all patients admitted to acute, non-specialist hospitals in England between 2007 and 2011 with a primary diagnosis of acute myocardial infarction, acute cerebrovascular disease or fracture of neck of femur or a primary procedure of coronary artery bypass graft or repair of abdominal aortic aneurysm. We compared standardised mortality ratios (SMRs) from non-hierarchical models with SMRs from hierarchical models, without and with shrinkage estimates of the predicted probabilities (Model 1 and Model 2). The SMRs from standard logistic and hierarchical models were highly statistically significantly correlated (r>0.91, p= 0.01). More outliers were recorded in the standard logistic regression than hierarchical modelling only when using shrinkage estimates (Model 2): 21 hospitals (out of a cumulative number of 565 pairs of hospitals under study) changed from a low outlier and 8 hospitals changed from a high outlier based on the logistic regression to a not-an-outlier based on shrinkage estimates. Both standard logistic and hierarchical modelling have identified nearly the same hospitals as mortality outliers. The choice of methodological approach should, however, also consider whether the modelling aim is
judgment or improvement, as shrinkage may be more appropriate for the former than the latter. Keywords Standardised mortality ratios . Logistic regression . Hierarchical logistic regression
Introduction
This article is part of the Topical Collection on Systems-Level Quality Improvement
There is a growing interest in assessing health care performance and classifying hospitals as mortality outliers. Hospital mortality is typically compared using the ratio of observed to expected deaths–standardised mortality ratio (SMR)–e.g. for all primary diagnoses combined as the HSMR [1–3] or more commonly for diagnosis- or procedure-specific SMRs. Although standard logistic regression is often employed to adjust for case-mix differences between hospitals, in principle one should adjust for the hierarchical structure of the data, with patients clustered within hospitals [4–6] or even additionally within surgeons, as the patients cannot be assumed to be independent of each other [7–10]. Within a previous analysis we found similar results between the two modelling strategies and some implementation issues when using hierarchical modelling to produce an aggregated measure for all primary diagnoses [11]. In this article we focus on SMRs for selected diagnoses and procedure groups with the aim of comparing mortality outlier status for each modelling strategy. For the hierarchical approach, we compare SMRs with and without shrinkage.
R. Alexandrescu : A. Bottle : B. Jarman : P. Aylin Dr. Foster Unit at Imperial College, Department of Primary Care and Public Health, Imperial College London, London EC4Y 8EN, UK
Methods
R. Alexandrescu (*) Department of Palliative Care, Policy and Rehabilitation, King’s College London, London SE5 9PJ, UK e-mail: [email protected]
This study uses Hospital Episode Statistics [12, 13], an administrative data set for all patients admitted to National Hospital Service (public) hospitals in England. It includes
29, Page 2 of 7
socio-demographic information and health care details such as the patient’s primary and secondary diagnosis (International Classification of Diseases, Tenth Revision (ICD-10)), surgical procedure (Office of Population Censuses and Surveys, Classification of Surgical operations and Procedures (OPCS4)), and discharge status (including in-hospital death). For each patient, we assigned a Clinical Classification Software (CCS) [14] group and a corresponding subgroup based on the primary diagnosis, a Charlson Comorbidity Score [15] using information from secondary diagnosis fields and an arealevel socioeconomic deprivation score (Carstairs deprivation quintiles) [16] using the postcode of residence. The basic unit of the dataset was the finished consultant episode, covering the continuous period of time during which a patient was under the care of one consultant. Episodes of care were linked into admissions and admissions ending in transfer to another hospital were linked together to avoid multiple counting. Our study population consisted of patients admitted to acute, non-specialist hospitals in England during financial years 2007/8 through 2011/2 with a primary diagnosis of acute myocardial infarction (AMI) (CCS group 100), Acute cerebrovascular disease (CVD) (CCS group 109) or fracture of neck of femur (fracture NoF) (CCS group 226), or patients having undergone coronary artery bypass graft (CABG) (OPCS4 code K40-46 or repair of abdominal aortic aneurysm (AAA) (OPCS4 code L18-21). We excluded people from the analysis if they had an invalid or missing age, sex or length of stay. We also excluded hospitals that had performed on average less than six operations per year, which were considered more likely to appear as a result of coding error or very atypical centres. On this basis we excluded 38 hospitals providing 50 patients undergoing CABG (0.11 % of total CABG cases) and 15 hospitals providing 89 patients undergoing AAA repair (0.33 % of AAA repair cases). No cases were excluded from the selected CCS groups based on a threshold volume. Of note, due to changing in the coding of AMI subcategories that took place in 2005 [17], all datasets have been restricted to the most recent 5 years, from 2007 to 2011. “Standard” (i.e. non-hierarchical) logistic models were fitted for each of the diagnosis and procedure groups separately. The dependent variable was defined as in-hospital death at any time during a patient’s stay in hospital (for the diagnosis groups) and as in-hospital death within 30 days of procedure (for the procedure groups). Within the surgical groups, we used a 30-day after intervention measure of hospital performance rather than in-hospital mortality because this has been used in other studies modelling surgical procedures, both CABG and AAA repair datasets [18, 19]. The independent variables included in the models were selected based on previous statistical modelling [3]: age, sex, ethnic group, month of admission and year of discharge, comorbidity, deprivation, admission source and method, the number of emergency admissions in the previous 12 months,
J Med Syst (2014) 38:29
palliative care and CCS subgroup (for CVD) or procedure subgroup (for repair of AAA). Except for Charlson comorbidity index, all the candidate predictive variables were made appropriately categorical and put through a backward elimination process with threshold p value for statistically significance of 0.1. We used a less conservative approach, i.e., a cut-off of p
SYSTEMS-LEVEL QUALITY IMPROVEMENT
Classifying Hospitals as Mortality Outliers: Logistic Versus Hierarchical Logistic Models Roxana Alexandrescu & Alex Bottle & Brian Jarman & Paul Aylin
Received: 29 November 2013 / Accepted: 10 March 2014 / Published online: 8 April 2014 # Springer Science+Business Media New York 2014
Abstract The use of hierarchical logistic regression for provider profiling has been recommended due to the clustering of patients within hospitals, but has some associated difficulties. We assess changes in hospital outlier status based on standard logistic versus hierarchical logistic modelling of mortality. The study population consisted of all patients admitted to acute, non-specialist hospitals in England between 2007 and 2011 with a primary diagnosis of acute myocardial infarction, acute cerebrovascular disease or fracture of neck of femur or a primary procedure of coronary artery bypass graft or repair of abdominal aortic aneurysm. We compared standardised mortality ratios (SMRs) from non-hierarchical models with SMRs from hierarchical models, without and with shrinkage estimates of the predicted probabilities (Model 1 and Model 2). The SMRs from standard logistic and hierarchical models were highly statistically significantly correlated (r>0.91, p= 0.01). More outliers were recorded in the standard logistic regression than hierarchical modelling only when using shrinkage estimates (Model 2): 21 hospitals (out of a cumulative number of 565 pairs of hospitals under study) changed from a low outlier and 8 hospitals changed from a high outlier based on the logistic regression to a not-an-outlier based on shrinkage estimates. Both standard logistic and hierarchical modelling have identified nearly the same hospitals as mortality outliers. The choice of methodological approach should, however, also consider whether the modelling aim is
judgment or improvement, as shrinkage may be more appropriate for the former than the latter. Keywords Standardised mortality ratios . Logistic regression . Hierarchical logistic regression
Introduction
This article is part of the Topical Collection on Systems-Level Quality Improvement
There is a growing interest in assessing health care performance and classifying hospitals as mortality outliers. Hospital mortality is typically compared using the ratio of observed to expected deaths–standardised mortality ratio (SMR)–e.g. for all primary diagnoses combined as the HSMR [1–3] or more commonly for diagnosis- or procedure-specific SMRs. Although standard logistic regression is often employed to adjust for case-mix differences between hospitals, in principle one should adjust for the hierarchical structure of the data, with patients clustered within hospitals [4–6] or even additionally within surgeons, as the patients cannot be assumed to be independent of each other [7–10]. Within a previous analysis we found similar results between the two modelling strategies and some implementation issues when using hierarchical modelling to produce an aggregated measure for all primary diagnoses [11]. In this article we focus on SMRs for selected diagnoses and procedure groups with the aim of comparing mortality outlier status for each modelling strategy. For the hierarchical approach, we compare SMRs with and without shrinkage.
R. Alexandrescu : A. Bottle : B. Jarman : P. Aylin Dr. Foster Unit at Imperial College, Department of Primary Care and Public Health, Imperial College London, London EC4Y 8EN, UK
Methods
R. Alexandrescu (*) Department of Palliative Care, Policy and Rehabilitation, King’s College London, London SE5 9PJ, UK e-mail: [email protected]
This study uses Hospital Episode Statistics [12, 13], an administrative data set for all patients admitted to National Hospital Service (public) hospitals in England. It includes
29, Page 2 of 7
socio-demographic information and health care details such as the patient’s primary and secondary diagnosis (International Classification of Diseases, Tenth Revision (ICD-10)), surgical procedure (Office of Population Censuses and Surveys, Classification of Surgical operations and Procedures (OPCS4)), and discharge status (including in-hospital death). For each patient, we assigned a Clinical Classification Software (CCS) [14] group and a corresponding subgroup based on the primary diagnosis, a Charlson Comorbidity Score [15] using information from secondary diagnosis fields and an arealevel socioeconomic deprivation score (Carstairs deprivation quintiles) [16] using the postcode of residence. The basic unit of the dataset was the finished consultant episode, covering the continuous period of time during which a patient was under the care of one consultant. Episodes of care were linked into admissions and admissions ending in transfer to another hospital were linked together to avoid multiple counting. Our study population consisted of patients admitted to acute, non-specialist hospitals in England during financial years 2007/8 through 2011/2 with a primary diagnosis of acute myocardial infarction (AMI) (CCS group 100), Acute cerebrovascular disease (CVD) (CCS group 109) or fracture of neck of femur (fracture NoF) (CCS group 226), or patients having undergone coronary artery bypass graft (CABG) (OPCS4 code K40-46 or repair of abdominal aortic aneurysm (AAA) (OPCS4 code L18-21). We excluded people from the analysis if they had an invalid or missing age, sex or length of stay. We also excluded hospitals that had performed on average less than six operations per year, which were considered more likely to appear as a result of coding error or very atypical centres. On this basis we excluded 38 hospitals providing 50 patients undergoing CABG (0.11 % of total CABG cases) and 15 hospitals providing 89 patients undergoing AAA repair (0.33 % of AAA repair cases). No cases were excluded from the selected CCS groups based on a threshold volume. Of note, due to changing in the coding of AMI subcategories that took place in 2005 [17], all datasets have been restricted to the most recent 5 years, from 2007 to 2011. “Standard” (i.e. non-hierarchical) logistic models were fitted for each of the diagnosis and procedure groups separately. The dependent variable was defined as in-hospital death at any time during a patient’s stay in hospital (for the diagnosis groups) and as in-hospital death within 30 days of procedure (for the procedure groups). Within the surgical groups, we used a 30-day after intervention measure of hospital performance rather than in-hospital mortality because this has been used in other studies modelling surgical procedures, both CABG and AAA repair datasets [18, 19]. The independent variables included in the models were selected based on previous statistical modelling [3]: age, sex, ethnic group, month of admission and year of discharge, comorbidity, deprivation, admission source and method, the number of emergency admissions in the previous 12 months,
J Med Syst (2014) 38:29
palliative care and CCS subgroup (for CVD) or procedure subgroup (for repair of AAA). Except for Charlson comorbidity index, all the candidate predictive variables were made appropriately categorical and put through a backward elimination process with threshold p value for statistically significance of 0.1. We used a less conservative approach, i.e., a cut-off of p
