t h e s u r g e o n 1 3 ( 2 0 1 5 ) 2 1 3 e2 1 7
Available online at www.sciencedirect.com
ScienceDirect The Surgeon, Journal of the Royal Colleges of Surgeons of Edinburgh and Ireland www.thesurgeon.net
Measuring surgical performance: A risky game? F. Kiernan a,*, F. Rahman b a b
Royal College of Surgeons of Ireland, Beaumont Hospital, Dublin 9, Ireland Health and Social Care Information Centre, St George's University Hospital, Tooting, London, UK
article info
abstract
Article history:
Background: Interest in performance measurement has been driven by increased demand
Received 20 October 2014
for better indicators of hospital quality of care. This is due in part to policy makers wishing
Received in revised form
to benchmark standards of care and implement quality improvements, and also by an
17 December 2014
increased demand for transparency and accountability.
Accepted 18 December 2014
Approach: We describe the role of performance measurement, which is not only about
Available online 28 January 2015
quality improvement, but also serves as a guide in allocating resources within health systems, and between health, education, and social welfare systems. As hospital based
Keywords:
healthcare is responsible for the most cost within the healthcare system, and treats the
Mortality
most severely ill of patients, it is no surprise that performance measurement has focused
Performance measurement
attention on hospital based care, and in particular on surgery, as an important means of
Risk-adjustment
improving quality and accountability. We are particularly concerned about the choice of
Gaming
mortality as an outcome measure in surgery, as this choice assumes that all mortality in
Outcomes
surgery is preventable. In reality, as a low quality indicator of care it risks both gaming, and
Reporting
cream-skimming, unless accurate risk adjustment exists. Further concerns relate to the public reporting of this outcome measure. Conclusions: As mortality rates are an imperfect measure of quality, the reputation of individual surgeons will be threatened by the public release of this data. Significant effort should be made to communicate the results to the public in an appropriate manner. © 2015 Royal College of Surgeons of Edinburgh (Scottish charity number SC005317) and Royal College of Surgeons in Ireland. Published by Elsevier Ltd. All rights reserved.
Healthcare is technophobic. Digital technology has the power to collect and analyse data over populations and time, yet the healthcare sector has been slow to follow the lead of others in using this data to improve care. The evolution of ‘big data’ has given non-healthcare sectors the ability to accurately measure and compare performance, both across jurisdictions and over time, while we remain focused on the idea that information technology cannot capture the multi-dimensional aspects of healthcare necessary for accurate comparisons of performance. The evidence suggests otherwise. Frameworks to
analyse both clinical and cost-effectiveness are continually improving, properly designed software can accurately risk adjust, and the results of this data can be immediately delivered to those who require it. In memory-computing has been used in South Korea to decrease unnecessary antibiotic use, leading to lower costs and a decrease in resistant organisms, and Darmouth-Hitchcock have used large level datasets to decrease length of stay for total knee replacement.1 In measuring health performance, technology is no longer an obstacle. Yet using ‘big data’ in healthcare remains
* Corresponding author. E-mail address: [email protected] (F. Kiernan). http://dx.doi.org/10.1016/j.surge.2014.12.001 1479-666X/© 2015 Royal College of Surgeons of Edinburgh (Scottish charity number SC005317) and Royal College of Surgeons in Ireland. Published by Elsevier Ltd. All rights reserved.
214
t h e s u r g e o n 1 3 ( 2 0 1 5 ) 2 1 3 e2 1 7
controversial, in part due to ethical concerns regarding data protection for patients but increasingly because of concerns regarding protection of healthcare practitioners. Delivering value in healthcare is an increasingly popular topic, driven by increased patient demands, expensive technologies, and an awareness of financial restraint, particularly emphasized by the global economic crisis. While the term ‘performance measurement’ appears to have become synonymous with ‘quality improvement’, in reality it also serves as a guide in the allocation of resources between various interventions, and between the healthcare system and education, or the healthcare system and social welfare. Measuring performance is part of a move towards demand-driven rather than supply-driven healthcare, which makes providers accountable for their outcomes, and clarifies where resources are spent. Ultimately performance is to health policy, what evidence-based medicine is to clinical care. The result is that the role of the healthcare provider has become entwined with the role of the manager, the economist, and the accountant. The care provided by clinicians accounts, understandably, for the most significant expenditure within the healthcare system.2 Healthcare demand is influenced by clinicians, and supply of healthcare is delivered by clinicians. Therefore, it is not surprising that the work of clinicians is considered to be particularly relevant in measuring the performance of the health system. However, the health system is more than a collection of doctors, and it is more than the sum of their reported outcomes. It involves preventative care, primary, secondary and tertiary care, as well as education and social welfare. Furthermore, the goals of the health care system are more than just improved health. They also include accountability, financial protection and equity of access. The World Health Report of 2000 marked a move away from examining improvements in medical care, and included equity, fairness of financial contribution, and responsiveness as goals of the healthcare system.3 Indeed, measurement of level of health of the population accounted for only 25% of the overall score. In country analysis of health system performance should take this weighting system into account when designing their own ranking systems. More recent discussions have focused on the importance of directing resources at areas of health gain, rather than measuring and focusing on non-preventable health loss. Focusing on mortality as a health loss, rather than potential health gains from improvements in quality of life, is one of the reasons why mental healthcare does not receive adequate funding in healthcare budgets. Therefore, although hospital-related healthcare accounts for the most costly care, in order to deliver outcomes that truly matter to the population, we need to look beyond merely the hospital environment, and focus on the outcomes that matter to patients. Improving quality of care for the population requires improvements in community-based care, public health, education and employment. Interest in performance measurement in surgery has been driven by an increased demand for better indicators of quality in hospital care. Reporting this data is in part due to an increased emphasis on demonstrating transparency in healthcare provision, and in part due to an awareness of the role of public reporting as a mechanism to improve quality of
care. Berwick's framework for quality improvement shows that public reporting of outcomes leads to improvements in quality from one of two methods e either patients select better providers of care, or the data provides information on areas of underperformance, leading to a stimulus for improvement from the providers.4 However, the assumption that the publication of this data will result in improvements in quality rests on the assumption that the outcome being measured is amenable to quality improvement. Mortality is often used as a marker of quality of healthcare, predominantly because of the ease with which data on mortality can be collected. However, this data should only be used as a marker of quality if mortality was potentially preventable. While surgical mortality may be avoidable in certain situations, and for certain patients, assuming that it is unavoidable for all is contrary to scientific knowledge. ‘What's measured is what matters', and inappropriate indicators of quality may lead to resources being spent in areas of minimal benefit, at the expense of other areas of care. As the focus has now been put on measuring the mortality rates of individual surgeons, it is time to ask if the choice of mortality as an indicator is appropriate, and what unintended negative effects may result from its use. Decades of analyses of the use of mortality as an outcome measurement in hospitals, has clearly demonstrated that the signal-to-noise ratio is too low for mortality to be reliably used as a reflection of quality.5 This may be either because there is no real correlation between in-hospital mortality and quality of hospital care, or because of the small sample size, coding issues, or methodological concerns.4 In reality, in-hospital death is rarely preventable, and a significant body of evidence has shown that in cases where it is, alterations in care usually resulted in delaying time to death, rather than preventing it completely. Results from surgical care in the United States, the initiators of performance measurements, has demonstrated that hospitals with the highest mortality rates were not the hospitals that were considered to have provided the poorest quality of care.5 A comparison of four different methodological approaches to measuring hospital wide mortality for the same dataset demonstrated significant variations in mortality, depending on the inclusion and exclusion criteria chosen and the statistical approach used.6 The move towards reporting the mortality rates of individual surgeons is entirely based on evidence from cardiac surgery. Work from both the US and England demonstrated a positive benefit of reporting individual cardio-thoracic surgeon's mortality rates. Data from the New York State CABG program is often cited as an example of where publishing outcomes leads to improved performance as a trend of decreased CABG related mortality was observed following public reporting.7,8 In England, public reporting of mortality first took place in cardiothoracic surgery after reconfiguration of services following high mortality rates at Bristol Royal Infirmary.9 The specialty initially published hospital level data in 1998 and then individual surgeons' results in 2006. However, while mortality may be an appropriate outcome measure in cardiac surgery, in other specialties it has been shown to be a poor predictor of quality.10 If unexpected deaths are of particular importance, perhaps death in low risk groups may be a more realistic quality indicator than overall
t h e s u r g e o n 1 3 ( 2 0 1 5 ) 2 1 3 e2 1 7
mortality? As colorectal, urological, and orthopaedic surgeons become the next specialties to be scrutinized in the NHS, it is time to question not only whether or not mortality is an appropriate measure, but also how long patients should be followed for. Mortality from colorectal cancer at 30 days after surgery or chemotherapy/radiotherapy is a marker of quality in Scotland, but it is less clear if 30 day mortality should be considered to be a decisive point. In comparisons of mortality after liver transplantation in the United States versus the United Kingdom and Ireland, mortality was higher in the UK and Ireland after 90 days, but much lower at 1 year.11 This should lead us to also question if an individual surgeon should be considered as being solely responsible for the patient care. Both hospital factors and patient factors relate to mortality, yet many hospital factors are not included in risk adjustment. The number of surgical nurses on a ward, and their educational level, is associated with post-operative mortality,12 yet it does not factor in risk adjustment algorithms in the UK, and is, therefore, unlikely to factor in an equivalent Irish model. Hospital management, and how management relates to employees is a further factor in mortality rates,13 and is beyond the control of the surgical team. The NCEPOD of 2010 identified infrastructure as being a cause of mortality, rather than surgical error. Furthermore, publication of individual surgeon's mortality rates diverts public attention away from deficiencies in the health system.14 While the concept of reporting individual outcomes was believed to be an essential part of moving towards a model of accountable care, in reality the term ‘accountable care’ relates to co-ordinated care delivery, with teams of clinicians responsible for the care of a patient. Discussions should include the effect of the reporting of this data on clinician and public behaviour. Dysfunctional behaviour is known to result from the fear and distrust that commonly occurs after the reporting of results. Either the collection of small amounts of high quality data, or the collection of larger amounts of poorer quality data, will lead to the distortion of results. By measuring what is easy to collect, but necessarily related to quality, resources can be misallocated. Attaching a high-powered incentive to this data, either through remuneration or a risk to reputation, will lead to perverse outcomes. 41% of the reduction in New York's riskadjusted mortality has been described as being related to gaming,15 where intentional up-coding has been reported, in order to reduce the observed-to-expected mortality ratio. While gaming has not yet been reported in the collection of surgical mortality in England, data manipulation and gaming have been reported following other NHS policies, including the alteration of hospital consultation data to show an apparent reduction in waiting list times.16 Cream-skimming, or the avoidance of high-risk patients, is a further potential consequence. In cases where providers refuse to accept the highest risk patients, the failure to provide care for these patients may lead to overall higher mortality and higher costs for the population. Further evidence from the New York Cardiac Reporting System revealed that 62% of surveyed cardiac surgeons in New York refused to operate on a patient in the previous year because of public reporting [ref]. Furthermore, in Pennsylvania, 59% of cardiologists had increased difficulty referring high-risk patients to surgeons
215
following public reporting of data.17 While accurate riskadjustment should prevent cream-skimming, attempts to report mortality publicly have led to increasing debate about current risk-adjustment methods, and the degree to which these measures accurately reflect patient safety and quality.18 Different risk adjustment methods provide different pictures of performance, depending on the source of data, the weighting system, and the variables chosen, which has resulted in different rankings of hospitals based on mortality rates.6 Although the P-POSSUM is used as a risk-adjusted surgical audit tool, it has been shown to be inaccurate for both mortality prediction in gynaecological patients19 and in surgical patients in the United States.20 Furthermore, there is evidence of significant discrepancies between the proprietary comorbidity indices (Disease-Staging and APR-DRGs) and nonproprietary indices21 used in risk adjustment. In the midst of these concerns regarding these measures, the need for automated data linkage has been considered as a means of improving data collection.22 Big data, therefore, may have a particular role in accurate risk adjustment, as sociodemographic, physiological and medical information from 4.6 million patients of the Veterans Administration, was shown to enable the accurate prediction of morbidity and mortality.23 The public reporting of surgical outcomes could be seen as a form of reward and punishment, both of which are a flawed and harmful means of attempting to improve quality. A study in this journal in 2013 described the importance of social support and a sense of community at work in job satisfaction and work engagement.24 Public reporting and ranking of surgeons has resulted in discord due to both the ranking of surgeons, and the manner in which the data was collected. Furthermore this discord has resulted in debate that ultimately distracted from the attempted effort to improve quality.25 Engagement with clinical staff is an essential part of producing meaningful performance measures, and a collaborative relationship between clinicians and management, based on mutual trust, is required for long-term progress. Clinicians should be involved in the choice of each performance measure to ensure it is designed according to its intended use, as a measure designed for improvement should not be used for judgment. In addition, professionals who deliver healthcare should not merely have a say in nature of the choice of outcome measure, but also in how it is communicated to the public. The demand for greater accountability in healthcare is appropriate, and patients have a right to know that surgeons operate to a sufficiently high standard. However, the responsibility for ensuring this lies with relevant medical councils and professional bodies, rather than the media. Reporting outcomes of surgical procedures in a way that is fair to both patients and surgeons requires a two-way conversation. The medical profession, patients, politicians, and the media will need to be educated in how to interpret this data. Surgeon level mortality data in England is currently published online via the NHS choices website. The data for different surgical sub-specialties is presented in different formats with varying levels of detail. Although this data is
216
t h e s u r g e o n 1 3 ( 2 0 1 5 ) 2 1 3 e2 1 7
available for patient viewing, a key consideration is whether this data is presented in a way that is comprehensible by most patients. For example, categorizing surgical performance by percentiles may be clear for surgeons but its relevance for patients is debatable. Patients with poor numeracy and literacy skills may find tables of statistics difficult to contextualize and there is increasing interest in exploring new methods of presenting data.5 Surgeon mortality data from Scotland is presented as an Excel spreadsheet, in an unadjusted form, with a short caveat explaining that out of context data may be misleading to patients.26 In addition, there is little evidence that patients use this data in making choices. In studying the effect of publishing clinical effectiveness data on CABG in Pennsylvania, fewer than 1% of patients used the rating of their surgeon and hospital in selecting their healthcare provider.27 Providing data in a league table is a response to the theory that giving people more information allows them to make better choices. However, the reality is that league tables are only of benefit when the initial quality of care is very poor, and that they can improve care when patients are quality elastic. Those who are quality inelastic will not respond to this information, and will remain in low quality care, a standard which may further deteriorate if resource allocation is based on evidence of improvements in quality of care. When the inelastic group are the poor, less well educated, sick, elderly or immigrants, league tables may be harmful to those who need healthcare the most. We would argue that larger sample sizes be used in measuring performance, in order to ensure accuracy. Therefore mortality should be collected at a hospital level, rather than at an individual surgeon level. Attention must be paid to the distinction between data collection for cost analysis, and data collection for assessing quality. An international consensus is required not only for the choice of indicator, but also for risk adjustment measurement. Performance measurement should be treated as a science, with risk adjustment requiring input of our colleagues in health economics and public health medicine to determine how income and socioeconomic status, along with medical comorbidities, will affect outcome. In addition, standards should be set by professional bodies, and outliers on this scale, who have riskadjusted mortality rates higher than expected, should be investigated. Anomalies in expected outcomes can be determined to be either due to a lack of resources or to poor quality, and management of these anomalies should depend on the cause. Accountability and transparency can be ensured, while also protecting the right to privacy of the relevant healthcare professional.
Conclusion Information is central to improving how the health system responds to the needs of the population it serves. Improving health outcomes, delivering equitable care, ensuring financial protection, and providing accountable care to citizens requires high quality data collection and accurate performance measurement. Public reporting of data can regain public trust by demonstrating that the healthcare system is
open to criticism and creates a culture of openness and transparency. However, delivering improvements in the healthcare system requires accurate data collection, based on clinical assessment of need, rather than political decisions. To measure what matters to patients, we need to measure what is preventable. Measures to decrease gaming, and ensure risk adjustment are necessary to prevent perverse outcomes. Furthermore, the data needs to be communicated to those who use it, in a manner that adds clarity rather than confusion. Ensuring that these measures are met requires comprehensive data collection that evaluates the institution, the clinicians, and the patient. Providing care for a surgical patient is multi-faceted and complex, analysing that care is no different. Data collection on mortality may be easy to collect, process and disseminate, but it overlooks system wide deficiencies, and emphasizes health loss over health gain. In the era of ‘big data’, we should harness our ability to obtain accurate information, which can be used to assess outcomes that are potentially preventable, and that matter to patients.
references
1. https://hbr.org/resources/pdfs/comm/sap/18826_HBR_SAP_ Healthcare_Aug_2014.pdf [accessed online 01.12.14]. 2. Dixon J. Reform and the National Health Service. Political Q 2012;83(2):343e52. 3. World Health Report (2000). The World Health Report 2000 e health systems: improving performance http://www.who.int/ whr/2000/en/whr00_en.pdf [accessed online 10.09.14]. 4. Berwick D. Disseminating innovations in health care. JAMA 2003;289(15):1696e975. 5. Lilford R, Pronovost P. Using hospital mortality rates to judge hospital performance: a bad idea that just won't go away. Br Med J 2010;340:955e7. 6. Iezzoni L. The risks of risk adjustment. JAMA 1997; 19;278(19):1600e7. 7. Thomas J, Hofer T. Research evidence on the validity of riskadjusted mortality rate as a measure of hospital quality of care. Med Care Res Rev 1998;4(55):371e404. 8. Chassin MR. Achieving and sustaining improved quality: lessons from New York State and cardiac surgery. Health Aff 2002;21(4):40e51. 9. Hannan EL, Siu AL, Kumar D, Kilburn Jr H, Chassin MR. The decline in coronary artery bypass graft surgery mortality in New York State: the role of surgeon volume. JAMA 1995;273(3):209e13. 10. Keogh B, Dussek J, Watson D, Magee P, Whatley D. Public confidence and cardiac surgical outcome. Br Med J 1998;316:1759e64. 11. Hofer TP, Hayward RA. Identifying poor-quality hospitals. Can hospital mortality rates detect quality problems for medical diagnoses? Med Care 1996; Aug;34(8):737e53. 12. Dawwas MF, Gimson AE, Lewsey JD, Copley LP, van der Meulen JHP, on behalf of the UK and Ireland Liver Transplant Audit. Survival after liver transplantation in the United Kingdom and Ireland compared with the United States. Gut 2007;56:1606e13. 13. Aiken LH, Sloane DM, Bruyneel L, Van den Heede K, et al., for the RN4CAST consortium. Nurse staffing and education and hospital mortality in nine European countries: a retrospective observational study. Lancet 2014; May 24;383(9931):1824e30.
t h e s u r g e o n 1 3 ( 2 0 1 5 ) 2 1 3 e2 1 7
14. West MA, Borrill C, Dawson J. The link between the management of employees and patient mortality in acute hospitals. Int J Hum Resour Manag 2002;13(8):1299e310. 15. Westaby S. Publishing individual surgeons' death rates prompts risk averse behaviour. Br Med J 2014; 12;349:g5026. 16. Green J, Wintfield N. Report cards on cardiac surgeons: assessing New York state's approach. N Engl J Med 1995;332:1229e32. 17. Schneider EC, Epstein AM. Influence of cardiac-surgery performance reports on referral practices and access to care d a survey of cardiovascular specialists' surgery: influence on referral practices and access to care. N. Engl J Med 1996;335(4):251e6. 18. Smith P. On the unintended consequences of publishing performance data in the public sector. Int J Public Adm 1995;18(2e3):277e310. 19. Das N, Talaat AS, Naik R, et al. Risk adjusted surgical audit in gynaecological oncology: P-POSSUM does not predict outcome. Eur J Surg Oncol 2006;32(10):1135e8. 20. Bennett-Guerrero E, Hyam JA, Shaefi S, et al. Comparison of PPOSSUM risk-adjusted mortality rates after surgery between patients in the USA and the UK. Br J Surg 2003;90(12):1593e8.
217
21. Grendar J, Shaheen AA, Myers RP, et al. Predicting in-hospital mortality in patients undergoing complex gastrointestinal surgery: determining the optimal risk adjustment method. Arch Surg 2012;147(2):126e35. 22. IBruce J, Russell EM, Mollison J, Krukowski ZH. The measurement and monitoring of surgical adverse events. Health Technol Assess 2001;5(22):1e194. 23. Wang L, Porter B, Maynard C, Evans G, Bryson C, Sun H, et al. Predicting risk of hospitalization or death among patients receiving primary care in the Veterans Health Administration. Med Care 2013;51:368e73. 24. Mache S, Vitzthum K, Klapp BF, Danzer G. Surgeons' work engagement: influencing factors and relations to job and life satisfaction. Surg 2014;12(4):181e90. 25. Zikmund-Fisher JB, Exe LN, Witteman OH. Numeracy and literacy independently predict patients' ability to identify outof-range test results. J Med Internet Res 2014;16(8):e187. 26. http://www.indicators.scot.nhs.uk/Surgical/Main.html). [accessed online 25.09.14]. 27. Schneider EC, Epstein AM. Use of public performance reports. JAMA 1998;279(20):1638.
Available online at www.sciencedirect.com
ScienceDirect The Surgeon, Journal of the Royal Colleges of Surgeons of Edinburgh and Ireland www.thesurgeon.net
Measuring surgical performance: A risky game? F. Kiernan a,*, F. Rahman b a b
Royal College of Surgeons of Ireland, Beaumont Hospital, Dublin 9, Ireland Health and Social Care Information Centre, St George's University Hospital, Tooting, London, UK
article info
abstract
Article history:
Background: Interest in performance measurement has been driven by increased demand
Received 20 October 2014
for better indicators of hospital quality of care. This is due in part to policy makers wishing
Received in revised form
to benchmark standards of care and implement quality improvements, and also by an
17 December 2014
increased demand for transparency and accountability.
Accepted 18 December 2014
Approach: We describe the role of performance measurement, which is not only about
Available online 28 January 2015
quality improvement, but also serves as a guide in allocating resources within health systems, and between health, education, and social welfare systems. As hospital based
Keywords:
healthcare is responsible for the most cost within the healthcare system, and treats the
Mortality
most severely ill of patients, it is no surprise that performance measurement has focused
Performance measurement
attention on hospital based care, and in particular on surgery, as an important means of
Risk-adjustment
improving quality and accountability. We are particularly concerned about the choice of
Gaming
mortality as an outcome measure in surgery, as this choice assumes that all mortality in
Outcomes
surgery is preventable. In reality, as a low quality indicator of care it risks both gaming, and
Reporting
cream-skimming, unless accurate risk adjustment exists. Further concerns relate to the public reporting of this outcome measure. Conclusions: As mortality rates are an imperfect measure of quality, the reputation of individual surgeons will be threatened by the public release of this data. Significant effort should be made to communicate the results to the public in an appropriate manner. © 2015 Royal College of Surgeons of Edinburgh (Scottish charity number SC005317) and Royal College of Surgeons in Ireland. Published by Elsevier Ltd. All rights reserved.
Healthcare is technophobic. Digital technology has the power to collect and analyse data over populations and time, yet the healthcare sector has been slow to follow the lead of others in using this data to improve care. The evolution of ‘big data’ has given non-healthcare sectors the ability to accurately measure and compare performance, both across jurisdictions and over time, while we remain focused on the idea that information technology cannot capture the multi-dimensional aspects of healthcare necessary for accurate comparisons of performance. The evidence suggests otherwise. Frameworks to
analyse both clinical and cost-effectiveness are continually improving, properly designed software can accurately risk adjust, and the results of this data can be immediately delivered to those who require it. In memory-computing has been used in South Korea to decrease unnecessary antibiotic use, leading to lower costs and a decrease in resistant organisms, and Darmouth-Hitchcock have used large level datasets to decrease length of stay for total knee replacement.1 In measuring health performance, technology is no longer an obstacle. Yet using ‘big data’ in healthcare remains
* Corresponding author. E-mail address: [email protected] (F. Kiernan). http://dx.doi.org/10.1016/j.surge.2014.12.001 1479-666X/© 2015 Royal College of Surgeons of Edinburgh (Scottish charity number SC005317) and Royal College of Surgeons in Ireland. Published by Elsevier Ltd. All rights reserved.
214
t h e s u r g e o n 1 3 ( 2 0 1 5 ) 2 1 3 e2 1 7
controversial, in part due to ethical concerns regarding data protection for patients but increasingly because of concerns regarding protection of healthcare practitioners. Delivering value in healthcare is an increasingly popular topic, driven by increased patient demands, expensive technologies, and an awareness of financial restraint, particularly emphasized by the global economic crisis. While the term ‘performance measurement’ appears to have become synonymous with ‘quality improvement’, in reality it also serves as a guide in the allocation of resources between various interventions, and between the healthcare system and education, or the healthcare system and social welfare. Measuring performance is part of a move towards demand-driven rather than supply-driven healthcare, which makes providers accountable for their outcomes, and clarifies where resources are spent. Ultimately performance is to health policy, what evidence-based medicine is to clinical care. The result is that the role of the healthcare provider has become entwined with the role of the manager, the economist, and the accountant. The care provided by clinicians accounts, understandably, for the most significant expenditure within the healthcare system.2 Healthcare demand is influenced by clinicians, and supply of healthcare is delivered by clinicians. Therefore, it is not surprising that the work of clinicians is considered to be particularly relevant in measuring the performance of the health system. However, the health system is more than a collection of doctors, and it is more than the sum of their reported outcomes. It involves preventative care, primary, secondary and tertiary care, as well as education and social welfare. Furthermore, the goals of the health care system are more than just improved health. They also include accountability, financial protection and equity of access. The World Health Report of 2000 marked a move away from examining improvements in medical care, and included equity, fairness of financial contribution, and responsiveness as goals of the healthcare system.3 Indeed, measurement of level of health of the population accounted for only 25% of the overall score. In country analysis of health system performance should take this weighting system into account when designing their own ranking systems. More recent discussions have focused on the importance of directing resources at areas of health gain, rather than measuring and focusing on non-preventable health loss. Focusing on mortality as a health loss, rather than potential health gains from improvements in quality of life, is one of the reasons why mental healthcare does not receive adequate funding in healthcare budgets. Therefore, although hospital-related healthcare accounts for the most costly care, in order to deliver outcomes that truly matter to the population, we need to look beyond merely the hospital environment, and focus on the outcomes that matter to patients. Improving quality of care for the population requires improvements in community-based care, public health, education and employment. Interest in performance measurement in surgery has been driven by an increased demand for better indicators of quality in hospital care. Reporting this data is in part due to an increased emphasis on demonstrating transparency in healthcare provision, and in part due to an awareness of the role of public reporting as a mechanism to improve quality of
care. Berwick's framework for quality improvement shows that public reporting of outcomes leads to improvements in quality from one of two methods e either patients select better providers of care, or the data provides information on areas of underperformance, leading to a stimulus for improvement from the providers.4 However, the assumption that the publication of this data will result in improvements in quality rests on the assumption that the outcome being measured is amenable to quality improvement. Mortality is often used as a marker of quality of healthcare, predominantly because of the ease with which data on mortality can be collected. However, this data should only be used as a marker of quality if mortality was potentially preventable. While surgical mortality may be avoidable in certain situations, and for certain patients, assuming that it is unavoidable for all is contrary to scientific knowledge. ‘What's measured is what matters', and inappropriate indicators of quality may lead to resources being spent in areas of minimal benefit, at the expense of other areas of care. As the focus has now been put on measuring the mortality rates of individual surgeons, it is time to ask if the choice of mortality as an indicator is appropriate, and what unintended negative effects may result from its use. Decades of analyses of the use of mortality as an outcome measurement in hospitals, has clearly demonstrated that the signal-to-noise ratio is too low for mortality to be reliably used as a reflection of quality.5 This may be either because there is no real correlation between in-hospital mortality and quality of hospital care, or because of the small sample size, coding issues, or methodological concerns.4 In reality, in-hospital death is rarely preventable, and a significant body of evidence has shown that in cases where it is, alterations in care usually resulted in delaying time to death, rather than preventing it completely. Results from surgical care in the United States, the initiators of performance measurements, has demonstrated that hospitals with the highest mortality rates were not the hospitals that were considered to have provided the poorest quality of care.5 A comparison of four different methodological approaches to measuring hospital wide mortality for the same dataset demonstrated significant variations in mortality, depending on the inclusion and exclusion criteria chosen and the statistical approach used.6 The move towards reporting the mortality rates of individual surgeons is entirely based on evidence from cardiac surgery. Work from both the US and England demonstrated a positive benefit of reporting individual cardio-thoracic surgeon's mortality rates. Data from the New York State CABG program is often cited as an example of where publishing outcomes leads to improved performance as a trend of decreased CABG related mortality was observed following public reporting.7,8 In England, public reporting of mortality first took place in cardiothoracic surgery after reconfiguration of services following high mortality rates at Bristol Royal Infirmary.9 The specialty initially published hospital level data in 1998 and then individual surgeons' results in 2006. However, while mortality may be an appropriate outcome measure in cardiac surgery, in other specialties it has been shown to be a poor predictor of quality.10 If unexpected deaths are of particular importance, perhaps death in low risk groups may be a more realistic quality indicator than overall
t h e s u r g e o n 1 3 ( 2 0 1 5 ) 2 1 3 e2 1 7
mortality? As colorectal, urological, and orthopaedic surgeons become the next specialties to be scrutinized in the NHS, it is time to question not only whether or not mortality is an appropriate measure, but also how long patients should be followed for. Mortality from colorectal cancer at 30 days after surgery or chemotherapy/radiotherapy is a marker of quality in Scotland, but it is less clear if 30 day mortality should be considered to be a decisive point. In comparisons of mortality after liver transplantation in the United States versus the United Kingdom and Ireland, mortality was higher in the UK and Ireland after 90 days, but much lower at 1 year.11 This should lead us to also question if an individual surgeon should be considered as being solely responsible for the patient care. Both hospital factors and patient factors relate to mortality, yet many hospital factors are not included in risk adjustment. The number of surgical nurses on a ward, and their educational level, is associated with post-operative mortality,12 yet it does not factor in risk adjustment algorithms in the UK, and is, therefore, unlikely to factor in an equivalent Irish model. Hospital management, and how management relates to employees is a further factor in mortality rates,13 and is beyond the control of the surgical team. The NCEPOD of 2010 identified infrastructure as being a cause of mortality, rather than surgical error. Furthermore, publication of individual surgeon's mortality rates diverts public attention away from deficiencies in the health system.14 While the concept of reporting individual outcomes was believed to be an essential part of moving towards a model of accountable care, in reality the term ‘accountable care’ relates to co-ordinated care delivery, with teams of clinicians responsible for the care of a patient. Discussions should include the effect of the reporting of this data on clinician and public behaviour. Dysfunctional behaviour is known to result from the fear and distrust that commonly occurs after the reporting of results. Either the collection of small amounts of high quality data, or the collection of larger amounts of poorer quality data, will lead to the distortion of results. By measuring what is easy to collect, but necessarily related to quality, resources can be misallocated. Attaching a high-powered incentive to this data, either through remuneration or a risk to reputation, will lead to perverse outcomes. 41% of the reduction in New York's riskadjusted mortality has been described as being related to gaming,15 where intentional up-coding has been reported, in order to reduce the observed-to-expected mortality ratio. While gaming has not yet been reported in the collection of surgical mortality in England, data manipulation and gaming have been reported following other NHS policies, including the alteration of hospital consultation data to show an apparent reduction in waiting list times.16 Cream-skimming, or the avoidance of high-risk patients, is a further potential consequence. In cases where providers refuse to accept the highest risk patients, the failure to provide care for these patients may lead to overall higher mortality and higher costs for the population. Further evidence from the New York Cardiac Reporting System revealed that 62% of surveyed cardiac surgeons in New York refused to operate on a patient in the previous year because of public reporting [ref]. Furthermore, in Pennsylvania, 59% of cardiologists had increased difficulty referring high-risk patients to surgeons
215
following public reporting of data.17 While accurate riskadjustment should prevent cream-skimming, attempts to report mortality publicly have led to increasing debate about current risk-adjustment methods, and the degree to which these measures accurately reflect patient safety and quality.18 Different risk adjustment methods provide different pictures of performance, depending on the source of data, the weighting system, and the variables chosen, which has resulted in different rankings of hospitals based on mortality rates.6 Although the P-POSSUM is used as a risk-adjusted surgical audit tool, it has been shown to be inaccurate for both mortality prediction in gynaecological patients19 and in surgical patients in the United States.20 Furthermore, there is evidence of significant discrepancies between the proprietary comorbidity indices (Disease-Staging and APR-DRGs) and nonproprietary indices21 used in risk adjustment. In the midst of these concerns regarding these measures, the need for automated data linkage has been considered as a means of improving data collection.22 Big data, therefore, may have a particular role in accurate risk adjustment, as sociodemographic, physiological and medical information from 4.6 million patients of the Veterans Administration, was shown to enable the accurate prediction of morbidity and mortality.23 The public reporting of surgical outcomes could be seen as a form of reward and punishment, both of which are a flawed and harmful means of attempting to improve quality. A study in this journal in 2013 described the importance of social support and a sense of community at work in job satisfaction and work engagement.24 Public reporting and ranking of surgeons has resulted in discord due to both the ranking of surgeons, and the manner in which the data was collected. Furthermore this discord has resulted in debate that ultimately distracted from the attempted effort to improve quality.25 Engagement with clinical staff is an essential part of producing meaningful performance measures, and a collaborative relationship between clinicians and management, based on mutual trust, is required for long-term progress. Clinicians should be involved in the choice of each performance measure to ensure it is designed according to its intended use, as a measure designed for improvement should not be used for judgment. In addition, professionals who deliver healthcare should not merely have a say in nature of the choice of outcome measure, but also in how it is communicated to the public. The demand for greater accountability in healthcare is appropriate, and patients have a right to know that surgeons operate to a sufficiently high standard. However, the responsibility for ensuring this lies with relevant medical councils and professional bodies, rather than the media. Reporting outcomes of surgical procedures in a way that is fair to both patients and surgeons requires a two-way conversation. The medical profession, patients, politicians, and the media will need to be educated in how to interpret this data. Surgeon level mortality data in England is currently published online via the NHS choices website. The data for different surgical sub-specialties is presented in different formats with varying levels of detail. Although this data is
216
t h e s u r g e o n 1 3 ( 2 0 1 5 ) 2 1 3 e2 1 7
available for patient viewing, a key consideration is whether this data is presented in a way that is comprehensible by most patients. For example, categorizing surgical performance by percentiles may be clear for surgeons but its relevance for patients is debatable. Patients with poor numeracy and literacy skills may find tables of statistics difficult to contextualize and there is increasing interest in exploring new methods of presenting data.5 Surgeon mortality data from Scotland is presented as an Excel spreadsheet, in an unadjusted form, with a short caveat explaining that out of context data may be misleading to patients.26 In addition, there is little evidence that patients use this data in making choices. In studying the effect of publishing clinical effectiveness data on CABG in Pennsylvania, fewer than 1% of patients used the rating of their surgeon and hospital in selecting their healthcare provider.27 Providing data in a league table is a response to the theory that giving people more information allows them to make better choices. However, the reality is that league tables are only of benefit when the initial quality of care is very poor, and that they can improve care when patients are quality elastic. Those who are quality inelastic will not respond to this information, and will remain in low quality care, a standard which may further deteriorate if resource allocation is based on evidence of improvements in quality of care. When the inelastic group are the poor, less well educated, sick, elderly or immigrants, league tables may be harmful to those who need healthcare the most. We would argue that larger sample sizes be used in measuring performance, in order to ensure accuracy. Therefore mortality should be collected at a hospital level, rather than at an individual surgeon level. Attention must be paid to the distinction between data collection for cost analysis, and data collection for assessing quality. An international consensus is required not only for the choice of indicator, but also for risk adjustment measurement. Performance measurement should be treated as a science, with risk adjustment requiring input of our colleagues in health economics and public health medicine to determine how income and socioeconomic status, along with medical comorbidities, will affect outcome. In addition, standards should be set by professional bodies, and outliers on this scale, who have riskadjusted mortality rates higher than expected, should be investigated. Anomalies in expected outcomes can be determined to be either due to a lack of resources or to poor quality, and management of these anomalies should depend on the cause. Accountability and transparency can be ensured, while also protecting the right to privacy of the relevant healthcare professional.
Conclusion Information is central to improving how the health system responds to the needs of the population it serves. Improving health outcomes, delivering equitable care, ensuring financial protection, and providing accountable care to citizens requires high quality data collection and accurate performance measurement. Public reporting of data can regain public trust by demonstrating that the healthcare system is
open to criticism and creates a culture of openness and transparency. However, delivering improvements in the healthcare system requires accurate data collection, based on clinical assessment of need, rather than political decisions. To measure what matters to patients, we need to measure what is preventable. Measures to decrease gaming, and ensure risk adjustment are necessary to prevent perverse outcomes. Furthermore, the data needs to be communicated to those who use it, in a manner that adds clarity rather than confusion. Ensuring that these measures are met requires comprehensive data collection that evaluates the institution, the clinicians, and the patient. Providing care for a surgical patient is multi-faceted and complex, analysing that care is no different. Data collection on mortality may be easy to collect, process and disseminate, but it overlooks system wide deficiencies, and emphasizes health loss over health gain. In the era of ‘big data’, we should harness our ability to obtain accurate information, which can be used to assess outcomes that are potentially preventable, and that matter to patients.
references
1. https://hbr.org/resources/pdfs/comm/sap/18826_HBR_SAP_ Healthcare_Aug_2014.pdf [accessed online 01.12.14]. 2. Dixon J. Reform and the National Health Service. Political Q 2012;83(2):343e52. 3. World Health Report (2000). The World Health Report 2000 e health systems: improving performance http://www.who.int/ whr/2000/en/whr00_en.pdf [accessed online 10.09.14]. 4. Berwick D. Disseminating innovations in health care. JAMA 2003;289(15):1696e975. 5. Lilford R, Pronovost P. Using hospital mortality rates to judge hospital performance: a bad idea that just won't go away. Br Med J 2010;340:955e7. 6. Iezzoni L. The risks of risk adjustment. JAMA 1997; 19;278(19):1600e7. 7. Thomas J, Hofer T. Research evidence on the validity of riskadjusted mortality rate as a measure of hospital quality of care. Med Care Res Rev 1998;4(55):371e404. 8. Chassin MR. Achieving and sustaining improved quality: lessons from New York State and cardiac surgery. Health Aff 2002;21(4):40e51. 9. Hannan EL, Siu AL, Kumar D, Kilburn Jr H, Chassin MR. The decline in coronary artery bypass graft surgery mortality in New York State: the role of surgeon volume. JAMA 1995;273(3):209e13. 10. Keogh B, Dussek J, Watson D, Magee P, Whatley D. Public confidence and cardiac surgical outcome. Br Med J 1998;316:1759e64. 11. Hofer TP, Hayward RA. Identifying poor-quality hospitals. Can hospital mortality rates detect quality problems for medical diagnoses? Med Care 1996; Aug;34(8):737e53. 12. Dawwas MF, Gimson AE, Lewsey JD, Copley LP, van der Meulen JHP, on behalf of the UK and Ireland Liver Transplant Audit. Survival after liver transplantation in the United Kingdom and Ireland compared with the United States. Gut 2007;56:1606e13. 13. Aiken LH, Sloane DM, Bruyneel L, Van den Heede K, et al., for the RN4CAST consortium. Nurse staffing and education and hospital mortality in nine European countries: a retrospective observational study. Lancet 2014; May 24;383(9931):1824e30.
t h e s u r g e o n 1 3 ( 2 0 1 5 ) 2 1 3 e2 1 7
14. West MA, Borrill C, Dawson J. The link between the management of employees and patient mortality in acute hospitals. Int J Hum Resour Manag 2002;13(8):1299e310. 15. Westaby S. Publishing individual surgeons' death rates prompts risk averse behaviour. Br Med J 2014; 12;349:g5026. 16. Green J, Wintfield N. Report cards on cardiac surgeons: assessing New York state's approach. N Engl J Med 1995;332:1229e32. 17. Schneider EC, Epstein AM. Influence of cardiac-surgery performance reports on referral practices and access to care d a survey of cardiovascular specialists' surgery: influence on referral practices and access to care. N. Engl J Med 1996;335(4):251e6. 18. Smith P. On the unintended consequences of publishing performance data in the public sector. Int J Public Adm 1995;18(2e3):277e310. 19. Das N, Talaat AS, Naik R, et al. Risk adjusted surgical audit in gynaecological oncology: P-POSSUM does not predict outcome. Eur J Surg Oncol 2006;32(10):1135e8. 20. Bennett-Guerrero E, Hyam JA, Shaefi S, et al. Comparison of PPOSSUM risk-adjusted mortality rates after surgery between patients in the USA and the UK. Br J Surg 2003;90(12):1593e8.
217
21. Grendar J, Shaheen AA, Myers RP, et al. Predicting in-hospital mortality in patients undergoing complex gastrointestinal surgery: determining the optimal risk adjustment method. Arch Surg 2012;147(2):126e35. 22. IBruce J, Russell EM, Mollison J, Krukowski ZH. The measurement and monitoring of surgical adverse events. Health Technol Assess 2001;5(22):1e194. 23. Wang L, Porter B, Maynard C, Evans G, Bryson C, Sun H, et al. Predicting risk of hospitalization or death among patients receiving primary care in the Veterans Health Administration. Med Care 2013;51:368e73. 24. Mache S, Vitzthum K, Klapp BF, Danzer G. Surgeons' work engagement: influencing factors and relations to job and life satisfaction. Surg 2014;12(4):181e90. 25. Zikmund-Fisher JB, Exe LN, Witteman OH. Numeracy and literacy independently predict patients' ability to identify outof-range test results. J Med Internet Res 2014;16(8):e187. 26. http://www.indicators.scot.nhs.uk/Surgical/Main.html). [accessed online 25.09.14]. 27. Schneider EC, Epstein AM. Use of public performance reports. JAMA 1998;279(20):1638.
