Leading article
Publication of surgeon-specific outcomes D. Alderson1 and D. Cromwell2 1 University
Department of Surgery, School of Cancer Sciences, University of Birmingham, Birmingham B15 2TH, and 2 Clinical Effectiveness Unit, Royal College of Surgeons of England, London, UK (e-mail: [email protected])
Published online in Wiley Online Library (www.bjs.co.uk). DOI: 10.1002/bjs.9641
During 2013, a political directive to publish operative mortality rates for individual surgeons across a variety of procedures in England was met with divided opinion among the surgical community, patients, the media and organizations responsible for regulation and delivery of surgical services. Justification for the initiative from National Health Service England was based on the notion that publishing these figures would identify poor surgical performance, improve the quality of care for surgical patients, help the public to make more informed choices about where to have an operation, and inform healthcare planners about the distribution of surgical resources. The usefulness of this exercise remains open to debate1 . Nine surgical specialties published information during the summer of 2013, starting with figures on cardiac and vascular procedures. Some national media focused on a handful of surgeons with apparently high mortality rates after abdominal aortic aneurysm surgery, although their figures were within the expected range for the volume of activity provided by the Vascular Society of Great Britain and Ireland for 2013. The other surgical audits attracted no media interest, and no specific initiatives seem to have resulted from the information gathered. Surgeons in other healthcare systems might wonder whether they will also be required to provide this information in future, what consequences might ensue and how the profession should respond. To © 2014 BJS Society Ltd Published by John Wiley & Sons Ltd
date, the issue has not received much attention in other countries but, where it has, there seems little support. A survey2 in the USA indicated that, although surgeons there supported public reporting of quality metrics at an institutional level, only a minority were in favour of individual reporting and only about one-quarter felt that this would improve outcomes. That some surgeons obtained better results than others was first pointed out in 19793 , when both higher surgeon and provider (hospital) volume were shown to be associated with lower mortality across a range of complex operations. In the ensuing 35 years, evidence has accumulated in almost every branch of surgery that institutional and/or surgeon experience improves patient care. Reports4,5 suggesting that low-volume hospitals can do complex surgery equally well in terms of clinical outcomes, patient satisfaction and cost have been swamped by literature to the contrary. In some countries, this has resulted in the organization of surgical services that restricts the performance of high-risk operations to hospitals based on catchment populations or operative caseload6,7 . No country has yet rationalized services solely on operative mortality rates achieved by individual surgeons. Separating the impact of individual performance on patient outcomes remains difficult. Evaluations of new surgical interventions indicate that a myriad of institutional factors influence outcome8 . Can the publication
of an individual surgeon’s results truly indicate performance, or is this merely a surrogate reflecting available resources and overall performance within an institution? For example, in a study9 of coronary artery bypass grafting, aortic valve repair or abdominal aortic aneurysm repair, a substantially higher risk of death was seen among patients suffering serious complications after surgery in lower-volume hospitals compared with higher-volume hospitals. This suggests that the quality of postoperative care makes a vital contribution to outcome as well as operative skill. In an era of openness and public accountability, however, there is no reason why surgeons’ outcomes should not be freely available. Anything else implies that surgeons have something to hide. The notion of publishing a rating of a surgeon’s performance may seem unpleasant, but politicians and the media are likely to insist that patients have a right to know that surgeons are performing to a safe standard. The challenge for the surgical community is to ensure that the information is accurate and meaningful, and to work out how best to get this into the public domain. Reporting outcomes of specific operations fairly, in a way that can be accepted by the profession and understood by the public, is challenging. Few surgeons carry out large numbers of a specific operation in a short interval. A small denominator of procedures means that reliably identifying ‘outlying’ performance is BJS 2014; 101: 1335–1337
1336
difficult10 . If a long observation time is needed to acquire this information, other changes in patient management will influence results. There is the issue of selecting an appropriate outcome measure. Last year, the surgical specialties were instructed to publish postoperative mortality. Although this might be relevant for a small number of high-risk operations, it is not meaningful for many other procedures. For groin hernia surgery, recurrence might be the appropriate outcome. Patient selection, referral practices and risk stratification for individual patients are all difficult issues, particularly when surgeons typically undertake operations for conditions affecting older patients with co-morbidities. Finally, the process may introduce some perverse incentives. If the publication of results is not mandatory, poorly performing surgeons may not report, thereby penalizing surgeons who are open and transparent. Conservatism as a means of reducing short-term complications and mortality might deprive some patients of cure. There is also the danger that data quality will be compromised, particularly for complications whose definition is open to interpretation. In all of this, there is risk that training might be affected adversely. The surgical community should meet this challenge. It may seem unfair that surgeons are being evaluated in a way that does not apply to most other hospital doctors, but the surgeon–patient relationship is quite different from that between patient and anaesthetist. Regulatory bodies and specialist associations should work together to deal with the above points. If it is accepted that there is a need for the profession to place consultant-level results in the public domain, then clinical audit must be compulsory at unit and personal level. This demands that data collection, © 2014 BJS Society Ltd Published by John Wiley & Sons Ltd
D. Alderson and D. Cromwell
attribution of work and recording are accurate, adjustments are made for differences in case mix, and results that are made public are relevant to each operation and easily interpretable. The last point is worth stressing. The experience of the first round of publications highlighted how the figures can be misrepresented. Education of the public, politicians and surgeons in the interpretation of these data is vital. A small survey11 in the USA suggested that even among surgeons only about half had reasonable comprehension of data validity, accuracy or complexity. The undeserved criticism of individual surgeons by an unscientific popular press that alarms the public can be allayed only by accurate data collection. The surgical community should recognize this as its responsibility, but must be provided with the necessary resources to do this. The notion that publication of surgeons’ results helps the profession, as well as enabling patients to make more informed choices, is as yet unproven. Experiments have yet to be performed, and the design of studies to answer these questions will not be easy. If driven by external forces, the scientific rigour essential to do this properly may be compromised. Although it is important for the surgical community to rise to the challenge of knowing the value of each surgeon’s results, this should not be confused with improving outcomes for patients. This is always more likely to be achieved by compulsory national clinical audits that report on an institutional basis. Addressing the issues raised above does assist this task, but the delivery of other services essential to good surgical outcomes means that political pressure must be brought to bear so that all aspects of a service are considered, rather than the surgeon in isolation. Insisting on institutional publication www.bjs.co.uk
of results is far more likely than anything else to help patients in the long run. Disclosure
The authors declare no conflict of interest. References 1 NHS choices. Consultant Treatment Outcomes. http://www.nhs.uk/ choiceintheNHS/Yourchoices/ consultant-choice/Pages/consultantdata.aspx [accessed 6 February 2014]. 2 Sherman KL, Gordon EJ, Mahvi DM, Chung J, Bentrem DJ, Holl JL et al. Surgeons’ perceptions of public reporting of hospital and individual surgeon quality. Med Care 2013; 51: 1069–1075. 3 Luft HS, Bunker JP, Enthoven AC. Should operations be regionalized? The empirical relation between surgical volume and mortality. N Engl J Med 1979; 301: 1364–1369. 4 Killeen SD, O’Sullivan MJ, Coffey JC, Kirwan WO, Redmond HP. Provider volume and outcomes for oncological procedures. Br J Surg 2005; 92: 389–402. 5 Yoshioka R, Yasunaga H, Hasegawa K, Horiguchi H, Fushimi K, Aoki T et al. Impact of hospital volume on hospital mortality, length of stay and total costs after pancreaticoduodenectomy. Br J Surg 2014; 101: 523–529. 6 Finks JF, Osborne NH, Birkmeyer JD. Trends in hospital volume and operative mortality for high-risk surgery. N Engl J Med 2011; 364: 2128–2137. 7 De Wilde RF, Besselink MGH, van der Tweel I, de Hingh IHJT, van Eijck CHJ, Dejong CHC et al. Impact of nationwide centralization of pancreaticoduodenectomy on hospital mortality. Br J Surg 2012; 99: 404–410. 8 Ergina PL, Cook JA, Blazeby JM, Boutron I, Clavien PA, Reeves BC et al. Challenges in evaluating surgical
BJS 2014; 101: 1335–1337
Publication of surgeon-specific outcomes
innovation. Lancet 2009; 374: 1097–1104. 9 Gonzalez AA, Dimick JB, Birkmeyer JD, Ghaferi AA. Understanding the volume–outcome effect in cardiovascular surgery: the role of failure to rescue. JAMA Surg 2014; 149: 119–123.
1337
10 Walker K, Neuburger J, Groene O, Cromwell DA, van der Meulen J. Public reporting of surgeon outcomes: low numbers of procedures lead to false complacency. Lancet 2013; 382: 1674–1677. 11 Yi SG, Wray NP, Jones SL, Bass BL, Nishioki J, Brann S et al.
Surgeon-specific performance reports in general surgery: an observational study of initial implementation and adoption. J Am Coll Surg 2013; 217: 636–647.
If you wish to comment on this, or any other article published in the BJS, please visit the on-line correspondence section of the website (www.bjs.co.uk).
© 2014 BJS Society Ltd Published by John Wiley & Sons Ltd
www.bjs.co.uk
BJS 2014; 101: 1335–1337
Publication of surgeon-specific outcomes D. Alderson1 and D. Cromwell2 1 University
Department of Surgery, School of Cancer Sciences, University of Birmingham, Birmingham B15 2TH, and 2 Clinical Effectiveness Unit, Royal College of Surgeons of England, London, UK (e-mail: [email protected])
Published online in Wiley Online Library (www.bjs.co.uk). DOI: 10.1002/bjs.9641
During 2013, a political directive to publish operative mortality rates for individual surgeons across a variety of procedures in England was met with divided opinion among the surgical community, patients, the media and organizations responsible for regulation and delivery of surgical services. Justification for the initiative from National Health Service England was based on the notion that publishing these figures would identify poor surgical performance, improve the quality of care for surgical patients, help the public to make more informed choices about where to have an operation, and inform healthcare planners about the distribution of surgical resources. The usefulness of this exercise remains open to debate1 . Nine surgical specialties published information during the summer of 2013, starting with figures on cardiac and vascular procedures. Some national media focused on a handful of surgeons with apparently high mortality rates after abdominal aortic aneurysm surgery, although their figures were within the expected range for the volume of activity provided by the Vascular Society of Great Britain and Ireland for 2013. The other surgical audits attracted no media interest, and no specific initiatives seem to have resulted from the information gathered. Surgeons in other healthcare systems might wonder whether they will also be required to provide this information in future, what consequences might ensue and how the profession should respond. To © 2014 BJS Society Ltd Published by John Wiley & Sons Ltd
date, the issue has not received much attention in other countries but, where it has, there seems little support. A survey2 in the USA indicated that, although surgeons there supported public reporting of quality metrics at an institutional level, only a minority were in favour of individual reporting and only about one-quarter felt that this would improve outcomes. That some surgeons obtained better results than others was first pointed out in 19793 , when both higher surgeon and provider (hospital) volume were shown to be associated with lower mortality across a range of complex operations. In the ensuing 35 years, evidence has accumulated in almost every branch of surgery that institutional and/or surgeon experience improves patient care. Reports4,5 suggesting that low-volume hospitals can do complex surgery equally well in terms of clinical outcomes, patient satisfaction and cost have been swamped by literature to the contrary. In some countries, this has resulted in the organization of surgical services that restricts the performance of high-risk operations to hospitals based on catchment populations or operative caseload6,7 . No country has yet rationalized services solely on operative mortality rates achieved by individual surgeons. Separating the impact of individual performance on patient outcomes remains difficult. Evaluations of new surgical interventions indicate that a myriad of institutional factors influence outcome8 . Can the publication
of an individual surgeon’s results truly indicate performance, or is this merely a surrogate reflecting available resources and overall performance within an institution? For example, in a study9 of coronary artery bypass grafting, aortic valve repair or abdominal aortic aneurysm repair, a substantially higher risk of death was seen among patients suffering serious complications after surgery in lower-volume hospitals compared with higher-volume hospitals. This suggests that the quality of postoperative care makes a vital contribution to outcome as well as operative skill. In an era of openness and public accountability, however, there is no reason why surgeons’ outcomes should not be freely available. Anything else implies that surgeons have something to hide. The notion of publishing a rating of a surgeon’s performance may seem unpleasant, but politicians and the media are likely to insist that patients have a right to know that surgeons are performing to a safe standard. The challenge for the surgical community is to ensure that the information is accurate and meaningful, and to work out how best to get this into the public domain. Reporting outcomes of specific operations fairly, in a way that can be accepted by the profession and understood by the public, is challenging. Few surgeons carry out large numbers of a specific operation in a short interval. A small denominator of procedures means that reliably identifying ‘outlying’ performance is BJS 2014; 101: 1335–1337
1336
difficult10 . If a long observation time is needed to acquire this information, other changes in patient management will influence results. There is the issue of selecting an appropriate outcome measure. Last year, the surgical specialties were instructed to publish postoperative mortality. Although this might be relevant for a small number of high-risk operations, it is not meaningful for many other procedures. For groin hernia surgery, recurrence might be the appropriate outcome. Patient selection, referral practices and risk stratification for individual patients are all difficult issues, particularly when surgeons typically undertake operations for conditions affecting older patients with co-morbidities. Finally, the process may introduce some perverse incentives. If the publication of results is not mandatory, poorly performing surgeons may not report, thereby penalizing surgeons who are open and transparent. Conservatism as a means of reducing short-term complications and mortality might deprive some patients of cure. There is also the danger that data quality will be compromised, particularly for complications whose definition is open to interpretation. In all of this, there is risk that training might be affected adversely. The surgical community should meet this challenge. It may seem unfair that surgeons are being evaluated in a way that does not apply to most other hospital doctors, but the surgeon–patient relationship is quite different from that between patient and anaesthetist. Regulatory bodies and specialist associations should work together to deal with the above points. If it is accepted that there is a need for the profession to place consultant-level results in the public domain, then clinical audit must be compulsory at unit and personal level. This demands that data collection, © 2014 BJS Society Ltd Published by John Wiley & Sons Ltd
D. Alderson and D. Cromwell
attribution of work and recording are accurate, adjustments are made for differences in case mix, and results that are made public are relevant to each operation and easily interpretable. The last point is worth stressing. The experience of the first round of publications highlighted how the figures can be misrepresented. Education of the public, politicians and surgeons in the interpretation of these data is vital. A small survey11 in the USA suggested that even among surgeons only about half had reasonable comprehension of data validity, accuracy or complexity. The undeserved criticism of individual surgeons by an unscientific popular press that alarms the public can be allayed only by accurate data collection. The surgical community should recognize this as its responsibility, but must be provided with the necessary resources to do this. The notion that publication of surgeons’ results helps the profession, as well as enabling patients to make more informed choices, is as yet unproven. Experiments have yet to be performed, and the design of studies to answer these questions will not be easy. If driven by external forces, the scientific rigour essential to do this properly may be compromised. Although it is important for the surgical community to rise to the challenge of knowing the value of each surgeon’s results, this should not be confused with improving outcomes for patients. This is always more likely to be achieved by compulsory national clinical audits that report on an institutional basis. Addressing the issues raised above does assist this task, but the delivery of other services essential to good surgical outcomes means that political pressure must be brought to bear so that all aspects of a service are considered, rather than the surgeon in isolation. Insisting on institutional publication www.bjs.co.uk
of results is far more likely than anything else to help patients in the long run. Disclosure
The authors declare no conflict of interest. References 1 NHS choices. Consultant Treatment Outcomes. http://www.nhs.uk/ choiceintheNHS/Yourchoices/ consultant-choice/Pages/consultantdata.aspx [accessed 6 February 2014]. 2 Sherman KL, Gordon EJ, Mahvi DM, Chung J, Bentrem DJ, Holl JL et al. Surgeons’ perceptions of public reporting of hospital and individual surgeon quality. Med Care 2013; 51: 1069–1075. 3 Luft HS, Bunker JP, Enthoven AC. Should operations be regionalized? The empirical relation between surgical volume and mortality. N Engl J Med 1979; 301: 1364–1369. 4 Killeen SD, O’Sullivan MJ, Coffey JC, Kirwan WO, Redmond HP. Provider volume and outcomes for oncological procedures. Br J Surg 2005; 92: 389–402. 5 Yoshioka R, Yasunaga H, Hasegawa K, Horiguchi H, Fushimi K, Aoki T et al. Impact of hospital volume on hospital mortality, length of stay and total costs after pancreaticoduodenectomy. Br J Surg 2014; 101: 523–529. 6 Finks JF, Osborne NH, Birkmeyer JD. Trends in hospital volume and operative mortality for high-risk surgery. N Engl J Med 2011; 364: 2128–2137. 7 De Wilde RF, Besselink MGH, van der Tweel I, de Hingh IHJT, van Eijck CHJ, Dejong CHC et al. Impact of nationwide centralization of pancreaticoduodenectomy on hospital mortality. Br J Surg 2012; 99: 404–410. 8 Ergina PL, Cook JA, Blazeby JM, Boutron I, Clavien PA, Reeves BC et al. Challenges in evaluating surgical
BJS 2014; 101: 1335–1337
Publication of surgeon-specific outcomes
innovation. Lancet 2009; 374: 1097–1104. 9 Gonzalez AA, Dimick JB, Birkmeyer JD, Ghaferi AA. Understanding the volume–outcome effect in cardiovascular surgery: the role of failure to rescue. JAMA Surg 2014; 149: 119–123.
1337
10 Walker K, Neuburger J, Groene O, Cromwell DA, van der Meulen J. Public reporting of surgeon outcomes: low numbers of procedures lead to false complacency. Lancet 2013; 382: 1674–1677. 11 Yi SG, Wray NP, Jones SL, Bass BL, Nishioki J, Brann S et al.
Surgeon-specific performance reports in general surgery: an observational study of initial implementation and adoption. J Am Coll Surg 2013; 217: 636–647.
If you wish to comment on this, or any other article published in the BJS, please visit the on-line correspondence section of the website (www.bjs.co.uk).
© 2014 BJS Society Ltd Published by John Wiley & Sons Ltd
www.bjs.co.uk
BJS 2014; 101: 1335–1337
