World J Surg DOI 10.1007/s00268-014-2897-0
SCIENTIFIC REVIEW
The Impact of Feedback of Surgical Outcome Data on Surgical Performance: A Systematic Review Mahiben Maruthappu • Abhishek Trehan • Ashton Barnett-Vanes • Peter McCulloch • Matthew J. Carty
Ó Socie´te´ Internationale de Chirurgie 2014
Abstract Background Increasing patient demands, costs and emphasis on safety have led to performance tracking of individual surgeons. Several methods of using these data, including feedback have been proposed. Our aim was to systematically review the impact of feedback of outcome data to surgeons on their performance. Study design MEDLINE, Embase, PsycINFO, AMED and the Cochrane Database of Systematic Reviews (from their inception to February 2013) were searched. Two reviewers independently reviewed citations using predetermined inclusion and exclusion criteria. Forty two data-points per study were extracted. Results The search strategy yielded 1,531 citations. Seven studies were eligible comprising 18,632 cases or procedures by 52 surgeons. Overall, feedback was found to be a powerful method for improving surgical outcomes or indicators of surgical performance, including reductions in hospital mortality after CABG of 24 % (P = 0.001), decreases of stroke and mortality following carotid endarterectomy from 5.2 to 2.3 %, improved ovarian cancer resection from 77 to 85 % (P = 0.157) and reductions in wound infection rates from 14 to 10.3 %. Improvements in performance occurred in concert with reduced costs: for hepaticojejunostomy, implementation of feedback was associated with a decrease in overall hospital costs from $24,446 to $20,240 (P \ 0.01). Similarly, total cost of carotid endarterectomy and following management decreased from $13,344 to $9548. Conclusions The available literature suggests that feedback can improve surgical performance and outcomes; however, given the heterogeneity and limited number of studies, in addition to their non-randomised nature, it is difficult to draw clear conclusions from the literature with regard to the efficacy of feedback and the specific nuances required to optimise the impact of feedback. There is a clear need for more rigorous studies to determine how feedback of outcome data may impact performance, and whether this low-cost intervention has potential to benefit surgical practice.
Mahiben Maruthappu and Abhishek Trehan have equally contributed to the development of this manuscript. M. Maruthappu A. Barnett-Vanes Imperial College London, London SW7 2AZ, UK e-mail: [email protected]
P. McCulloch Nuffield Department of Surgical Sciences, University of Oxford, Oxford OX3 9DU, UK
A. Trehan (&) Lincoln College, University of Oxford, Oxford OX1 3DR, UK e-mail: [email protected]
M. J. Carty Harvard Medical School, Boston, MA 02115, USA
123
World J Surg
Introduction
Methods
Surgery plays a key role in the delivery of modern clinical care [1]. Patients rightly expect routinely safe surgical care, and biomedical science is redefining patient experiences for a myriad of surgically treatable diseases. With advances in research, technology and training, expectations of surgical performance and outcomes have risen, as have levels of scrutiny. Recognition that patient satisfaction is only one indicator of surgical quality [2] and calls for greater transparency to improve safety [3] have fostered growing emphasis on the monitoring and evaluation of individual surgical performance, at both the institution [4] and national level [5]. Such trends are taking place across several countries; for example, in the United Kingdom, surgeons are expected to publish their individual performance data [6] as part of an era of ‘‘new openness in medicine’’ [7]. Several databases and sources of information have been developed to facilitate dissemination of performance data, such as online national and international clinical data registers. A priority for the surgical community is to ensure such monitoring translates into tangible and reproducible improvements in surgical performance, for the benefit of surgeons and ultimately patients. The American College of Surgeons’ National Surgical Quality Improvement Program which focuses on providing feedback to hospitals (rather than individual surgeons) has been shown to improve surgical outcomes [8]. In the UK, the impact of publication of individual surgical outcomes has yet to be assessed; further, the optimal means of using these data to provide individual feedback to surgeons is not clear. Such data could be used to provide specific information to surgeons on their observed performance compared to an expected standard, with the aim of improving subsequent performance and outcomes [9]. However, delivered inappropriately, feedback may be detrimental to performance [10]. Despite extensive literature on feedback strategies in medical education and clinical medicine [9, 11–13], few studies have examined this in the context of surgical performance. Although feedback is widely known to be a key component of surgical training, and individual surgeon data able to quantitatively support such feedback are available, and whilst several methods have been proposed to utilise outcome data for feedback, the impact of these methods on surgical performance and outcomes has not been reviewed. We performed a systematic review aiming to examine the feedback of outcome data and to synthesise the methods of feedback employed. For the purpose of this study, feedback was defined as an active process, comprising the measurement of individual surgeon outcome results, and subsequent relay of this information to the operating surgeon, through means such as open discussion, mentoring or coaching, with the aim of improving his or her performance.
Data sources and search strategy
123
The systematic review was conducted in accordance with PRISMA guidelines (http://www.prisma-statement.org). A comprehensive search was undertaken to determine the impact of feedback on surgical performance via the Ovid SP interface. The following databases were searched from inception to February 2013: MEDLINE, Embase, PsycINFO, AMED and the Cochrane Database of Systematic Reviews. We used two different domains of MeSH-terms and keywords combined by ‘‘AND’’, and within each domain the terms were combined by ‘‘OR’’. The first domain contained terms related to surgical skill and performance, whilst the second contained terms related to the impact of feedback. A detailed search strategy can be found in the Appendix. The search was limited to English publications with no other restrictions. Study selection Two reviewers independently reviewed citations and selected eligible studies based upon predetermined inclusion and exclusion criteria. Publications were selected for review if they satisfied the following inclusion criteria: article was published in a peer-reviewed journal; article described a study involving surgical patients; article investigated the impact of individual surgeon feedback (defined as either knowledge or relay of surgical outcomes) on surgical performance or outcome; article used a statistical unit that was patient or procedure focused (e.g. operative time or complication rate). The following exclusion criteria were applied to search results: the article was a conference abstract, editorial, letter, opinion, or review; the population studied was non-surgical (for example, radiology, pathology, diagnostic endoscopy); the article used qualitative rather than quantitative approaches (e.g. comments on video observation as opposed to duration, number of incisions); the study population was medical students; articles which proposed mechanisms of providing feedback, without quantitatively evaluating the impact of such mechanisms on postoperative outcomes. Two authors (MM, AT) independently examined all retrieved articles for inclusion. Any disagreement over inclusion or exclusion was resolved by consensus. Data extraction Forty two data-points per study were extracted using a predesigned data collection form including first author, year of publication, study aim, study type, study design (prospective, retrospective, experimental, observational, cross-sectional, longitudinal), study population, population setting (e.g. hospital), surgical speciality, surgical procedure analysed,
World J Surg
Studies identified from the databases n= 1531
References excluded: Conference abstract, editorial, letter, opinion, or review; n = 290 Non-surgical study; n = 93 Study described a means of feedback, but did not assess the impact of this feedback = 71 Inclusion criteria not met; n = 731
References retrieved for evaluation of abstracts n= 346 References excluded: Conference abstract, editorial, letter, opinion, or review; n=9 Non-surgical study; n = 15 Study described a means of feedback, but did not assess the impact of this feedback = 56 Medical student participants; n = 19 Inclusion criteria not met; n = 237
References retrieved for evaluation of full manuscripts n= 10
References excluded: Studies not focused on feedback of outcome data to surgeons; n = 3
References retrieved for review n= 7
Fig. 1 Search strategy
number of surgeons, types of feedback dissemination, content of feedback, frequency of feedback, measured outcomes (e.g. mortality, morbidity), length of followup, interventions following feedback and impact of interventions.
Results Study identification and selection Our search yielded 1,531 citations, of which seven studies were included in the final analysis comprising 18,632 cases or procedures by a total of 52 surgeons. Notably, 56 studies, although describing a means of feedback, did not assess it. A flow diagram of the search results is illustrated in Fig. 1.
Study characteristics Studies originated from the United States, Canada and Europe. All were before-after studies, covering a range of surgical specialties, including: cardiothoracic [14], neurosurgical [15, 16] and general surgery [17]. All included studies are shown in Table 1 and their basic characteristics are summarised in Table 2. The full data extraction from the studies can be found in the Appendix. Feedback dissemination and frequency Feedback was disseminated by a variety of methods. All seven studies examined delivered feedback as outcomespecific data [14–20], three studies accompanied this with
123
123
Year
2008
2000
2002
1996
1999
2002
2005
Reference
Aletti et al. [19]
Cornelius Olcott IV et al. [16]
Findlay et al. [15]
O’ Connor et al. [14]
Pitt et al. [20]
Reilly et al. [17]
Rodriguez et al. [18]
Spain
Scotland
USA
USA
Canada
USA
USA
Country
Gynaecology
General
Pancreatobiliary
Cardiothoracic
Neurosurgery, general and vascular
Neurosurgery, general and vascular
Gynaecology
Speciality
Hysterectomy
All general surgical procedures
Choledochojejunostomy, cholangiojejunostomy, or hepaticojejunostomy
Coronary artery bypass grafting
Carotid endarterectomy
Carotid endarterectomy
Surgical management of ovarian cancer
Surgical procedure
–
5
4 plus ‘‘several others’’
23
10
10
–
No. of participating surgeons
Table 1 Overview of studies included and impact of feedback on performance
980
2241
339
13,126
946
763
237
Total no. of cases
Before– after study
Before– after study
Before– after study
Before– after study
Before– after study
Before– after study
Before– after study
Study design
Infection rate
Infection rate
Mortality rate
Mortality rate
30 days
30 days
Not specified
30 days
30 days
2. 30 days
2. Mortality rate Mortality and stroke rate
1. 30 days
2. 3 months
2. Mortality rate 1. Stroke rate
1. 30 days
Length of follow up for primary outcome measure
1. Morbidity rate
Primary outcome measure
10.7 %
14 %
4.5 %
4.75 %
5.2 %
2. 3.8 %
1. 3.8 %
2. 8 %
1. 21 %
Primary outcome measure: prefeedback
6.0 %
10.3 %
0.7 %
3.61 %
2.3 %
2. 0 %
1. 0 %
2. 6 %
1. 20 %
Primary outcome measure: postfeedback
-43.9 %
-26 %
-85 %
-24 %
-56 %
%2. -100%
1. -100
2. -25 %
1. -5 %
Impact of feedback on primary outcome
Not provided
Not statistically significant
\0.05
0.001
Not provided
Not provided
2. 0.475
1. 0.819
P value
World J Surg
World J Surg
Feedback contents
Table 2 Basic characteristics of studies included Study characteristics
No. of studies
References
All studies
7
[14–20]
Written
1
[15]
Clinical meeting
1
[17]
Written and clinical meeting
3
[14, 16, 19]
Not stated
2
[18, 20]
Feedback dissemination
Feedback contents Outcomes
7
[14–20]
Benchmarking relative to peers
4
[14, 16, 18, 19]
Comparable literature-reported figures
1
[19]
Feedback frequency Single time-point
1
[20]
Monthly
1
[17]
Tri-annual Annually
1 3
[14] [15, 16, 18]
Not stated
1
[19]
Length of follow up 30 days
5
[14–18]
30 days (plus 3 months for mortality)
1
[19]
Not stated
1
[20]
Any outcome
6/7
[14–18, 20]a
Mortality
4/5
[14–16, 20]
Morbidity Infection rate
2/4 2/2
[16, 18] [17]a [18]
No. of studies reporting improvement in outcome/no. of studies assessing that outcome
a
Length of stay
2/3
[16, 20]
Hospital charges
2/2
[16, 20]
Not statistically significant improvement
clinical meetings to provide a forum for discussion and improvement [14, 16, 19]. One study mailed surgeons their results [15]. Frequency of feedback varied between studies, from a single time-point [20], monthly [17], three times per year [14] or annual intervals [15, 16, 18]. There was no difference in the impact of feedback between studies utilising different frequencies. However, one study did not specifically define the frequency, and this study saw no significant improvement in performance [19]. Four studies provided benchmarking relative to peers, with three studies observing an improvement, and one study reporting no significant improvement [19].
Feedback in these studies was provided in the form of patient outcome data, the nature and extent of which varied depending on the study objective. Five studies provided surgeons with morbidity or mortality rates [14–16, 19, 20]. Two studies focusing on wound contamination provided surgeons with feedback on wound infection rates, one in the context of clean elective surgery [17], and the other for abdominal hysterectomy [18]. Four of the studies examined provided feedback together with outcome benchmarking, either in comparison to their peers [18, 19] or institutions [16], and additionally to regional statistics [14]. One of these studies also included comparison with results from the recent literature [19]. Three studies provided additional operative details to surgeons, including the calculated appropriateness of the surgery undertaken [15], complication rates [14, 15], surgical complexity and the presence of residual disease [19]. One study provided additional non-clinical data to surgeons, such as hospital charges [20]. Impact of feedback Table 1 summarises the impact of feedback on surgical performance in the analysed studies. All studies noted improvements in their primary outcomes following implementation of measures that included feedback (i.e. the studies demonstrated improvements in outcomes they aimed to improve). One study involving 23 surgeons performing coronary artery bypass graft surgery (CABG) observed a 24 % reduction (P = 0.001) in hospital mortality compared to expected figures in the post-intervention period, which included feedback, quality improvement and site visits [14]. One study, including 10 surgeons performing carotid endarterectomy (CEA) observed the number of stroke and death complications decline over the study period, from an initial overall rate of 5.2–2.3 % [15]. Another study of CEA involving 10 surgeons observed a decrease in stroke and mortality after providing surgeons with feedback on morbidity and mortality rates; however, there was no decline in the incidence of postoperative myocardial infarction [16]. In a study investigating mortality in complex biliary surgery after the implementation of a clinical pathway that included outcome feedback to surgeons, a decrease in hospital mortality from 4.5 to 0.7 % (P \ 0.05) was observed [20]. In one study of ovarian cancer cases, implementation of a quality improvement program that included feedback for surgeons led to improved rates of optimal disease resection from 77 to 85 % (P = 0.157), improved rates of complete resection of all gross disease from 31 to 43 % (P = 0.188), despite an increase in the complexity of surgeries undertaken [19]. Two studies of wound infection
123
World J Surg
were included in this review. One study involving five surgeons, observed a drop in infection rates from 14 % without feedback to 10.3 % with feedback. To ascertain causality, cessation of feedback in this study led to a rebound of 12.8 % in infection rates which dropped to 8.3 % upon recommencement [17]. In the studies examined, those who utilised several interventions (including feedback) to improve performance appeared to have greater impact on one or more outcome indicators. Notably, feedback was demonstrated to have important financial consequences. For hepaticojejunostomy, implementation of feedback was associated with a decrease in overall hospital costs from $24,446 to $20,240 for the operation and subsequent hospital stay (P \ 0.01) [16] over a 3-year period. Similarly, the total cost of carotid endarterectomy and following management decreased from $13,344 to $9548; this was due to both reduced length of stay and decreased use of preoperative angiography [17]. These reductions in cost occurred in conjunction with improved outcomes.
Discussion Our review included seven studies assessing the impact of feedback of outcomes on surgical performance. Studies utilised feedback either as a periodic intervention over a defined period of time, or as a perioperative intervention. On the whole, although the available evidence was limited, feedback was generally found to be a powerful method for improving indicators of surgical performance, in addition to reducing costs. Strengths and limitations Several studies highlighted marked improvements in outcomes following implementation of feedback. On the surface, such findings appear encouraging. Taking hospital mortality as an example, a drop was observed across several studies, in one by 24 % [14] and another from 4.5 to 0.7 % [20]. However, the studies included were highly heterogeneous; conducted in different decades and clinical settings, for different procedures, using different methodologies and outcomes, with variability in the educational and technological experience of participating surgeons. This limited comparisons between studies and conclusions that could be drawn with regard to the factors influencing feedback. It could be argued that a finer definition of feedback may have generated more homogenous studies; however, we believe that our search strategy was robust. Further, the selected studies were identified independently by two authors, with a broader definition intentionally implemented to improve sample size and literature assessment.
123
The studies examined were also poorly designed; few delivered feedback interventions in isolation. For example, one study provided it as part of a package of measures, including quality improvement and site visits [14]. Another study implemented an entire clinical pathway, of which feedback was only one component [20]. Only one study sought to examine the impact of feedback cessation and reintroduction [17]. Thus, despite offering insight into the impact of feedback, in most studies, we were unable to delineate its true effect. Few studies adequately adjusted outcomes for patient-risk, clustering or the surgical learning curve, the latter of which may have led to improvements in performance over time, regardless of a feedback intervention. Given the lack of randomisation in any of the studies, it was also difficult to distinguish the relative impact of feedback and the Hawthorne effect. It is also important to consider the possibility of publication bias; all studies demonstrated that feedback resulted in improvements in performance (significant or not significant); however, it is possible that those showing the opposite result may not have been selected for publication. Finally, in our literature search, numerous studies were excluded as they focused on describing a means of feedback or how it could be implemented, rather than evaluating the impact of this feedback mechanism on performance. Implications Feedback is known to improve performance in certain situations [21] and professional practice [22] amongst clinicians, and improve competency in surgical tasks [23], and collaboration amongst surgeons [24]. Despite shortcomings in the design of several studies included in this review, it is clear that feedback can impact positively on surgical performance, patient outcomes and costs. However, the execution of feedback is critical to its effectiveness, and when conducted incorrectly it may result in little or no change. Discerning what feedback factors or variants confer the most substantial positive impact is a research priority. In our review, there was no significant difference in performance-improvement between studies utilising different frequencies of feedback. However, given the marked heterogeneity between the examined studies, it is challenging to draw a clear conclusion. Four studies provided benchmarking relative to peers, with three studies observing an improvement, suggesting this is likely to improve the effectiveness of feedback [19]. In the studies examined, those that utilised several interventions (including feedback) to improve performance appeared to have greater impact on one or more outcome indicators. Clinical meetings and forums for open discussion were often used in the studies included, and are known to aid improvements in professional practice [25]. Following dissemination of
World J Surg
individualised feedback, such meetings may offer a simple method to maximise the impact of feedback. From a research standpoint, the small number of studies included in this review highlights the need for more substantive investigation in this area. Therefore, when seeking to determine the impact of feedback on surgical performance and patient outcomes, we believe future studies should consider the effect of the source (oral/written), facilitator (expert/non-expert), timing (long-term outcomes, short-term e.g. weekly assessment, or perioperative) and duration of feedback (months/years). Further, studies should consider the surgeon’s involvement in feedback (either active or passive), the amount of detail included in feedback, and the opportunities available for discussion, correction and learning.
Conclusions The available literature suggests that feedback can improve surgical performance and outcomes; however, given the heterogeneity and limited number of studies, in addition to their non-randomised nature, it is difficult to draw clear conclusions from the literature with regard to the efficacy of feedback and the specific nuances required to optimise the impact of feedback. There is a clear need for more rigorous studies to determine how feedback of outcome data may impact performance, and whether this low-cost intervention has potential to benefit surgical practice. Financial Disclosure and Products Statement None of the participating authors has a conflicting financial interest related to the work detailed in this manuscript, nor do any of the authors maintain a financial stake in any product, device or drug cited in this report.
Appendix: search strategy Databases:
Ovid MEDLINE(R) Embase AMED PsycINFO
Search Strategy via Ovid SP Interface: ----------------------------------------------------------------------------------------------------1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
surgical skill$.ti,ab,sh. surgical performance.ti,ab,sh. surgical training.ti,ab,sh. surgical education.ti,ab,sh. surgical competenc$.ti,ab,sh. surgical proficiency.ti,ab,sh. surgical ability.ti,ab,sh. surgical expertise.ti,ab,sh. (surgeon$ adj4 performance).ti,ab. (surgeon$ adj4 experience$).ti,ab. (surgeon$ adj4 assess$).ti,ab. (surgeon$ adj4 skill$).ti,ab. (surgeon$ adj4 individual).ti,ab. (surgeon$ adj4 learning).ti,ab. exp Specialties, surgical/mt, st exp Surgical procedures, operative/st or/1-16 Feedback.ti,ab. Knowledge of results.ti,ab. Self-assessment.ti,ab. exp Employee Performance Appraisal/mt, st, sn exp Process Assessment/mt or/18-22 17 and 23 limit 24 to english language
3419 947 12533 1627 362 106 52 795 864 16159 3064 2773 1704 1094 21043 26882 85503 130446 2075 11951 1290 369 145256 1514 1451
----------------------------------------------------------------------------------------------------Database:
Cochrane Database of Systematic Reviews
Search Strategy via PubMed Interface: ----------------------------------------------------------------------------------------------------1
Cochrane Database Syst Rev[Journal] AND (Surgery OR Surgeon OR Surgical) AND (Training OR Performance OR Skill OR Skills OR Competence OR Competency OR Proficiency OR Ability OR Expertise OR Learning Curve) AND (Feedback OR Knowledge of Results (Psychology) OR Self-Assessment OR Education, Medical, Continuing/methods)
123
123
2005
Rodriguez et al. [18]
Country
USA
USA
Canada
USA
USA
Scotland
Spain
Reference
Aletti et al. [19]
Cornelius Olcott IV et al. [16]
Findlay et al. [15]
O’ Connor et al. [14]
Pitt et al. [20]
Reilly et al. [17]
Rodriguez et al. [18]
Study participants
1999 2002
Pitt et al. [20] Reilly et al. [17]
1
1
1
1
1
1
1
Hospital setting
1
1 1
1
0
0 0
0
1 (in part)
0
0
Retrospective
1
1 1
1
1
1
1
Gynaecology
General
Pancreatobiliary
Cardiothoracic
Neurosurgery, general and vascular
0
0 0
0
0
0
0
Observational
0
0 0
0
0
0
0
Hysterectomy
All general surgical procedures
Complex biliary surgery (choledochojejunostomy, cholangiojejunostomy, or hepaticojejunostomy)
Coronary artery bypass grafting
Carotid endarterectomy
Carotid endarterectomy
1
1 1
1
1
1
1
–
5
4 plus ‘several others’
23
10
10
–
–
–
66
6,638
291
105
105
–
–
107
6,488
184
188
132
No. of cases in period 2
2
3 2
2
4
5
2
166
249
181
222
154
No. of cases in period 4
Number of study periods
No. of cases in period 3
Before–after study
Before–after study Before–after study
Before–after study
Before–after study
Before–after study
Before–after study
Study design
No. of cases in period 1
Longitudinal
No. of participating surgeons
Cross-sectional
Surgical management of ovarian cancer
Surgical procedure
Experimental
Neurosurgery, general and vascular
Gynaecology
Surgeon speciality
1 (in part)
2002
1996
Findlay et al. [15]
1 1
2008
2000
Aletti et al. [19]
Cornelius Olcott IV et al. [16]
O’ Connor et al. [14]
Prospective
Year
Author
Study design
Data extraction from reviewed studies
135
No. of cases in period 5
5
5 4
6
3
6
6
980
2241
339
13,126
946
763
237
Total no. of cases
Total study duration (years)
World J Surg
0
1
1
1
1
Cornelius Olcott IV et al. [16]
Findlay et al. [15]
O’ Connor et al. [14]
0
0
1
Aletti et al. [19]
Measured outcome— appropriateness if surgery
Measured outcome— Mortality
Reference
0
1
1
1
1
Measured outcome— Morbidity
0
1
Reilly et al. [17]
Rodriguez et al. [18]
Outcomes and improvement
0
1
Pitt et al. [20]
0
O’ Connor et al. [14]
0
0
1
1
Findlay et al. [15]
0
0
0
0
0
Measured outcome— Infection rate
0
1
1
Aletti et al. [19]
Qualitative oral feedback by observer
Content of feedback = outcomes
Cornelius Olcott IV et al. [16]
Reference
Feedback
0
0
0
1
Measured outcome— operative time
0
0
0
1
1
0
1
1
–
–
1
1
1
0
0
0
1
1
1
–
–
1
0
1
1
0
0
1
0
30
30
30
30 (morbidity); 90 (mortality)
Length of follow up (days)
Feedback dissemination— clinical session/meeting
Measured outcome— Hospital charges
Feedback dissemination— written
Measured outcome— length of hospital stay
Other information disseminated with feedback/other parallel measures
–
–
1
1
1
0
–
–
Improvement in outcomes
1
0
0
1
0
1
1
0
0
0
1
0
0
0
0
No improvement in outcomes
Annually
Monthly
Once (after period ‘‘200 )
Three times per year
Essentially once a year
Annually
‘‘Periodically’’
Frequency of feedback
Improvement in outcomes (but not statistically significant)
Feedback provided with outcomes of overall hospital and/or other surgeons
Intervention following feedback
0
0
0
0
0
0
1
Feedback provided with literature outcomes
World J Surg
123
0 0 1 – 30 0 0 0 1 0
0
0
Reilly et al. [17]
Rodriguez et al. [18]
0
0 1 0 0
1 Pitt et al. [20]
0
1
0
0
0
30
‘‘Implementation of recommendations for change’’
0 0
Measured outcome— Mortality
1
0
0
0
1
–
–
1
0
References
Reference
Measured outcome— appropriateness if surgery
Measured outcome— Morbidity
Measured outcome— Infection rate
Measured outcome— operative time
Measured outcome— length of hospital stay
Measured outcome— Hospital charges
Length of follow up (days)
Intervention following feedback
Improvement in outcomes
Improvement in outcomes (but not statistically significant)
No improvement in outcomes
World J Surg
123
1. Fraher EP, Knapton A, Sheldon GF, Meyer A, Ricketts TC (2013) Projecting surgeon supply using a dynamic model. Ann Surg 257(5):867–872 2. Lyu H, Wick EC, Housman M, Freischlag J, Makary MA (2013) Patient satisfaction as a possible indicator of quality surgical care. JAMA Surg 148(4):362–367 3. Cooper M, Makary MA (2012) A comprehensive unit-based safety program (CUSP) in surgery: improving quality through transparency. Surg Clin North Am 92(1):51–63 4. Cleveland Clinic. Treatment Outcomes 2011 (http://my.cleve landclinic.org/heart/about-heart-vascular-institute/treatment-out comes.aspx) 5. Shahian DM, Jacobs JP, Edwards FH, Brennan JM, Dokholyan RS, Prager RL et al (2013) The society of thoracic surgeons national database. Heart 99:1494–1501. http://www.ncbi.nlm.nih. gov/pubmed/23335498 6. Godlee F (2012) Measure your team’s performance, and publish the results. BMJ 345:e4590 7. Royal College of Surgeons of England. RCS hails landmark publication of individual surgeons’ outcomes data. 2013 (https:// www.rcseng.ac.uk/news/rcs-hails-landmark-publication-of-indivi dual-surgeons2019-outcomes-data#.U-UoVRaOdss) 8. Khuri SF, Daley J, Henderson W, Hur K, Demakis J, Aust JB, Chong V, Fabri PJ, Gibbs JO, Grover F, Hammermeister K, Irvin G 3rd, McDonald G, Passaro E Jr, Phillips L, Scamman F, Spencer J, Stremple Jf (1998) The Department of Veterans Affairs’ NSQIP: the first national, validated, outcome-based, riskadjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg 228(4):491–507 9. van de Ridder JM, Stokking KM, McGaghie WC, ten Cate OT (2008) What is feedback in clinical education? Med Educ 42(2):189–197 10. Kluger A, DeNisi A (1996) The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 119(2):254–284 11. Bienstock JL, Katz NT, Cox SM, Hueppchen N, Erickson S, Puscheck EE et al (2007) To the point: medical education reviews—providing feedback. Am J Obstet Gynecol 196(6):508–513 12. Hewson MG, Little ML (1998) Giving feedback in medical education: verification of recommended techniques. J Gen Intern Med 13(2):111–116 13. Cantillon Peter, Sargeant Joan (2008) Giving feedback in clinical settings. BMJ 337:a1961 14. O’Connor GT, Plume SK, Olmstead EM, Morton JR, Maloney CT, Nugent WC et al (1996) A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. The Northern New England Cardiovascular Disease Study Group. JAMA 275(11):841–846 15. Findlay JM, Nykolyn L, Lubkey TB, Wong JH, Mouradian M, Senthilselvan A (2002) Auditing carotid endarterectomy: a regional experience. Can J Neurol Sci 29(4):326–332 16. Olcott CIV, Mitchell RS, Steinberg GK, Zarins CK (2000) Institutional peer review can reduce the risk and cost of carotid endarterectomy. Arch Surg 135(8):939–942 17. Reilly J, McIntosh J, Currie K (2002) Changing surgical practice through feedback of performance data. J Adv Nurs 38(6):607–614 18. Rodriguez JF, Trobo AR, Garcia MV, Martinez MJ, Millan CP, Vazquez MC et al (2006) The effect of performance feedback on wound infection rate in abdominal hysterectomy. Am J Infect Control 34(4):182–187
World J Surg 19. Aletti GD, Dowdy SC, Gostout BS, Jones MB, Stanhope RC, Wilson TO et al (2009) Quality improvement in the surgical approach to advanced ovarian cancer: the Mayo Clinic experience. J Am Coll Surg 208(4):614–620 20. Pitt HA, Murray KP, Bowman HM, Coleman J, Gordon TA, Yeo CJ et al (1999) Clinical pathway implementation improves outcomes for complex biliary surgery. Surgery 126(4):751–756 discussion 756–8 21. Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB (2006) Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7. Med Teach 28(2):117–128 22. Jamtvedt G, Young JM, Kristoffersen DT, O’Brien MA, Oxman AD (2006) Does telling people what they have been doing change
what they do? A systematic review of the effects of audit and feedback. Qual Saf Health Care 15(6):433–436 23. Rogers DA, Regehr G, Howdieshell TR, Yeh KA, Palm E (2000) The impact of external feedback on computer-assisted learning for surgical technical skill training. Am J Surg 179(4):341–343 24. Miller DC, Murtagh DS, Suh RS, Knapp PM, Schuster TG, Dunn RL et al (2011) Regional collaboration to improve radiographic staging practices among men with early stage prostate cancer. J Urol 186(3):844–849 25. Forsetlund L, Bjorndal A, Rashidian A, Jamtvedt G, O’Brien MA, Wolf F, et al. (2009) Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database Syst Rev (2):CD003030
123
SCIENTIFIC REVIEW
The Impact of Feedback of Surgical Outcome Data on Surgical Performance: A Systematic Review Mahiben Maruthappu • Abhishek Trehan • Ashton Barnett-Vanes • Peter McCulloch • Matthew J. Carty
Ó Socie´te´ Internationale de Chirurgie 2014
Abstract Background Increasing patient demands, costs and emphasis on safety have led to performance tracking of individual surgeons. Several methods of using these data, including feedback have been proposed. Our aim was to systematically review the impact of feedback of outcome data to surgeons on their performance. Study design MEDLINE, Embase, PsycINFO, AMED and the Cochrane Database of Systematic Reviews (from their inception to February 2013) were searched. Two reviewers independently reviewed citations using predetermined inclusion and exclusion criteria. Forty two data-points per study were extracted. Results The search strategy yielded 1,531 citations. Seven studies were eligible comprising 18,632 cases or procedures by 52 surgeons. Overall, feedback was found to be a powerful method for improving surgical outcomes or indicators of surgical performance, including reductions in hospital mortality after CABG of 24 % (P = 0.001), decreases of stroke and mortality following carotid endarterectomy from 5.2 to 2.3 %, improved ovarian cancer resection from 77 to 85 % (P = 0.157) and reductions in wound infection rates from 14 to 10.3 %. Improvements in performance occurred in concert with reduced costs: for hepaticojejunostomy, implementation of feedback was associated with a decrease in overall hospital costs from $24,446 to $20,240 (P \ 0.01). Similarly, total cost of carotid endarterectomy and following management decreased from $13,344 to $9548. Conclusions The available literature suggests that feedback can improve surgical performance and outcomes; however, given the heterogeneity and limited number of studies, in addition to their non-randomised nature, it is difficult to draw clear conclusions from the literature with regard to the efficacy of feedback and the specific nuances required to optimise the impact of feedback. There is a clear need for more rigorous studies to determine how feedback of outcome data may impact performance, and whether this low-cost intervention has potential to benefit surgical practice.
Mahiben Maruthappu and Abhishek Trehan have equally contributed to the development of this manuscript. M. Maruthappu A. Barnett-Vanes Imperial College London, London SW7 2AZ, UK e-mail: [email protected]
P. McCulloch Nuffield Department of Surgical Sciences, University of Oxford, Oxford OX3 9DU, UK
A. Trehan (&) Lincoln College, University of Oxford, Oxford OX1 3DR, UK e-mail: [email protected]
M. J. Carty Harvard Medical School, Boston, MA 02115, USA
123
World J Surg
Introduction
Methods
Surgery plays a key role in the delivery of modern clinical care [1]. Patients rightly expect routinely safe surgical care, and biomedical science is redefining patient experiences for a myriad of surgically treatable diseases. With advances in research, technology and training, expectations of surgical performance and outcomes have risen, as have levels of scrutiny. Recognition that patient satisfaction is only one indicator of surgical quality [2] and calls for greater transparency to improve safety [3] have fostered growing emphasis on the monitoring and evaluation of individual surgical performance, at both the institution [4] and national level [5]. Such trends are taking place across several countries; for example, in the United Kingdom, surgeons are expected to publish their individual performance data [6] as part of an era of ‘‘new openness in medicine’’ [7]. Several databases and sources of information have been developed to facilitate dissemination of performance data, such as online national and international clinical data registers. A priority for the surgical community is to ensure such monitoring translates into tangible and reproducible improvements in surgical performance, for the benefit of surgeons and ultimately patients. The American College of Surgeons’ National Surgical Quality Improvement Program which focuses on providing feedback to hospitals (rather than individual surgeons) has been shown to improve surgical outcomes [8]. In the UK, the impact of publication of individual surgical outcomes has yet to be assessed; further, the optimal means of using these data to provide individual feedback to surgeons is not clear. Such data could be used to provide specific information to surgeons on their observed performance compared to an expected standard, with the aim of improving subsequent performance and outcomes [9]. However, delivered inappropriately, feedback may be detrimental to performance [10]. Despite extensive literature on feedback strategies in medical education and clinical medicine [9, 11–13], few studies have examined this in the context of surgical performance. Although feedback is widely known to be a key component of surgical training, and individual surgeon data able to quantitatively support such feedback are available, and whilst several methods have been proposed to utilise outcome data for feedback, the impact of these methods on surgical performance and outcomes has not been reviewed. We performed a systematic review aiming to examine the feedback of outcome data and to synthesise the methods of feedback employed. For the purpose of this study, feedback was defined as an active process, comprising the measurement of individual surgeon outcome results, and subsequent relay of this information to the operating surgeon, through means such as open discussion, mentoring or coaching, with the aim of improving his or her performance.
Data sources and search strategy
123
The systematic review was conducted in accordance with PRISMA guidelines (http://www.prisma-statement.org). A comprehensive search was undertaken to determine the impact of feedback on surgical performance via the Ovid SP interface. The following databases were searched from inception to February 2013: MEDLINE, Embase, PsycINFO, AMED and the Cochrane Database of Systematic Reviews. We used two different domains of MeSH-terms and keywords combined by ‘‘AND’’, and within each domain the terms were combined by ‘‘OR’’. The first domain contained terms related to surgical skill and performance, whilst the second contained terms related to the impact of feedback. A detailed search strategy can be found in the Appendix. The search was limited to English publications with no other restrictions. Study selection Two reviewers independently reviewed citations and selected eligible studies based upon predetermined inclusion and exclusion criteria. Publications were selected for review if they satisfied the following inclusion criteria: article was published in a peer-reviewed journal; article described a study involving surgical patients; article investigated the impact of individual surgeon feedback (defined as either knowledge or relay of surgical outcomes) on surgical performance or outcome; article used a statistical unit that was patient or procedure focused (e.g. operative time or complication rate). The following exclusion criteria were applied to search results: the article was a conference abstract, editorial, letter, opinion, or review; the population studied was non-surgical (for example, radiology, pathology, diagnostic endoscopy); the article used qualitative rather than quantitative approaches (e.g. comments on video observation as opposed to duration, number of incisions); the study population was medical students; articles which proposed mechanisms of providing feedback, without quantitatively evaluating the impact of such mechanisms on postoperative outcomes. Two authors (MM, AT) independently examined all retrieved articles for inclusion. Any disagreement over inclusion or exclusion was resolved by consensus. Data extraction Forty two data-points per study were extracted using a predesigned data collection form including first author, year of publication, study aim, study type, study design (prospective, retrospective, experimental, observational, cross-sectional, longitudinal), study population, population setting (e.g. hospital), surgical speciality, surgical procedure analysed,
World J Surg
Studies identified from the databases n= 1531
References excluded: Conference abstract, editorial, letter, opinion, or review; n = 290 Non-surgical study; n = 93 Study described a means of feedback, but did not assess the impact of this feedback = 71 Inclusion criteria not met; n = 731
References retrieved for evaluation of abstracts n= 346 References excluded: Conference abstract, editorial, letter, opinion, or review; n=9 Non-surgical study; n = 15 Study described a means of feedback, but did not assess the impact of this feedback = 56 Medical student participants; n = 19 Inclusion criteria not met; n = 237
References retrieved for evaluation of full manuscripts n= 10
References excluded: Studies not focused on feedback of outcome data to surgeons; n = 3
References retrieved for review n= 7
Fig. 1 Search strategy
number of surgeons, types of feedback dissemination, content of feedback, frequency of feedback, measured outcomes (e.g. mortality, morbidity), length of followup, interventions following feedback and impact of interventions.
Results Study identification and selection Our search yielded 1,531 citations, of which seven studies were included in the final analysis comprising 18,632 cases or procedures by a total of 52 surgeons. Notably, 56 studies, although describing a means of feedback, did not assess it. A flow diagram of the search results is illustrated in Fig. 1.
Study characteristics Studies originated from the United States, Canada and Europe. All were before-after studies, covering a range of surgical specialties, including: cardiothoracic [14], neurosurgical [15, 16] and general surgery [17]. All included studies are shown in Table 1 and their basic characteristics are summarised in Table 2. The full data extraction from the studies can be found in the Appendix. Feedback dissemination and frequency Feedback was disseminated by a variety of methods. All seven studies examined delivered feedback as outcomespecific data [14–20], three studies accompanied this with
123
123
Year
2008
2000
2002
1996
1999
2002
2005
Reference
Aletti et al. [19]
Cornelius Olcott IV et al. [16]
Findlay et al. [15]
O’ Connor et al. [14]
Pitt et al. [20]
Reilly et al. [17]
Rodriguez et al. [18]
Spain
Scotland
USA
USA
Canada
USA
USA
Country
Gynaecology
General
Pancreatobiliary
Cardiothoracic
Neurosurgery, general and vascular
Neurosurgery, general and vascular
Gynaecology
Speciality
Hysterectomy
All general surgical procedures
Choledochojejunostomy, cholangiojejunostomy, or hepaticojejunostomy
Coronary artery bypass grafting
Carotid endarterectomy
Carotid endarterectomy
Surgical management of ovarian cancer
Surgical procedure
–
5
4 plus ‘‘several others’’
23
10
10
–
No. of participating surgeons
Table 1 Overview of studies included and impact of feedback on performance
980
2241
339
13,126
946
763
237
Total no. of cases
Before– after study
Before– after study
Before– after study
Before– after study
Before– after study
Before– after study
Before– after study
Study design
Infection rate
Infection rate
Mortality rate
Mortality rate
30 days
30 days
Not specified
30 days
30 days
2. 30 days
2. Mortality rate Mortality and stroke rate
1. 30 days
2. 3 months
2. Mortality rate 1. Stroke rate
1. 30 days
Length of follow up for primary outcome measure
1. Morbidity rate
Primary outcome measure
10.7 %
14 %
4.5 %
4.75 %
5.2 %
2. 3.8 %
1. 3.8 %
2. 8 %
1. 21 %
Primary outcome measure: prefeedback
6.0 %
10.3 %
0.7 %
3.61 %
2.3 %
2. 0 %
1. 0 %
2. 6 %
1. 20 %
Primary outcome measure: postfeedback
-43.9 %
-26 %
-85 %
-24 %
-56 %
%2. -100%
1. -100
2. -25 %
1. -5 %
Impact of feedback on primary outcome
Not provided
Not statistically significant
\0.05
0.001
Not provided
Not provided
2. 0.475
1. 0.819
P value
World J Surg
World J Surg
Feedback contents
Table 2 Basic characteristics of studies included Study characteristics
No. of studies
References
All studies
7
[14–20]
Written
1
[15]
Clinical meeting
1
[17]
Written and clinical meeting
3
[14, 16, 19]
Not stated
2
[18, 20]
Feedback dissemination
Feedback contents Outcomes
7
[14–20]
Benchmarking relative to peers
4
[14, 16, 18, 19]
Comparable literature-reported figures
1
[19]
Feedback frequency Single time-point
1
[20]
Monthly
1
[17]
Tri-annual Annually
1 3
[14] [15, 16, 18]
Not stated
1
[19]
Length of follow up 30 days
5
[14–18]
30 days (plus 3 months for mortality)
1
[19]
Not stated
1
[20]
Any outcome
6/7
[14–18, 20]a
Mortality
4/5
[14–16, 20]
Morbidity Infection rate
2/4 2/2
[16, 18] [17]a [18]
No. of studies reporting improvement in outcome/no. of studies assessing that outcome
a
Length of stay
2/3
[16, 20]
Hospital charges
2/2
[16, 20]
Not statistically significant improvement
clinical meetings to provide a forum for discussion and improvement [14, 16, 19]. One study mailed surgeons their results [15]. Frequency of feedback varied between studies, from a single time-point [20], monthly [17], three times per year [14] or annual intervals [15, 16, 18]. There was no difference in the impact of feedback between studies utilising different frequencies. However, one study did not specifically define the frequency, and this study saw no significant improvement in performance [19]. Four studies provided benchmarking relative to peers, with three studies observing an improvement, and one study reporting no significant improvement [19].
Feedback in these studies was provided in the form of patient outcome data, the nature and extent of which varied depending on the study objective. Five studies provided surgeons with morbidity or mortality rates [14–16, 19, 20]. Two studies focusing on wound contamination provided surgeons with feedback on wound infection rates, one in the context of clean elective surgery [17], and the other for abdominal hysterectomy [18]. Four of the studies examined provided feedback together with outcome benchmarking, either in comparison to their peers [18, 19] or institutions [16], and additionally to regional statistics [14]. One of these studies also included comparison with results from the recent literature [19]. Three studies provided additional operative details to surgeons, including the calculated appropriateness of the surgery undertaken [15], complication rates [14, 15], surgical complexity and the presence of residual disease [19]. One study provided additional non-clinical data to surgeons, such as hospital charges [20]. Impact of feedback Table 1 summarises the impact of feedback on surgical performance in the analysed studies. All studies noted improvements in their primary outcomes following implementation of measures that included feedback (i.e. the studies demonstrated improvements in outcomes they aimed to improve). One study involving 23 surgeons performing coronary artery bypass graft surgery (CABG) observed a 24 % reduction (P = 0.001) in hospital mortality compared to expected figures in the post-intervention period, which included feedback, quality improvement and site visits [14]. One study, including 10 surgeons performing carotid endarterectomy (CEA) observed the number of stroke and death complications decline over the study period, from an initial overall rate of 5.2–2.3 % [15]. Another study of CEA involving 10 surgeons observed a decrease in stroke and mortality after providing surgeons with feedback on morbidity and mortality rates; however, there was no decline in the incidence of postoperative myocardial infarction [16]. In a study investigating mortality in complex biliary surgery after the implementation of a clinical pathway that included outcome feedback to surgeons, a decrease in hospital mortality from 4.5 to 0.7 % (P \ 0.05) was observed [20]. In one study of ovarian cancer cases, implementation of a quality improvement program that included feedback for surgeons led to improved rates of optimal disease resection from 77 to 85 % (P = 0.157), improved rates of complete resection of all gross disease from 31 to 43 % (P = 0.188), despite an increase in the complexity of surgeries undertaken [19]. Two studies of wound infection
123
World J Surg
were included in this review. One study involving five surgeons, observed a drop in infection rates from 14 % without feedback to 10.3 % with feedback. To ascertain causality, cessation of feedback in this study led to a rebound of 12.8 % in infection rates which dropped to 8.3 % upon recommencement [17]. In the studies examined, those who utilised several interventions (including feedback) to improve performance appeared to have greater impact on one or more outcome indicators. Notably, feedback was demonstrated to have important financial consequences. For hepaticojejunostomy, implementation of feedback was associated with a decrease in overall hospital costs from $24,446 to $20,240 for the operation and subsequent hospital stay (P \ 0.01) [16] over a 3-year period. Similarly, the total cost of carotid endarterectomy and following management decreased from $13,344 to $9548; this was due to both reduced length of stay and decreased use of preoperative angiography [17]. These reductions in cost occurred in conjunction with improved outcomes.
Discussion Our review included seven studies assessing the impact of feedback of outcomes on surgical performance. Studies utilised feedback either as a periodic intervention over a defined period of time, or as a perioperative intervention. On the whole, although the available evidence was limited, feedback was generally found to be a powerful method for improving indicators of surgical performance, in addition to reducing costs. Strengths and limitations Several studies highlighted marked improvements in outcomes following implementation of feedback. On the surface, such findings appear encouraging. Taking hospital mortality as an example, a drop was observed across several studies, in one by 24 % [14] and another from 4.5 to 0.7 % [20]. However, the studies included were highly heterogeneous; conducted in different decades and clinical settings, for different procedures, using different methodologies and outcomes, with variability in the educational and technological experience of participating surgeons. This limited comparisons between studies and conclusions that could be drawn with regard to the factors influencing feedback. It could be argued that a finer definition of feedback may have generated more homogenous studies; however, we believe that our search strategy was robust. Further, the selected studies were identified independently by two authors, with a broader definition intentionally implemented to improve sample size and literature assessment.
123
The studies examined were also poorly designed; few delivered feedback interventions in isolation. For example, one study provided it as part of a package of measures, including quality improvement and site visits [14]. Another study implemented an entire clinical pathway, of which feedback was only one component [20]. Only one study sought to examine the impact of feedback cessation and reintroduction [17]. Thus, despite offering insight into the impact of feedback, in most studies, we were unable to delineate its true effect. Few studies adequately adjusted outcomes for patient-risk, clustering or the surgical learning curve, the latter of which may have led to improvements in performance over time, regardless of a feedback intervention. Given the lack of randomisation in any of the studies, it was also difficult to distinguish the relative impact of feedback and the Hawthorne effect. It is also important to consider the possibility of publication bias; all studies demonstrated that feedback resulted in improvements in performance (significant or not significant); however, it is possible that those showing the opposite result may not have been selected for publication. Finally, in our literature search, numerous studies were excluded as they focused on describing a means of feedback or how it could be implemented, rather than evaluating the impact of this feedback mechanism on performance. Implications Feedback is known to improve performance in certain situations [21] and professional practice [22] amongst clinicians, and improve competency in surgical tasks [23], and collaboration amongst surgeons [24]. Despite shortcomings in the design of several studies included in this review, it is clear that feedback can impact positively on surgical performance, patient outcomes and costs. However, the execution of feedback is critical to its effectiveness, and when conducted incorrectly it may result in little or no change. Discerning what feedback factors or variants confer the most substantial positive impact is a research priority. In our review, there was no significant difference in performance-improvement between studies utilising different frequencies of feedback. However, given the marked heterogeneity between the examined studies, it is challenging to draw a clear conclusion. Four studies provided benchmarking relative to peers, with three studies observing an improvement, suggesting this is likely to improve the effectiveness of feedback [19]. In the studies examined, those that utilised several interventions (including feedback) to improve performance appeared to have greater impact on one or more outcome indicators. Clinical meetings and forums for open discussion were often used in the studies included, and are known to aid improvements in professional practice [25]. Following dissemination of
World J Surg
individualised feedback, such meetings may offer a simple method to maximise the impact of feedback. From a research standpoint, the small number of studies included in this review highlights the need for more substantive investigation in this area. Therefore, when seeking to determine the impact of feedback on surgical performance and patient outcomes, we believe future studies should consider the effect of the source (oral/written), facilitator (expert/non-expert), timing (long-term outcomes, short-term e.g. weekly assessment, or perioperative) and duration of feedback (months/years). Further, studies should consider the surgeon’s involvement in feedback (either active or passive), the amount of detail included in feedback, and the opportunities available for discussion, correction and learning.
Conclusions The available literature suggests that feedback can improve surgical performance and outcomes; however, given the heterogeneity and limited number of studies, in addition to their non-randomised nature, it is difficult to draw clear conclusions from the literature with regard to the efficacy of feedback and the specific nuances required to optimise the impact of feedback. There is a clear need for more rigorous studies to determine how feedback of outcome data may impact performance, and whether this low-cost intervention has potential to benefit surgical practice. Financial Disclosure and Products Statement None of the participating authors has a conflicting financial interest related to the work detailed in this manuscript, nor do any of the authors maintain a financial stake in any product, device or drug cited in this report.
Appendix: search strategy Databases:
Ovid MEDLINE(R) Embase AMED PsycINFO
Search Strategy via Ovid SP Interface: ----------------------------------------------------------------------------------------------------1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
surgical skill$.ti,ab,sh. surgical performance.ti,ab,sh. surgical training.ti,ab,sh. surgical education.ti,ab,sh. surgical competenc$.ti,ab,sh. surgical proficiency.ti,ab,sh. surgical ability.ti,ab,sh. surgical expertise.ti,ab,sh. (surgeon$ adj4 performance).ti,ab. (surgeon$ adj4 experience$).ti,ab. (surgeon$ adj4 assess$).ti,ab. (surgeon$ adj4 skill$).ti,ab. (surgeon$ adj4 individual).ti,ab. (surgeon$ adj4 learning).ti,ab. exp Specialties, surgical/mt, st exp Surgical procedures, operative/st or/1-16 Feedback.ti,ab. Knowledge of results.ti,ab. Self-assessment.ti,ab. exp Employee Performance Appraisal/mt, st, sn exp Process Assessment/mt or/18-22 17 and 23 limit 24 to english language
3419 947 12533 1627 362 106 52 795 864 16159 3064 2773 1704 1094 21043 26882 85503 130446 2075 11951 1290 369 145256 1514 1451
----------------------------------------------------------------------------------------------------Database:
Cochrane Database of Systematic Reviews
Search Strategy via PubMed Interface: ----------------------------------------------------------------------------------------------------1
Cochrane Database Syst Rev[Journal] AND (Surgery OR Surgeon OR Surgical) AND (Training OR Performance OR Skill OR Skills OR Competence OR Competency OR Proficiency OR Ability OR Expertise OR Learning Curve) AND (Feedback OR Knowledge of Results (Psychology) OR Self-Assessment OR Education, Medical, Continuing/methods)
123
123
2005
Rodriguez et al. [18]
Country
USA
USA
Canada
USA
USA
Scotland
Spain
Reference
Aletti et al. [19]
Cornelius Olcott IV et al. [16]
Findlay et al. [15]
O’ Connor et al. [14]
Pitt et al. [20]
Reilly et al. [17]
Rodriguez et al. [18]
Study participants
1999 2002
Pitt et al. [20] Reilly et al. [17]
1
1
1
1
1
1
1
Hospital setting
1
1 1
1
0
0 0
0
1 (in part)
0
0
Retrospective
1
1 1
1
1
1
1
Gynaecology
General
Pancreatobiliary
Cardiothoracic
Neurosurgery, general and vascular
0
0 0
0
0
0
0
Observational
0
0 0
0
0
0
0
Hysterectomy
All general surgical procedures
Complex biliary surgery (choledochojejunostomy, cholangiojejunostomy, or hepaticojejunostomy)
Coronary artery bypass grafting
Carotid endarterectomy
Carotid endarterectomy
1
1 1
1
1
1
1
–
5
4 plus ‘several others’
23
10
10
–
–
–
66
6,638
291
105
105
–
–
107
6,488
184
188
132
No. of cases in period 2
2
3 2
2
4
5
2
166
249
181
222
154
No. of cases in period 4
Number of study periods
No. of cases in period 3
Before–after study
Before–after study Before–after study
Before–after study
Before–after study
Before–after study
Before–after study
Study design
No. of cases in period 1
Longitudinal
No. of participating surgeons
Cross-sectional
Surgical management of ovarian cancer
Surgical procedure
Experimental
Neurosurgery, general and vascular
Gynaecology
Surgeon speciality
1 (in part)
2002
1996
Findlay et al. [15]
1 1
2008
2000
Aletti et al. [19]
Cornelius Olcott IV et al. [16]
O’ Connor et al. [14]
Prospective
Year
Author
Study design
Data extraction from reviewed studies
135
No. of cases in period 5
5
5 4
6
3
6
6
980
2241
339
13,126
946
763
237
Total no. of cases
Total study duration (years)
World J Surg
0
1
1
1
1
Cornelius Olcott IV et al. [16]
Findlay et al. [15]
O’ Connor et al. [14]
0
0
1
Aletti et al. [19]
Measured outcome— appropriateness if surgery
Measured outcome— Mortality
Reference
0
1
1
1
1
Measured outcome— Morbidity
0
1
Reilly et al. [17]
Rodriguez et al. [18]
Outcomes and improvement
0
1
Pitt et al. [20]
0
O’ Connor et al. [14]
0
0
1
1
Findlay et al. [15]
0
0
0
0
0
Measured outcome— Infection rate
0
1
1
Aletti et al. [19]
Qualitative oral feedback by observer
Content of feedback = outcomes
Cornelius Olcott IV et al. [16]
Reference
Feedback
0
0
0
1
Measured outcome— operative time
0
0
0
1
1
0
1
1
–
–
1
1
1
0
0
0
1
1
1
–
–
1
0
1
1
0
0
1
0
30
30
30
30 (morbidity); 90 (mortality)
Length of follow up (days)
Feedback dissemination— clinical session/meeting
Measured outcome— Hospital charges
Feedback dissemination— written
Measured outcome— length of hospital stay
Other information disseminated with feedback/other parallel measures
–
–
1
1
1
0
–
–
Improvement in outcomes
1
0
0
1
0
1
1
0
0
0
1
0
0
0
0
No improvement in outcomes
Annually
Monthly
Once (after period ‘‘200 )
Three times per year
Essentially once a year
Annually
‘‘Periodically’’
Frequency of feedback
Improvement in outcomes (but not statistically significant)
Feedback provided with outcomes of overall hospital and/or other surgeons
Intervention following feedback
0
0
0
0
0
0
1
Feedback provided with literature outcomes
World J Surg
123
0 0 1 – 30 0 0 0 1 0
0
0
Reilly et al. [17]
Rodriguez et al. [18]
0
0 1 0 0
1 Pitt et al. [20]
0
1
0
0
0
30
‘‘Implementation of recommendations for change’’
0 0
Measured outcome— Mortality
1
0
0
0
1
–
–
1
0
References
Reference
Measured outcome— appropriateness if surgery
Measured outcome— Morbidity
Measured outcome— Infection rate
Measured outcome— operative time
Measured outcome— length of hospital stay
Measured outcome— Hospital charges
Length of follow up (days)
Intervention following feedback
Improvement in outcomes
Improvement in outcomes (but not statistically significant)
No improvement in outcomes
World J Surg
123
1. Fraher EP, Knapton A, Sheldon GF, Meyer A, Ricketts TC (2013) Projecting surgeon supply using a dynamic model. Ann Surg 257(5):867–872 2. Lyu H, Wick EC, Housman M, Freischlag J, Makary MA (2013) Patient satisfaction as a possible indicator of quality surgical care. JAMA Surg 148(4):362–367 3. Cooper M, Makary MA (2012) A comprehensive unit-based safety program (CUSP) in surgery: improving quality through transparency. Surg Clin North Am 92(1):51–63 4. Cleveland Clinic. Treatment Outcomes 2011 (http://my.cleve landclinic.org/heart/about-heart-vascular-institute/treatment-out comes.aspx) 5. Shahian DM, Jacobs JP, Edwards FH, Brennan JM, Dokholyan RS, Prager RL et al (2013) The society of thoracic surgeons national database. Heart 99:1494–1501. http://www.ncbi.nlm.nih. gov/pubmed/23335498 6. Godlee F (2012) Measure your team’s performance, and publish the results. BMJ 345:e4590 7. Royal College of Surgeons of England. RCS hails landmark publication of individual surgeons’ outcomes data. 2013 (https:// www.rcseng.ac.uk/news/rcs-hails-landmark-publication-of-indivi dual-surgeons2019-outcomes-data#.U-UoVRaOdss) 8. Khuri SF, Daley J, Henderson W, Hur K, Demakis J, Aust JB, Chong V, Fabri PJ, Gibbs JO, Grover F, Hammermeister K, Irvin G 3rd, McDonald G, Passaro E Jr, Phillips L, Scamman F, Spencer J, Stremple Jf (1998) The Department of Veterans Affairs’ NSQIP: the first national, validated, outcome-based, riskadjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg 228(4):491–507 9. van de Ridder JM, Stokking KM, McGaghie WC, ten Cate OT (2008) What is feedback in clinical education? Med Educ 42(2):189–197 10. Kluger A, DeNisi A (1996) The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 119(2):254–284 11. Bienstock JL, Katz NT, Cox SM, Hueppchen N, Erickson S, Puscheck EE et al (2007) To the point: medical education reviews—providing feedback. Am J Obstet Gynecol 196(6):508–513 12. Hewson MG, Little ML (1998) Giving feedback in medical education: verification of recommended techniques. J Gen Intern Med 13(2):111–116 13. Cantillon Peter, Sargeant Joan (2008) Giving feedback in clinical settings. BMJ 337:a1961 14. O’Connor GT, Plume SK, Olmstead EM, Morton JR, Maloney CT, Nugent WC et al (1996) A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. The Northern New England Cardiovascular Disease Study Group. JAMA 275(11):841–846 15. Findlay JM, Nykolyn L, Lubkey TB, Wong JH, Mouradian M, Senthilselvan A (2002) Auditing carotid endarterectomy: a regional experience. Can J Neurol Sci 29(4):326–332 16. Olcott CIV, Mitchell RS, Steinberg GK, Zarins CK (2000) Institutional peer review can reduce the risk and cost of carotid endarterectomy. Arch Surg 135(8):939–942 17. Reilly J, McIntosh J, Currie K (2002) Changing surgical practice through feedback of performance data. J Adv Nurs 38(6):607–614 18. Rodriguez JF, Trobo AR, Garcia MV, Martinez MJ, Millan CP, Vazquez MC et al (2006) The effect of performance feedback on wound infection rate in abdominal hysterectomy. Am J Infect Control 34(4):182–187
World J Surg 19. Aletti GD, Dowdy SC, Gostout BS, Jones MB, Stanhope RC, Wilson TO et al (2009) Quality improvement in the surgical approach to advanced ovarian cancer: the Mayo Clinic experience. J Am Coll Surg 208(4):614–620 20. Pitt HA, Murray KP, Bowman HM, Coleman J, Gordon TA, Yeo CJ et al (1999) Clinical pathway implementation improves outcomes for complex biliary surgery. Surgery 126(4):751–756 discussion 756–8 21. Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB (2006) Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7. Med Teach 28(2):117–128 22. Jamtvedt G, Young JM, Kristoffersen DT, O’Brien MA, Oxman AD (2006) Does telling people what they have been doing change
what they do? A systematic review of the effects of audit and feedback. Qual Saf Health Care 15(6):433–436 23. Rogers DA, Regehr G, Howdieshell TR, Yeh KA, Palm E (2000) The impact of external feedback on computer-assisted learning for surgical technical skill training. Am J Surg 179(4):341–343 24. Miller DC, Murtagh DS, Suh RS, Knapp PM, Schuster TG, Dunn RL et al (2011) Regional collaboration to improve radiographic staging practices among men with early stage prostate cancer. J Urol 186(3):844–849 25. Forsetlund L, Bjorndal A, Rashidian A, Jamtvedt G, O’Brien MA, Wolf F, et al. (2009) Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database Syst Rev (2):CD003030
123
