Research Report
Teaching Medical Error Disclosure to Residents Using Patient-Centered Simulation Training Sara Sukalich, MD, John O. Elliott, PhD, MPH, and Gina Ruffner, EMT-P
Abstract Purpose To determine whether a standardized patient encounter and self-guided tutorial would improve first-year residents’ selfefficacy for disclosing medical errors. Method In 2011, 55 first-year residents participated in a simulation in which they disclosed an error to a standardized patient playing the part of a family member. Residents completed the simulation twice, four weeks apart, and completed presession knowledge and self-efficacy (based on the Accreditation Council for Graduate Medical Education [ACGME] core competencies) assessments and repeated the self-efficacy assessment
D
isclosure, the process of bringing to light an unintended outcome, is essential for maintaining trust between physicians, patients, and their family members. Surveys of patients revealed that a large majority preferred to be informed immediately of errors, even minor ones.1,2 Patients also expect to be given more information about an unintended injury during treatment than doctors believe should be given.3
Dr. Sukalich is director, Department of Medical Education, OhioHealth Riverside Methodist Hospital, Columbus, Ohio. Dr. Elliott is research specialist, Department of Medical Education, OhioHealth Riverside Methodist Hospital, Columbus, Ohio. Ms. Ruffner is simulation center manager, Center for Medical Education and Innovation (CME+I), OhioHealth Riverside Methodist Hospital, Columbus, Ohio. Correspondence should be addressed to Dr. Sukalich, Department of Medical Education, 3535 Olentangy River Rd., Columbus, OH 43214-3998; telephone: (614) 566-2426; e-mail: [email protected]. Acad Med. 2014;89:136–143. First published online November 25, 2013 doi: 10.1097/ACM.0000000000000046 Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A173.
136
after the sessions. Residents reviewed the videos of their encounters either alone (self-debrief) or with a faculty observer (faculty debrief). Between sessions, they completed a self-paced learning tutorial. Two external faculty also rated the residents’ performances using videos of the encounters. Results Residents’ self-efficacy significantly increased from a Session 1 pretest mean (standard deviation) score of 119.6 (26.6) to a Session 2 posttest score of 150.3 (24.9) for all ACGME competencies (P < .001, Cohen's d = 1.19). The external reviewers’ ratings provided additional, objective support for
Furthermore, medical errors are common; previous research found that 62% of trainees and 88% of faculty physicians reported making medical mistakes.4 One study found that over 90% of physicians and trainees (medical students and residents) reported they would or should disclose a hypothetical error (major, minor, and those causing no harm); however, only 41% reported actually doing so in real instances.5 Not only is the act of disclosing a medical error the “right thing to do,” but some hospital accreditation requirements and state laws also mandate it.6 In an effort to increase health care professionals’ willingness to embrace disclosure, the National Quality Forum put forward an evidence-based safe practice guideline on the disclosure of serious unanticipated outcomes.7 This guideline recommends providing an explanation to the patient and his or her family about what happened, potential implications or consequences of the error, a commitment to investigate what went wrong, feedback regarding the findings of the investigation, and an apology or expression of regret from the physician. However, throughout training, physicians are taught that the goal
residents’ improvement on questions assessing ACGME competencies (P = .001). Comparisons of the self-efficacy of residents in the self-debrief versus faculty debrief groups yielded no significant differences on any ACGME competencies. Conclusions Timely, explicit, and empathetic disclosure of medical errors to patients and family is essential to maintaining trust and is an important part of patient-centered medical care. This intervention easily could be replicated in other settings and is applicable to many members of the health care team, not just to residents.
in providing care is to improve health, not cause harm, which inherently affects their error disclosure behaviors. Other barriers to fully disclosing an error include fear of potential malpractice litigation in admitting a mistake, the culture of medicine, and the psychological impact of facing mistakes and apologizing for them.8–11 The art of disclosure and the use of effective, open communication are skills not readily taught in most undergraduate medical education environments. Thus, a gap exists in residents’ competency when they enter their graduate medical training, a gap that often continues as they move into medical practice. The majority of prior research on the disclosure of medical errors has focused on health professionals primarily through the use of surveys of the attitudes and beliefs of health sciences students,12 medical students,13 and residents/ physicians.5,14–18 One investigation used written scenarios to assess how residents would disclose medical errors.19 In terms of the methods for teaching error disclosure, previous studies have
Academic Medicine, Vol. 89, No. 1 / January 2014
Research Report
described the use of didactic educational modules,12 seminars,20 DVDs of patients followed by discussion,4 and roleplaying between faculty and students.21 Although the use of standardized patient encounters is a well-established method for assessing clinical skills,22 very few studies have examined a standardized patient method for teaching error disclosure to physicians or residents.12,23,24 Therefore, we argue in favor of creating a robust training and competency assessment program to teach residents how to disclose medical errors and improve communication skills that includes a standardized patient encounter and is based on the National Quality Forum’s disclosure guidelines.7 The main goal of this study was to determine whether a standardized patient encounter scenario, followed by a selfguided tutorial, would improve residents’ self-efficacy for disclosing medical errors. We also were interested in exploring whether a self-debriefing procedure would be as effective as a faculty-led debriefing. To further establish the effectiveness of this training method, we sought to determine whether external faculty members could objectively detect improvements in residents’ error disclosure skills via the review of randomly assigned, video-recorded, standardized patient encounters. Method
In March 2011, we invited all 55 postgraduate year (PGY) 1 residents at the OhioHealth Riverside Methodist Hospital to participate in our study via announcements at program-specific meetings as well as in a letter from the principal investigator (S.S.) and the hospital’s physician vice president of medical education. The residents received no incentive to participate, and we informed them, as required by the OhioHealth institutional review board, that data collected as part of our project would be considered educational research. We deidentified all data that we collected through the use of a studyspecific identification number that we could use to link all assessments. The OhioHealth institutional review board approved our project as exempt. Study design First, we developed the scenario to use in our study. Next, we trained
Academic Medicine, Vol. 89, No. 1 / January 2014
the standardized patients and the internal faculty who would be debriefing residents. The residents then completed the simulation scenario twice, approximately four weeks apart. Between sessions, they completed an online, self-paced learning tutorial on the art of disclosure (15 minutes in length). Before completing each session, the residents took a medical knowledge test and a self-efficacy assessment. After completing each session, they repeated the self-efficacy assessment. The educational material that we compiled for the self-paced learning tutorial included best practice guidelines and references from the National Quality Forum guidelines on disclosure.7 As part of the tutorial, the residents also reviewed the OhioHealth Riverside Methodist Hospital’s standard policy and procedures for the disclosure of unanticipated events. We required all residents to provide a certificate of completion for the tutorial before they started the second session. The first session took place in March 2011 and the second in April 2011. We videotaped all sessions for analysis. After all residents completed the sessions, two trained external faculty members reviewed and evaluated the residents’ error disclosure skills, using video recordings of the encounters. Scenario The research team developed the scenario for this project. It was meant to offer a realistic situation that a PGY 1 resident might be required to address (see Appendix 1). We did not pilot the scenario prior to the study but did ask other faculty involved in simulation education at our institution to review it for face validity. Residents read the scenario and then immediately entered the simulation with the standardized patient at our in-house simulation center (Center for Medical Education and Innovation [CME+I]). Each encounter lasted roughly 10 minutes. Training To ensure that the two standardized patients (acting as a family member) had the necessary qualifications for performance, they underwent a one-hour instruction session, during which we detailed the background of the scenario and reviewed our expectations of them,
and a one-hour practice session with a CME+I staff member. As part of the training, we provided them with a list of 15 questions that we expected them to ask during each encounter with a resident (see Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/A173). This training took place in March 2011. As part of the training process, we provided the participating internal faculty members with an example of a videotaped scenario focusing on a disclosure conversation for them to review, a copy of our hospital’s policies/ procedures for error disclosure, a printed copy of the scenario, a medical debriefing guide used in our simulation lab for other resident training, a copy of the assessment questions on which we would evaluate the residents, and a copy of the 2008 Canadian Medical Protective Association report, titled “Communicating With Your Patient About Harm: Disclosure of Adverse Events.”25 Ten faculty members, representing family medicine, internal medicine, obstetrics–gynecology, and general surgery, participated. Assessments Prior to each session, the residents completed an online medical knowledge test via a computer terminal in the CME+I. The test included 21 true/false questions covering information in our hospital’s policies for error disclosure, which are based on best practices. To quantify the skills needed to provide full disclosure of medical errors, we also created an online resident self-efficacy assessment based on the Accreditation Council for Graduate Medical Education (ACGME) core competencies (medical knowledge, interpersonal and communication skills, patient care/clinical skills, professionalism, systems-based practice, and practice-based learning and improvement). These six core competencies provide a set of standard principles by which residents are to be evaluated as well as a general framework for curriculum development.26,27 They also have been used for teaching risk management to residents.10,28 The self-efficacy assessment included 21 questions (see Appendix 2). The residents rated their confidence in performing aspects of error disclosure on a ninepoint Likert scale (0 = not very confident, 9 = very confident). Self-efficacy, a well-
137
Research Report
established social learning theory,29 has been used to evaluate the high-fidelity training of surgery residents in pediatric trauma30 as well as in the disclosure of medical errors.20 The residents completed the self-efficacy assessment via a computer terminal in the CME+I. Debriefing After each simulation session, each resident completed a video review of the encounter. To assess the impact of the debriefing process, we randomly assigned the residents to review the video alone (n = 29, “self-debrief ”) or with a faculty member (n = 26, “faculty debrief ”). We instructed residents who completed the self-debrief to watch their video and self-reflect on suggested items like “What went well?” “What would I change?” and “How might this impact the way I treat patients?” Residents remained in their assigned debriefing group for both sessions. The self-debrief sessions lasted approximately 10 minutes. Residents in the faculty debrief group watched their video with one faculty member immediately following each standardized patient encounter. Faculty members provided verbal feedback during the video review. The faculty debrief sessions lasted approximately 15 minutes. External faculty reviewers To provide more objective ratings of the residents’ skills, two trained faculty reviewers outside our program watched all of the videotaped encounters during June and July 2011. A computer technician from the CME+I randomly sorted the videotaped encounters so that the reviewers were blinded to which session (first or second) they were viewing. We provided DVD copies of the videotaped encounters with a deidentified study-specific number to the external reviewers. After the external reviewers conducted their assessments, we provided the randomization key to the statistician to properly link each resident’s Session 1 and 2 data. The paper-based external faculty assessment tool was similar to the residents’ self-efficacy assessment but was revised to include anchors for each number on the Likert scale. We also removed the two questions rating the documentation in the patient’s medical
138
record, because they were not directly observable in the simulation, resulting in a 19-question assessment.
we developed has very high reliability (Cronbach α = 0.98; 95% confidence interval [CI]: 0.97–0.99; P < .001).
The principal investigator (S.S.) trained the external reviewers in an effort to address potential issues with interrater reliability. As part of the training, each external reviewer watched and assessed a small sample of 10 videos to assess initial interrater reliability. The intraclass correlations for these assessments ranged from −0.33 to 0.82, suggesting reasonable overall agreement.
Intraclass correlations also indicated moderate agreement between the two external reviewers (Session 1: α = 0.40; 95% CI: 0.31–0.53; P < .001; Session 2: α = 0.44; 95% CI: 0.34–0.56; P < .001). Because of this level of agreement, we averaged the scores for Reviewer 1 and Reviewer 2 for each session for analyses via paired t tests. Finally, we created a total score on the external reviewers’ assessment by summing the ratings of all the questions. We conducted all analyses using SPSS version 19.0 (IBM Corp., Armonk, New York).
Statistical analysis We calculated each resident’s knowledge assessment total score by adding up the total number of correct answers. We used paired t tests to compare knowledge assessment total scores and χ2 tests to compare individual responses on the knowledge assessment between Sessions 1 and 2. We summarized each ACGME core competency domain in the self-efficacy assessment using subscale scores consistent with the ACGME core competency guidelines.31 We summed the ratings of all the questions to create a total score on the self-efficacy assessment. We compared the residents’ responses to the self-efficacy assessments and external reviewers’ ratings via paired t tests, which examine individual change on the basis of matched data. We used a Bonferroni correction method to adjust for multiple comparisons in the Session 1 pre–post, Session 2 pre–post, and Session 1 preSession 2 post analyses. Residents had to complete all assessments to be included in the data analysis of each session. If a resident did not complete an assessment because he or she had to leave for clinical duties or a computer glitch caused the loss of the assessment data, we deleted the data listwise from the analysis for that session. Thus, we report both the numerator and denominator for all percentages in the Results to reflect that some calculations included a different number of participants. Because we found no formal assessments of the skills for the disclosure of medical errors in an extensive review of the literature, we created our own measurement scale (see Appendix 2). Intraclass correlations demonstrated that the residents’ self-efficacy assessment that
Results
Fifty-five PGY 1 residents participated: 36 (65.5%) men and 19 (34.5%) women. Four specialties were represented: 38 (69.1%) from internal medicine, 7 (12.7%) from general surgery, 5 (9.1%) from family medicine, and 5 (9.1%) from obstetrics–gynecology. Only 2 residents were foreign medical graduates (3.6%). Complete data were available for 53 residents. Medical knowledge We found no significant differences between the pretest mean (standard deviation [SD]) score of 18.6 (1.3) and posttest score of 19.0 (1.2) in residents’ knowledge of institutional policies (P = .118) (see Supplemental Digital Table 1 at http://links.lww.com/ ACADMED/A173). An examination of responses to the individual knowledge questions revealed three questions that were more commonly missed, including incorrectly identifying time constraints (Question #6) as a commonly cited barrier to physicians’ error disclosure. On this question, only 17.0% (9/53) of residents in Session 1 and 11.3% (6/53) in Session 2 correctly identified that time constraints were not a commonly cited barrier (P = .574). In both sessions, only 35.8% (19/53) identified correctly that staying with the patient and family until the chief medical office or risk management arrived (Question #10) was not one of the immediate steps to be taken after a medical error (P = .239). We found some increase in the percentage of residents who correctly identified that the process of disclosure of unanticipated events does not include naming or listing
Academic Medicine, Vol. 89, No. 1 / January 2014
Research Report
interpersonal and communication skills, patient care/clinical skills, professionalism, and practice-based learning and improvement (all P < .001). These results suggest that residents showed additional improvement from participating in a second session and from the self-paced learning tutorial.
the person or team involved in the event (Question #12) from Session 1 (40/53; 75.5%) to Session 2 (50/53; 94.3%), but this change was not statistically significant (P = .145). Responses to only one question (Question #14) improved significantly (P = .011) from pretest to posttest: 88.7% (47/53) of participants before Session 1 versus 96.2% (51/53) of participants before Session 2 correctly identified that disclosure of an unanticipated event should not include speculation regarding the cause of the event.
In addition, residents’ self-efficacy significantly increased from Session 1 pretest to Session 2 posttest for all six of the ACGME competencies (P < .001). The difference in increases is most evident in the total scores: from the baseline mean (SD) score of 119.6 (26.6) to the last posttest score of 150.3 (24.9) (P < .001), demonstrating a large effect size (Cohen's d = 1.19).
Self-efficacy for disclosing medical errors For Session 1, we found a significant increase in residents’ self-efficacy for disclosing medical errors between the pre and post assessment. We saw this increase for all of the ACGME core competencies (P < .001) (see Table 1). These results suggest that the increase in residents’ self-efficacy was sustained over the onemonth period between the posttest for Session 1 and the pretest for Session 2 (a difference of one point or less).
Self-debrief versus faculty debrief In comparing the self-efficacy of residents in the self-debrief (n = 28) versus faculty debrief (n = 24) groups using independent-samples t tests, we found no significant differences on any of the ACGME competencies. The self-debrief group’s self-efficacy total score mean (SD) for Session 1 pretest was 119.64 (27.6), while for Session 2 posttest it was 145.6 (28.5), a change of 25.9 points (95% CI: 15.3–36.6; P < .001). The faculty debrief group’s self-efficacy total
For Session 2, we also found a significant improvement in five of the six ACGME core competencies: medical knowledge,
score mean (SD) for Session 1 pretest was 120.7 (26.7), while for Session 2 posttest it was 156.1 (19.5), a change of 35.4 points (95% CI: 26.0–44.9). On the basis of overlapping 95% CIs, we found no significant difference in residents’ selfefficacy between the self-debrief group and faculty debrief group. External reviewers Several videos had technical issues and could not be reviewed; thus, we evaluated the data from 44 residents who completed Session 1 and Session 2. We combined the external reviewers’ ratings (see Supplemental Digital Table 2 at http:// links.lww.com/ACADMED/A173 for final intraclass correlation coefficients). We found significant improvement in responses to eight assessment questions related to interpersonal and communication skills, professionalism, systems-based practice, and practicebased learning and improvement (all P values ≤ .003; see Table 2). The external reviewers’ total score mean (SD) also significantly improved from Session 1 to Session 2: 94.6 (17.4) to 106.8 (20.0) (P < .001), demonstrating a medium effect size (Cohen's d = 0.65).
Table 1 Results of Postgraduate Year 1 Residents’ Self-Assessments of Self-Efficacy in Disclosing Medical Errors, Based on the Accreditation Council for Graduate Medical Education (ACGME) Core Competencies, OhioHealth Riverside Methodist Hospital, 2011* Total possible score
Pretest mean (SD)
Posttest mean (SD)
P value
Pretest mean (SD)
Posttest mean (SD)
P value
Session 2 posttest versus Session 1 pretest (n = 53): P value
Medical knowledge Interpersonal and communication skills
18
10.6 (2.8)
12.8 (2.7)
Teaching Medical Error Disclosure to Residents Using Patient-Centered Simulation Training Sara Sukalich, MD, John O. Elliott, PhD, MPH, and Gina Ruffner, EMT-P
Abstract Purpose To determine whether a standardized patient encounter and self-guided tutorial would improve first-year residents’ selfefficacy for disclosing medical errors. Method In 2011, 55 first-year residents participated in a simulation in which they disclosed an error to a standardized patient playing the part of a family member. Residents completed the simulation twice, four weeks apart, and completed presession knowledge and self-efficacy (based on the Accreditation Council for Graduate Medical Education [ACGME] core competencies) assessments and repeated the self-efficacy assessment
D
isclosure, the process of bringing to light an unintended outcome, is essential for maintaining trust between physicians, patients, and their family members. Surveys of patients revealed that a large majority preferred to be informed immediately of errors, even minor ones.1,2 Patients also expect to be given more information about an unintended injury during treatment than doctors believe should be given.3
Dr. Sukalich is director, Department of Medical Education, OhioHealth Riverside Methodist Hospital, Columbus, Ohio. Dr. Elliott is research specialist, Department of Medical Education, OhioHealth Riverside Methodist Hospital, Columbus, Ohio. Ms. Ruffner is simulation center manager, Center for Medical Education and Innovation (CME+I), OhioHealth Riverside Methodist Hospital, Columbus, Ohio. Correspondence should be addressed to Dr. Sukalich, Department of Medical Education, 3535 Olentangy River Rd., Columbus, OH 43214-3998; telephone: (614) 566-2426; e-mail: [email protected]. Acad Med. 2014;89:136–143. First published online November 25, 2013 doi: 10.1097/ACM.0000000000000046 Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A173.
136
after the sessions. Residents reviewed the videos of their encounters either alone (self-debrief) or with a faculty observer (faculty debrief). Between sessions, they completed a self-paced learning tutorial. Two external faculty also rated the residents’ performances using videos of the encounters. Results Residents’ self-efficacy significantly increased from a Session 1 pretest mean (standard deviation) score of 119.6 (26.6) to a Session 2 posttest score of 150.3 (24.9) for all ACGME competencies (P < .001, Cohen's d = 1.19). The external reviewers’ ratings provided additional, objective support for
Furthermore, medical errors are common; previous research found that 62% of trainees and 88% of faculty physicians reported making medical mistakes.4 One study found that over 90% of physicians and trainees (medical students and residents) reported they would or should disclose a hypothetical error (major, minor, and those causing no harm); however, only 41% reported actually doing so in real instances.5 Not only is the act of disclosing a medical error the “right thing to do,” but some hospital accreditation requirements and state laws also mandate it.6 In an effort to increase health care professionals’ willingness to embrace disclosure, the National Quality Forum put forward an evidence-based safe practice guideline on the disclosure of serious unanticipated outcomes.7 This guideline recommends providing an explanation to the patient and his or her family about what happened, potential implications or consequences of the error, a commitment to investigate what went wrong, feedback regarding the findings of the investigation, and an apology or expression of regret from the physician. However, throughout training, physicians are taught that the goal
residents’ improvement on questions assessing ACGME competencies (P = .001). Comparisons of the self-efficacy of residents in the self-debrief versus faculty debrief groups yielded no significant differences on any ACGME competencies. Conclusions Timely, explicit, and empathetic disclosure of medical errors to patients and family is essential to maintaining trust and is an important part of patient-centered medical care. This intervention easily could be replicated in other settings and is applicable to many members of the health care team, not just to residents.
in providing care is to improve health, not cause harm, which inherently affects their error disclosure behaviors. Other barriers to fully disclosing an error include fear of potential malpractice litigation in admitting a mistake, the culture of medicine, and the psychological impact of facing mistakes and apologizing for them.8–11 The art of disclosure and the use of effective, open communication are skills not readily taught in most undergraduate medical education environments. Thus, a gap exists in residents’ competency when they enter their graduate medical training, a gap that often continues as they move into medical practice. The majority of prior research on the disclosure of medical errors has focused on health professionals primarily through the use of surveys of the attitudes and beliefs of health sciences students,12 medical students,13 and residents/ physicians.5,14–18 One investigation used written scenarios to assess how residents would disclose medical errors.19 In terms of the methods for teaching error disclosure, previous studies have
Academic Medicine, Vol. 89, No. 1 / January 2014
Research Report
described the use of didactic educational modules,12 seminars,20 DVDs of patients followed by discussion,4 and roleplaying between faculty and students.21 Although the use of standardized patient encounters is a well-established method for assessing clinical skills,22 very few studies have examined a standardized patient method for teaching error disclosure to physicians or residents.12,23,24 Therefore, we argue in favor of creating a robust training and competency assessment program to teach residents how to disclose medical errors and improve communication skills that includes a standardized patient encounter and is based on the National Quality Forum’s disclosure guidelines.7 The main goal of this study was to determine whether a standardized patient encounter scenario, followed by a selfguided tutorial, would improve residents’ self-efficacy for disclosing medical errors. We also were interested in exploring whether a self-debriefing procedure would be as effective as a faculty-led debriefing. To further establish the effectiveness of this training method, we sought to determine whether external faculty members could objectively detect improvements in residents’ error disclosure skills via the review of randomly assigned, video-recorded, standardized patient encounters. Method
In March 2011, we invited all 55 postgraduate year (PGY) 1 residents at the OhioHealth Riverside Methodist Hospital to participate in our study via announcements at program-specific meetings as well as in a letter from the principal investigator (S.S.) and the hospital’s physician vice president of medical education. The residents received no incentive to participate, and we informed them, as required by the OhioHealth institutional review board, that data collected as part of our project would be considered educational research. We deidentified all data that we collected through the use of a studyspecific identification number that we could use to link all assessments. The OhioHealth institutional review board approved our project as exempt. Study design First, we developed the scenario to use in our study. Next, we trained
Academic Medicine, Vol. 89, No. 1 / January 2014
the standardized patients and the internal faculty who would be debriefing residents. The residents then completed the simulation scenario twice, approximately four weeks apart. Between sessions, they completed an online, self-paced learning tutorial on the art of disclosure (15 minutes in length). Before completing each session, the residents took a medical knowledge test and a self-efficacy assessment. After completing each session, they repeated the self-efficacy assessment. The educational material that we compiled for the self-paced learning tutorial included best practice guidelines and references from the National Quality Forum guidelines on disclosure.7 As part of the tutorial, the residents also reviewed the OhioHealth Riverside Methodist Hospital’s standard policy and procedures for the disclosure of unanticipated events. We required all residents to provide a certificate of completion for the tutorial before they started the second session. The first session took place in March 2011 and the second in April 2011. We videotaped all sessions for analysis. After all residents completed the sessions, two trained external faculty members reviewed and evaluated the residents’ error disclosure skills, using video recordings of the encounters. Scenario The research team developed the scenario for this project. It was meant to offer a realistic situation that a PGY 1 resident might be required to address (see Appendix 1). We did not pilot the scenario prior to the study but did ask other faculty involved in simulation education at our institution to review it for face validity. Residents read the scenario and then immediately entered the simulation with the standardized patient at our in-house simulation center (Center for Medical Education and Innovation [CME+I]). Each encounter lasted roughly 10 minutes. Training To ensure that the two standardized patients (acting as a family member) had the necessary qualifications for performance, they underwent a one-hour instruction session, during which we detailed the background of the scenario and reviewed our expectations of them,
and a one-hour practice session with a CME+I staff member. As part of the training, we provided them with a list of 15 questions that we expected them to ask during each encounter with a resident (see Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/A173). This training took place in March 2011. As part of the training process, we provided the participating internal faculty members with an example of a videotaped scenario focusing on a disclosure conversation for them to review, a copy of our hospital’s policies/ procedures for error disclosure, a printed copy of the scenario, a medical debriefing guide used in our simulation lab for other resident training, a copy of the assessment questions on which we would evaluate the residents, and a copy of the 2008 Canadian Medical Protective Association report, titled “Communicating With Your Patient About Harm: Disclosure of Adverse Events.”25 Ten faculty members, representing family medicine, internal medicine, obstetrics–gynecology, and general surgery, participated. Assessments Prior to each session, the residents completed an online medical knowledge test via a computer terminal in the CME+I. The test included 21 true/false questions covering information in our hospital’s policies for error disclosure, which are based on best practices. To quantify the skills needed to provide full disclosure of medical errors, we also created an online resident self-efficacy assessment based on the Accreditation Council for Graduate Medical Education (ACGME) core competencies (medical knowledge, interpersonal and communication skills, patient care/clinical skills, professionalism, systems-based practice, and practice-based learning and improvement). These six core competencies provide a set of standard principles by which residents are to be evaluated as well as a general framework for curriculum development.26,27 They also have been used for teaching risk management to residents.10,28 The self-efficacy assessment included 21 questions (see Appendix 2). The residents rated their confidence in performing aspects of error disclosure on a ninepoint Likert scale (0 = not very confident, 9 = very confident). Self-efficacy, a well-
137
Research Report
established social learning theory,29 has been used to evaluate the high-fidelity training of surgery residents in pediatric trauma30 as well as in the disclosure of medical errors.20 The residents completed the self-efficacy assessment via a computer terminal in the CME+I. Debriefing After each simulation session, each resident completed a video review of the encounter. To assess the impact of the debriefing process, we randomly assigned the residents to review the video alone (n = 29, “self-debrief ”) or with a faculty member (n = 26, “faculty debrief ”). We instructed residents who completed the self-debrief to watch their video and self-reflect on suggested items like “What went well?” “What would I change?” and “How might this impact the way I treat patients?” Residents remained in their assigned debriefing group for both sessions. The self-debrief sessions lasted approximately 10 minutes. Residents in the faculty debrief group watched their video with one faculty member immediately following each standardized patient encounter. Faculty members provided verbal feedback during the video review. The faculty debrief sessions lasted approximately 15 minutes. External faculty reviewers To provide more objective ratings of the residents’ skills, two trained faculty reviewers outside our program watched all of the videotaped encounters during June and July 2011. A computer technician from the CME+I randomly sorted the videotaped encounters so that the reviewers were blinded to which session (first or second) they were viewing. We provided DVD copies of the videotaped encounters with a deidentified study-specific number to the external reviewers. After the external reviewers conducted their assessments, we provided the randomization key to the statistician to properly link each resident’s Session 1 and 2 data. The paper-based external faculty assessment tool was similar to the residents’ self-efficacy assessment but was revised to include anchors for each number on the Likert scale. We also removed the two questions rating the documentation in the patient’s medical
138
record, because they were not directly observable in the simulation, resulting in a 19-question assessment.
we developed has very high reliability (Cronbach α = 0.98; 95% confidence interval [CI]: 0.97–0.99; P < .001).
The principal investigator (S.S.) trained the external reviewers in an effort to address potential issues with interrater reliability. As part of the training, each external reviewer watched and assessed a small sample of 10 videos to assess initial interrater reliability. The intraclass correlations for these assessments ranged from −0.33 to 0.82, suggesting reasonable overall agreement.
Intraclass correlations also indicated moderate agreement between the two external reviewers (Session 1: α = 0.40; 95% CI: 0.31–0.53; P < .001; Session 2: α = 0.44; 95% CI: 0.34–0.56; P < .001). Because of this level of agreement, we averaged the scores for Reviewer 1 and Reviewer 2 for each session for analyses via paired t tests. Finally, we created a total score on the external reviewers’ assessment by summing the ratings of all the questions. We conducted all analyses using SPSS version 19.0 (IBM Corp., Armonk, New York).
Statistical analysis We calculated each resident’s knowledge assessment total score by adding up the total number of correct answers. We used paired t tests to compare knowledge assessment total scores and χ2 tests to compare individual responses on the knowledge assessment between Sessions 1 and 2. We summarized each ACGME core competency domain in the self-efficacy assessment using subscale scores consistent with the ACGME core competency guidelines.31 We summed the ratings of all the questions to create a total score on the self-efficacy assessment. We compared the residents’ responses to the self-efficacy assessments and external reviewers’ ratings via paired t tests, which examine individual change on the basis of matched data. We used a Bonferroni correction method to adjust for multiple comparisons in the Session 1 pre–post, Session 2 pre–post, and Session 1 preSession 2 post analyses. Residents had to complete all assessments to be included in the data analysis of each session. If a resident did not complete an assessment because he or she had to leave for clinical duties or a computer glitch caused the loss of the assessment data, we deleted the data listwise from the analysis for that session. Thus, we report both the numerator and denominator for all percentages in the Results to reflect that some calculations included a different number of participants. Because we found no formal assessments of the skills for the disclosure of medical errors in an extensive review of the literature, we created our own measurement scale (see Appendix 2). Intraclass correlations demonstrated that the residents’ self-efficacy assessment that
Results
Fifty-five PGY 1 residents participated: 36 (65.5%) men and 19 (34.5%) women. Four specialties were represented: 38 (69.1%) from internal medicine, 7 (12.7%) from general surgery, 5 (9.1%) from family medicine, and 5 (9.1%) from obstetrics–gynecology. Only 2 residents were foreign medical graduates (3.6%). Complete data were available for 53 residents. Medical knowledge We found no significant differences between the pretest mean (standard deviation [SD]) score of 18.6 (1.3) and posttest score of 19.0 (1.2) in residents’ knowledge of institutional policies (P = .118) (see Supplemental Digital Table 1 at http://links.lww.com/ ACADMED/A173). An examination of responses to the individual knowledge questions revealed three questions that were more commonly missed, including incorrectly identifying time constraints (Question #6) as a commonly cited barrier to physicians’ error disclosure. On this question, only 17.0% (9/53) of residents in Session 1 and 11.3% (6/53) in Session 2 correctly identified that time constraints were not a commonly cited barrier (P = .574). In both sessions, only 35.8% (19/53) identified correctly that staying with the patient and family until the chief medical office or risk management arrived (Question #10) was not one of the immediate steps to be taken after a medical error (P = .239). We found some increase in the percentage of residents who correctly identified that the process of disclosure of unanticipated events does not include naming or listing
Academic Medicine, Vol. 89, No. 1 / January 2014
Research Report
interpersonal and communication skills, patient care/clinical skills, professionalism, and practice-based learning and improvement (all P < .001). These results suggest that residents showed additional improvement from participating in a second session and from the self-paced learning tutorial.
the person or team involved in the event (Question #12) from Session 1 (40/53; 75.5%) to Session 2 (50/53; 94.3%), but this change was not statistically significant (P = .145). Responses to only one question (Question #14) improved significantly (P = .011) from pretest to posttest: 88.7% (47/53) of participants before Session 1 versus 96.2% (51/53) of participants before Session 2 correctly identified that disclosure of an unanticipated event should not include speculation regarding the cause of the event.
In addition, residents’ self-efficacy significantly increased from Session 1 pretest to Session 2 posttest for all six of the ACGME competencies (P < .001). The difference in increases is most evident in the total scores: from the baseline mean (SD) score of 119.6 (26.6) to the last posttest score of 150.3 (24.9) (P < .001), demonstrating a large effect size (Cohen's d = 1.19).
Self-efficacy for disclosing medical errors For Session 1, we found a significant increase in residents’ self-efficacy for disclosing medical errors between the pre and post assessment. We saw this increase for all of the ACGME core competencies (P < .001) (see Table 1). These results suggest that the increase in residents’ self-efficacy was sustained over the onemonth period between the posttest for Session 1 and the pretest for Session 2 (a difference of one point or less).
Self-debrief versus faculty debrief In comparing the self-efficacy of residents in the self-debrief (n = 28) versus faculty debrief (n = 24) groups using independent-samples t tests, we found no significant differences on any of the ACGME competencies. The self-debrief group’s self-efficacy total score mean (SD) for Session 1 pretest was 119.64 (27.6), while for Session 2 posttest it was 145.6 (28.5), a change of 25.9 points (95% CI: 15.3–36.6; P < .001). The faculty debrief group’s self-efficacy total
For Session 2, we also found a significant improvement in five of the six ACGME core competencies: medical knowledge,
score mean (SD) for Session 1 pretest was 120.7 (26.7), while for Session 2 posttest it was 156.1 (19.5), a change of 35.4 points (95% CI: 26.0–44.9). On the basis of overlapping 95% CIs, we found no significant difference in residents’ selfefficacy between the self-debrief group and faculty debrief group. External reviewers Several videos had technical issues and could not be reviewed; thus, we evaluated the data from 44 residents who completed Session 1 and Session 2. We combined the external reviewers’ ratings (see Supplemental Digital Table 2 at http:// links.lww.com/ACADMED/A173 for final intraclass correlation coefficients). We found significant improvement in responses to eight assessment questions related to interpersonal and communication skills, professionalism, systems-based practice, and practicebased learning and improvement (all P values ≤ .003; see Table 2). The external reviewers’ total score mean (SD) also significantly improved from Session 1 to Session 2: 94.6 (17.4) to 106.8 (20.0) (P < .001), demonstrating a medium effect size (Cohen's d = 0.65).
Table 1 Results of Postgraduate Year 1 Residents’ Self-Assessments of Self-Efficacy in Disclosing Medical Errors, Based on the Accreditation Council for Graduate Medical Education (ACGME) Core Competencies, OhioHealth Riverside Methodist Hospital, 2011* Total possible score
Pretest mean (SD)
Posttest mean (SD)
P value
Pretest mean (SD)
Posttest mean (SD)
P value
Session 2 posttest versus Session 1 pretest (n = 53): P value
Medical knowledge Interpersonal and communication skills
18
10.6 (2.8)
12.8 (2.7)
