535237
research-article2014
AJMXXX10.1177/1062860614535237American Journal of Medical QualityDoers et al
Article
Feedback to Achieve Improved Sign-out Technique
American Journal of Medical Quality 2015, Vol. 30(4) 353–358 © The Author(s) 2014 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/1062860614535237 ajmq.sagepub.com
Matthew E. Doers, BA1, Poonam Beniwal-Patel, MD2, Jessica Kuester, MD1,3, and Kathlyn E. Fletcher, MD, MA1,3
Abstract To maximize the quality of sign-out documents within the internal medicine residency, a quality improvement intervention was developed and implemented. Written sign-outs were collected from general medicine ward teams and graded using an 11-point checklist; in-person feedback was then given directly to the ward teams. Documentation of many of the 11 elements improved: mental status (22% to 66%, P < .0001), decisionality (40% to 66%, P < .0001), lab/test results (63% to 69%, P < .0001), level of acuity (34% to 50%, P < .0001), anticipatory guidance (69% to 82%, P < .0001), and future plans (35% to 38%, P < .0005). The use of vague language declined (41% to 26%, P < .0001). The mean total scores improved from 7.0 to 8.2 out of a possible 11 (P < .0001). As new house staff rotated onto the services, improvement over time was sustained with 1 feedback session per team, per month. Similar interventions could be made in other programs and other institutions. Keywords care transitions, continuity of care, medical education-communication skills, patient safety, medical student and residency education When the responsibility for patient care is transitioned from one clinician to another, an effective transfer of relevant patient information must be performed. Between the increasing use of hospitalists and Accreditation Council for Graduate Medical Education duty hour rules, the predominant model of inpatient care is one of shift work. Therefore, the importance of handoffs in maintaining continuity of care has never been greater.1,2 In fact, in one study, internal medicine chief residents estimated that patient information is “handed off” among residents 8 times during a 4-day hospital stay (an 11% increase since the implementation of the first work-hour restrictions in 2003).3 Previous data suggest that ineffective or incomplete transfers of information, also called sign-outs, can lead to lapses in patient safety and adverse events.4-6 Nationally, the importance of care transitions has been highlighted by the Joint Commission in 2006 by making the standardization of handoff procedures a National Patient Safety Goal.7 Though this recent attention has prompted many studies, there remains a lack of highquality evidence-based research concerning the effectiveness of sign-out improvement interventions.8 Despite the heightened awareness of low-quality signouts and the problems that arise from them, interns continue to overestimate the effectiveness of their handoffs,9 suggesting that a more directed approach to education of
proper sign-out processes is needed. Additionally, previous studies have shown that there is extensive variability in the content and quality of written sign-outs.2,3,10 In an attempt to assess and improve the quality of sign-outs at the Medical College of Wisconsin (MCW), the research team has begun the implementation of a feedback-based intervention within the internal medicine residency program. Similar work has been effective at other institutions, but previously published studies have been limited by a lack of longer-term follow-up.11,12
Local Problem The MCW internal medicine residency leadership has recently identified improving sign-out quality to be a priority for the program. This decision was based partly on local data from a study done by Fletcher and colleagues that showed deficiencies of content in many key areas of 1
Medical College of Wisconsin, Milwaukee, WI University of Wisconsin, Madison, WI 3 Clement J. Zablocki VA Medical Center, Milwaukee, WI 2
Corresponding Author: Kathlyn E. Fletcher, MD, MA, Medical College of Wisconsin and the Milwaukee VAMC, 8701 Watertown Plank Rd, Milwaukee, WI 53226. Email: [email protected]
Downloaded from ajm.sagepub.com at Stockholm University Library on August 11, 2015
354
American Journal of Medical Quality 30(4)
inpatient medicine sign-out documents.13 Additionally, adverse events have occurred that were at least partly attributable to poor patient handoffs. Handoffs on the internal medicine ward teams consist of a written signout document and a brief verbal report from the primary team. The written sign-out documents are populated by the primary team and contain variable amounts of detail. Based on reviewing these sign-out documents and discussing the process of handoffs with residents, the research team found that the primary factor that drives an effective handoff is the quality of the written sign-out document. Therefore, the team decided to increase efforts to improve the quality of handoffs, starting with the written sign-out document.
In response to the recent emphasis placed on improving sign-outs both at the national level by the Joint Commission7 and at the local level by the MCW internal medicine residency leadership, the research team designed an intervention to improve the quality of written sign-out documents of the 4 general internal medicine ward teams at the Milwaukee Veterans Affairs (VA) Medical Center. Each team consisted of an attending physician, a senior resident, 2 interns, 1 fourth-year medical student, and 2 third-year medical students. This project did not alter the existing electronic medical record (EMR)-linked sign-out template, which consists of 5 sections: patient identifiers (autoimported), code status (autoimported), allergies and medication list (autoimported), history of present illness (free text), and “To Do” (free text). Because the first 3 columns are imported automatically from the chart, the focus in this project is mainly on the last 2 free-text sections.
feedback-based intervention to improve the quality of these documents. Based on a review of the literature and local needs, the team identified 11 essential components of an effective sign-out document, which then were transformed into a checklist. This checklist was used to objectively grade the quality of sign-outs during the project. Each checklist item was worth 1 point and included the following: code status, mental status, anticipatory guidance, patient “decisionality,” history of present illness, active problem list, future plans, lab/test results, degree of acuity, no confusing/conflicting language (“Procedure tomorrow” written in one section and the results of that procedure listed in another), and no vague language (eg, “today” or “tomorrow”). Prior to the data collection period, members of the research team discussed the criteria as a team. The team also graded several sign-outs individually and then compared the assessments to ensure minimal intermember variability and maximal consistency and reproducibility. Once all team members agreed that scoring was consistent across graders, data collection for the project began. Subsequently, each sign-out document was assessed by only 1 study team member. Formal agreement was not calculated. This project occurred between January and June 2013, which allowed for long-term (6 months) tracking of the intervention. The research team actively intervened and collected data as described in the following section in January, February, April, and June. During each of these months, written sign-outs were collected from 4 general medicine ward teams for 3 consecutive weeks and graded by the research team using the 11-point checklist. The first week was treated as the baseline (n = 184 sign-out entries). The second week was the intervention week, and in the third week, the research team assessed the outcome of the intervention (n = 229).
Study Questions
The Intervention
The questions addressed in this study are the following: Can the quality of written sign-outs be improved with an intervention that includes a structured checklist and direct feedback? How many times must feedback be given to achieve higher quality written sign-outs? How long are the improvements sustained before another intervention is needed?
During the intervention week, a member of the quality improvement team downloaded the ward team sign-outs in the mornings and then assessed each patient entry per the 11-item sign-out checklist. The ward teams were provided with written and verbal feedback about the quality of their sign-out documents, using the checklist as a guide to the discussion. Though the feedback discussion was based on the checklist, the checklist itself was not directly provided to the ward teams, nor was it integrated into the EMR sign-out template. To provide this feedback, a member of the research team went to the ward team room at times when it was thought that the number of team members present would be maximized. If members of the primary ward team were absent, other ward team members were asked to share the information/graded sign-out with those who were absent. In the first month of
Intended Improvement and Setting
Methods Planning the Intervention and Methods of Evaluation The research team designed a prospective quality improvement project aimed at evaluating the baseline quality of sign-out documents and implementing a
Downloaded from ajm.sagepub.com at Stockholm University Library on August 11, 2015
355
Doers et al Table 1. Percentage of Time That Individual Items Were Recorded and Total Score From January and Overall. January Code status Mental status Anticipatory guidance Decisionality/Contact Medical Hx Active problems Future plans Lab/Test results Degree of acuity No confusing/conflicting language No vague language Total points
Overall
Preintervention (n = 75)
Postintervention (n = 129)
Preintervention (n = 184)
Postintervention (n = 229)
100% 12.0% 57.3% 32.0% 94.7% 97.3% 22.7% 54.7% 17.3% 81.3% 50.7% 6.2
100% 76.0%a 74.4%a 72.1%a 96.1% 100% 27.9%a 73.6%a 62.8%a 76.7% 72.1%a 8.3a
100% 21.7% 69.0% 39.7% 94% 98.9% 34.8% 63.0% 33.7% 85.9% 59.2% 7.0
100% 66.4%a 81.7%a 65.5%a 97% 100% 37.6%a 69.4%a 50.7%a 83.0% 73.8%a 8.2a
Abbreviation: Hx, history. a Indicates significant changes from the corresponding preintervention value.
implementation, feedback was given to 2 teams on 2 days during the intervention week, and feedback was given to the other 2 teams on 4 days. In the subsequent 3 intervention months, feedback was given only once per month per team. Based on requests from the residents, a brief educational primer was added to the intervention. Therefore, for each subsequent month starting in February, the chief resident gave a brief 5-minute PowerPoint presentation to residents that detailed the essential components of a highquality sign-out document. This occurred at the beginning of the month. Also beginning in February, a poster containing this information was posted in each medical team room for reference. This project met the criteria for quality improvement within the VA and was not submitted for institutional review board approval.
Analysis Sign-out entries were the unit of analysis. The research team did not grade the quality of sign-out entries for newly admitted patients when the sign-out entry was still incomplete. Mean total scores from the preintervention and postintervention weeks from each month were compared using unpaired 2-tailed t tests. The P value was set at
research-article2014
AJMXXX10.1177/1062860614535237American Journal of Medical QualityDoers et al
Article
Feedback to Achieve Improved Sign-out Technique
American Journal of Medical Quality 2015, Vol. 30(4) 353–358 © The Author(s) 2014 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/1062860614535237 ajmq.sagepub.com
Matthew E. Doers, BA1, Poonam Beniwal-Patel, MD2, Jessica Kuester, MD1,3, and Kathlyn E. Fletcher, MD, MA1,3
Abstract To maximize the quality of sign-out documents within the internal medicine residency, a quality improvement intervention was developed and implemented. Written sign-outs were collected from general medicine ward teams and graded using an 11-point checklist; in-person feedback was then given directly to the ward teams. Documentation of many of the 11 elements improved: mental status (22% to 66%, P < .0001), decisionality (40% to 66%, P < .0001), lab/test results (63% to 69%, P < .0001), level of acuity (34% to 50%, P < .0001), anticipatory guidance (69% to 82%, P < .0001), and future plans (35% to 38%, P < .0005). The use of vague language declined (41% to 26%, P < .0001). The mean total scores improved from 7.0 to 8.2 out of a possible 11 (P < .0001). As new house staff rotated onto the services, improvement over time was sustained with 1 feedback session per team, per month. Similar interventions could be made in other programs and other institutions. Keywords care transitions, continuity of care, medical education-communication skills, patient safety, medical student and residency education When the responsibility for patient care is transitioned from one clinician to another, an effective transfer of relevant patient information must be performed. Between the increasing use of hospitalists and Accreditation Council for Graduate Medical Education duty hour rules, the predominant model of inpatient care is one of shift work. Therefore, the importance of handoffs in maintaining continuity of care has never been greater.1,2 In fact, in one study, internal medicine chief residents estimated that patient information is “handed off” among residents 8 times during a 4-day hospital stay (an 11% increase since the implementation of the first work-hour restrictions in 2003).3 Previous data suggest that ineffective or incomplete transfers of information, also called sign-outs, can lead to lapses in patient safety and adverse events.4-6 Nationally, the importance of care transitions has been highlighted by the Joint Commission in 2006 by making the standardization of handoff procedures a National Patient Safety Goal.7 Though this recent attention has prompted many studies, there remains a lack of highquality evidence-based research concerning the effectiveness of sign-out improvement interventions.8 Despite the heightened awareness of low-quality signouts and the problems that arise from them, interns continue to overestimate the effectiveness of their handoffs,9 suggesting that a more directed approach to education of
proper sign-out processes is needed. Additionally, previous studies have shown that there is extensive variability in the content and quality of written sign-outs.2,3,10 In an attempt to assess and improve the quality of sign-outs at the Medical College of Wisconsin (MCW), the research team has begun the implementation of a feedback-based intervention within the internal medicine residency program. Similar work has been effective at other institutions, but previously published studies have been limited by a lack of longer-term follow-up.11,12
Local Problem The MCW internal medicine residency leadership has recently identified improving sign-out quality to be a priority for the program. This decision was based partly on local data from a study done by Fletcher and colleagues that showed deficiencies of content in many key areas of 1
Medical College of Wisconsin, Milwaukee, WI University of Wisconsin, Madison, WI 3 Clement J. Zablocki VA Medical Center, Milwaukee, WI 2
Corresponding Author: Kathlyn E. Fletcher, MD, MA, Medical College of Wisconsin and the Milwaukee VAMC, 8701 Watertown Plank Rd, Milwaukee, WI 53226. Email: [email protected]
Downloaded from ajm.sagepub.com at Stockholm University Library on August 11, 2015
354
American Journal of Medical Quality 30(4)
inpatient medicine sign-out documents.13 Additionally, adverse events have occurred that were at least partly attributable to poor patient handoffs. Handoffs on the internal medicine ward teams consist of a written signout document and a brief verbal report from the primary team. The written sign-out documents are populated by the primary team and contain variable amounts of detail. Based on reviewing these sign-out documents and discussing the process of handoffs with residents, the research team found that the primary factor that drives an effective handoff is the quality of the written sign-out document. Therefore, the team decided to increase efforts to improve the quality of handoffs, starting with the written sign-out document.
In response to the recent emphasis placed on improving sign-outs both at the national level by the Joint Commission7 and at the local level by the MCW internal medicine residency leadership, the research team designed an intervention to improve the quality of written sign-out documents of the 4 general internal medicine ward teams at the Milwaukee Veterans Affairs (VA) Medical Center. Each team consisted of an attending physician, a senior resident, 2 interns, 1 fourth-year medical student, and 2 third-year medical students. This project did not alter the existing electronic medical record (EMR)-linked sign-out template, which consists of 5 sections: patient identifiers (autoimported), code status (autoimported), allergies and medication list (autoimported), history of present illness (free text), and “To Do” (free text). Because the first 3 columns are imported automatically from the chart, the focus in this project is mainly on the last 2 free-text sections.
feedback-based intervention to improve the quality of these documents. Based on a review of the literature and local needs, the team identified 11 essential components of an effective sign-out document, which then were transformed into a checklist. This checklist was used to objectively grade the quality of sign-outs during the project. Each checklist item was worth 1 point and included the following: code status, mental status, anticipatory guidance, patient “decisionality,” history of present illness, active problem list, future plans, lab/test results, degree of acuity, no confusing/conflicting language (“Procedure tomorrow” written in one section and the results of that procedure listed in another), and no vague language (eg, “today” or “tomorrow”). Prior to the data collection period, members of the research team discussed the criteria as a team. The team also graded several sign-outs individually and then compared the assessments to ensure minimal intermember variability and maximal consistency and reproducibility. Once all team members agreed that scoring was consistent across graders, data collection for the project began. Subsequently, each sign-out document was assessed by only 1 study team member. Formal agreement was not calculated. This project occurred between January and June 2013, which allowed for long-term (6 months) tracking of the intervention. The research team actively intervened and collected data as described in the following section in January, February, April, and June. During each of these months, written sign-outs were collected from 4 general medicine ward teams for 3 consecutive weeks and graded by the research team using the 11-point checklist. The first week was treated as the baseline (n = 184 sign-out entries). The second week was the intervention week, and in the third week, the research team assessed the outcome of the intervention (n = 229).
Study Questions
The Intervention
The questions addressed in this study are the following: Can the quality of written sign-outs be improved with an intervention that includes a structured checklist and direct feedback? How many times must feedback be given to achieve higher quality written sign-outs? How long are the improvements sustained before another intervention is needed?
During the intervention week, a member of the quality improvement team downloaded the ward team sign-outs in the mornings and then assessed each patient entry per the 11-item sign-out checklist. The ward teams were provided with written and verbal feedback about the quality of their sign-out documents, using the checklist as a guide to the discussion. Though the feedback discussion was based on the checklist, the checklist itself was not directly provided to the ward teams, nor was it integrated into the EMR sign-out template. To provide this feedback, a member of the research team went to the ward team room at times when it was thought that the number of team members present would be maximized. If members of the primary ward team were absent, other ward team members were asked to share the information/graded sign-out with those who were absent. In the first month of
Intended Improvement and Setting
Methods Planning the Intervention and Methods of Evaluation The research team designed a prospective quality improvement project aimed at evaluating the baseline quality of sign-out documents and implementing a
Downloaded from ajm.sagepub.com at Stockholm University Library on August 11, 2015
355
Doers et al Table 1. Percentage of Time That Individual Items Were Recorded and Total Score From January and Overall. January Code status Mental status Anticipatory guidance Decisionality/Contact Medical Hx Active problems Future plans Lab/Test results Degree of acuity No confusing/conflicting language No vague language Total points
Overall
Preintervention (n = 75)
Postintervention (n = 129)
Preintervention (n = 184)
Postintervention (n = 229)
100% 12.0% 57.3% 32.0% 94.7% 97.3% 22.7% 54.7% 17.3% 81.3% 50.7% 6.2
100% 76.0%a 74.4%a 72.1%a 96.1% 100% 27.9%a 73.6%a 62.8%a 76.7% 72.1%a 8.3a
100% 21.7% 69.0% 39.7% 94% 98.9% 34.8% 63.0% 33.7% 85.9% 59.2% 7.0
100% 66.4%a 81.7%a 65.5%a 97% 100% 37.6%a 69.4%a 50.7%a 83.0% 73.8%a 8.2a
Abbreviation: Hx, history. a Indicates significant changes from the corresponding preintervention value.
implementation, feedback was given to 2 teams on 2 days during the intervention week, and feedback was given to the other 2 teams on 4 days. In the subsequent 3 intervention months, feedback was given only once per month per team. Based on requests from the residents, a brief educational primer was added to the intervention. Therefore, for each subsequent month starting in February, the chief resident gave a brief 5-minute PowerPoint presentation to residents that detailed the essential components of a highquality sign-out document. This occurred at the beginning of the month. Also beginning in February, a poster containing this information was posted in each medical team room for reference. This project met the criteria for quality improvement within the VA and was not submitted for institutional review board approval.
Analysis Sign-out entries were the unit of analysis. The research team did not grade the quality of sign-out entries for newly admitted patients when the sign-out entry was still incomplete. Mean total scores from the preintervention and postintervention weeks from each month were compared using unpaired 2-tailed t tests. The P value was set at
