ORIGINAL ARTICLE
Teaching Quality Improvement in Occupational Medicine Improving the Efficiency of Medical Evaluation for Commercial Drivers Atsushi Sorita, MD, MPH, David Raslau, MD, MPH, Mohammad Hassan Murad, MD, MPH, and Mark W. Steffen, MD, MPH
Objective: To describe a successful, resident-led quality improvement (QI) project that improved the efficiency of the Department of Transportation (DOT) medical examination process. Methods: After learning QI principles through didactics, workshops, and online modules, residents led a QI project to streamline the process of the DOT examination. An interdisciplinary group of key stakeholders collaborated to analyze the process and to design and implement interventions. Results: Following the Model for Improvement and Lean concepts, residents ran seven Plan-Do-Study-Act cycles over a 4-month period with multiple iteration and testing changes. Compared with the baseline, the team successfully reduced the total visit time (from check-in to check-out) by 28 minutes (102 minutes vs. 130 minutes; P < 0.001). The accuracy of certificate issuance, as proxy for quality of the examinations, improved after the interventions. Conclusions: Residents successfully improved the efficiency of the DOT examination process.
T
he Accreditation Council for Graduate Medical Education and the Royal College of Physicians and Surgeons of Canada have emphasized that trainees in medical education should become competent in quality improvement (QI).1,2 In response, many training programs developed QI curricula for trainee physicians utilizing didactics and group sessions,3 online modules,4 and experiential learning;5 and some used multipronged approaches.6,7 Nevertheless, the framework for effective curriculum in QI has not been well established, and the integration of QI education in graduate medical training is still challenging.1,8 Although many training programs require that their trainees conduct a QI project, the application of an established QI framework is often not rigorous in actual projects.9 Moreover, the literature on the application of and the education on QI principles in occupational medicine remains sparse.10–12 In this study, we describe a demonstrable QI project, led by residents in Preventive, Occupational, and Aerospace Medicine, which followed formal QI framework and methodology, resulting in a meaningful process improvement in the Department of Transportation (DOT) medical examination.
CONTEXT AND BACKGROUND Effective in May 2014, the Federal Motor Carriers Safety Administration in the United States created the National Registry of Certified Medical Examiners and required that all DOT examiners receive training on the Federal Motor Carriers Safety Administration’s physical standards and pass an examination to verify their understanding.13 Because of the expected shortage of certified medical examiners, the division’s leadership expected an increase in the number of examinations performed in our setting. Nevertheless, From the Division of Preventive, Occupational, and Aerospace Medicine, Mayo Clinic, Rochester, Minn. The authors declare no conflicts of interest. Address correspondence to: Mark W. Steffen, MD, MPH, Division of Preventive, Occupational, and Aerospace Medicine, Mayo Clinic, 200 1st St SW, Rochester, MN 55905 ([email protected]). C 2015 by American College of Occupational and Environmental Copyright Medicine DOI: 10.1097/JOM.0000000000000394
JOEM r Volume 57, Number 4, April 2015
benchmarking identified a quality gap in the efficiency of our practice: the total visit time (from check-in to check-out) was perceived much longer than average local practice. The practice leadership prioritized a project to improve the efficiency of these examinations. A preliminary meeting led to the formation of an interdisciplinary working group in August 2013 that included administrators, providers, residents, nurses, clinical assistants, and secretaries. This group developed an aim statement: to decrease the total visit time and internal costs by an average of 25% within 6 months while maintaining the accuracy of certificate completion.
METHODS This study is reported according to the Standards for Quality Improvement Reporting Excellence guidelines.14 Following institutional guidelines, ethical review was not required for this QI initiative.
Setting This project was performed by residents in Preventive, Occupational, and Aerospace Medicine at the Mayo Clinic in Rochester, Minnesota. The program requires residents to complete a QI project during residency to meet the competency in practice-based learning and improvement set by the Accreditation Council for Graduate Medical Education. The faculty receives training in QI methodology through the Quality Academy, an institutional group staffed by QI experts that provides structured QI education for the staff. The faculty also completes modules in communication, leadership, and teamwork principles.
Curriculum Components A curriculum in QI has been implemented in the residency program since 2005 that combines didactics, workshop, online modules, and experiential learning (ie, learning from direct experience).15 Residents received didactics by faculty and through the Quality Academy about evidence-based medicine, critical appraisal of evidence, conducting a literature search, study design, and an overview of QI methods. Residents also attended workshop-format classes on value stream mapping, change management, and QI methodology (which covered identifying, measuring, and prioritizing opportunities for improvement, selecting and applying appropriate quality tools and methods, and understanding how to sustain long-term improvement). Furthermore, residents completed online modules that covered quality measurement, patient safety, principles of leadership, and health policy. A total of 3 weeks of protected time (three 1-week periods) was provided during one academic year for this project, as well as flexibility for attending classes and meetings. The faculty met frequently with residents during the project and debriefed about progress and planned for the next step. Faculty communicated with the practice leadership and with administrative and support staff to secure buy-in and support for change.
Planning the Intervention The initial step was to form an interdisciplinary team consisting of representatives of all those who would be impacted by changes to the current DOT process, including secretaries, clinical assistants, 453
Copyright © 2015 American College of Occupational and Environmental Medicine. Unauthorized reproduction of this article is prohibited.
JOEM r Volume 57, Number 4, April 2015
Sorita et al
registered nurses (RNs), and providers (nurse practitioners, physician assistants, and physicians). During the first multidisciplinary team meeting, roles were identified: leaders (the residents), a physician champion, project members, and supporters. A brainstorming session was held and input from all members regarding potential interventions was entertained. The entire process was mapped out by shadowing patients throughout their visit (Fig. 1). Next, value stream mapping was used to describe the current and future processes. The team calculated total wait time, total process time, lead time, value percentage added, and first-time quality. Using these tools and lean concepts, the team identified areas of waste, which primarily included waiting, rework, motion, and overprocessing types of waste. Multiple possible interventions were discussed to reduce the waste and improve the efficiency. These were grouped into several categories using an affinity diagram. The team made the decision to move forward using the Model for Improvement by the Institute for Healthcare Improvement and rapid Plan-DoStudy-Act (PDSA) cycles. This allowed multiple, rapid changes over a short period while at the same time minimizing interruptions to the flow of the clinical day. Monthly meetings were held to discuss the changes that were made and to make corrections or plan additional interventions as necessary.
To determine the effect of the QI efforts, several measures were assessed, including total visit time (check-in to check-out) and total time spent in each step (check-in, RN exam, provider examination, and check-out). All DOT examinations were included except those that had a gap in time between RN and provider appointments. Measurements were obtained by reviewing patient tracking logs in the electronic medical record. As a countermeasure to ensure that the examination process maintained the level of quality, the team measured the number of certification errors for the DOT examinations for clinic employees. This was accomplished by having the fleet service at the institution provide feedback if any errors were identified in the certificates that were being provided by our practice. These measures were collected and reviewed at every team meeting. The degree of error in the patient tracking logs was considered to be constant throughout the project period because the process of keeping the logs did not change. The team agreed with the validity and reliability of these measures. Data collection was done by the team leaders (residents). The team also evaluated the effectiveness and implementation of each individual intervention by conducting staff interviews and group discussions.
PROCESS MAP - NEW
PROCESS MAP - BASELINE Check-in
Exam for employees?
Methods of Evaluation
Secretary prepares the certificate* Yes
Patient fills out release of information form
Check-in
No RN checks vitals and rooms the patient
CA checks vitals and rooms the patient*
RN EXAM: Blood pressure Vision by Titmus or Snelling Hearing test Urine test
CA prefills the certificate*
RN EXAM: Vision by Titmus* Hearing test Urine test
RN returns the patient to waiting area
PROVIDER EXAM: History taking Physical exam Complete certificate
CA rooms the patient for provider exam
PROVIDER EXAM: History taking Physical exam Complete certificate
The patient goes to the front desk
Provider laminates the DOT certificate
CA checks accuracy and laminates the certificate*
The patient goes to the front desk
Check-out
Exam for employees?
Yes
Certificate retained at the front desk
*New processes implemented during the project RN: registered nurse
No Check-out
454
CA: clinical assistants
FIGURE 1. Baseline and new process maps. The figure shows the high-level baseline process map on the left, which was created in the planning phase of the project. At baseline, there were separate steps for institutional employees, nurse examinations were not standardized, and providers filled out the certificate. The new process map on the right shows simplified processes with changes indicated with asterisk. In the new process, secretary and CA prefill the certificate, CA rooms the patient instead of RN, RN examinations were standardized, and CA laminates the certificate for providers. CA, clinical assistant; RN, registered nurse.
C 2015 American College of Occupational and Environmental Medicine
Copyright © 2015 American College of Occupational and Environmental Medicine. Unauthorized reproduction of this article is prohibited.
JOEM r Volume 57, Number 4, April 2015
Teaching QI to Improve Efficiency of DOT Examination
TABLE 1. PDSA Cycles PDSA Cycle
Interventions
#1 9/18/2013
CAs provided verbal instructions to patients reminding them of the urine examination and filling out the DOT form RN examination was standardized to use the Titmus machine for visual acuity checks CAs take vitals and room the patients, which were previously done by RNs RN examination time was shortened from 60 to 30 min
#2 10/3
#3 10/14
#4 10/25
#5 11/18
#6 12/30
#7 1/22/2014
CAs handed folders to patients, which contained written instructions for them and the providers regarding important aspects of the process Patients were kept in the same room for RN and provider visits CAs began performing a quality check on the certificates and laminating them in place of the providers CAs no longer waited for patients to complete the DOT form before rooming RNs began directly prompting the providers to see the patient as soon as the RNs finish their visits Secretaries and CAs prefilled certificates with information specific to patients and providers Redundant documents for the visit were eliminated Process for DOT examinations for institutional employees and others was standardized RN examination time was made 30 min for all appointments
Rationale for Intervention Examination was delayed waiting for patients who had urinated before examination or had not filled out the form Not all RNs were using the Titmus machine, which allows for a more expedited examination CAs normally do these for other clinic visits and this will free up RN time
Success of Implementation Fair (not all CAs were consistent in reminding)
Minimal
Good
Moderate
Good
Moderate
Most RNs did not need the full 60min with the reduced responsibilities
Fair (not all examinations were made 30 min) To save CAs’ time of verbally conveying Fair (felt cumbersome the information and to provide the by some CAs) providers with information about the changes To eliminate waste of having patients Fair (not all providers return to waiting area to be roomed complied with it) again Checking and laminating the certificates Good was taking time for providers
Patients can complete the form during waiting periods between examination components RNs were asking CAs to prompt the providers, which created waiting and waste This would reduce work for providers
Impact
Large
Moderate
Moderate
Moderate
Good
Minimal
Good
Moderate
Good
Moderate
These documents were not necessary for Good a DOT examination Good Slight nuanced differences between examinations for institutional employees and others created confusion Not all RN visits were 30 min because of Good system issue
Minimal Moderate
Large
CA, clinical assistant; DOT, Department of Transportation; RN, registered nurse.
Analysis Data were compiled to determine the total visit time before and after the intervention. A control chart (Individual-chart) for total visit time was constructed to determine whether the process was under control and whether the visit time changed significantly before and after the intervention. Comparisons were made between baseline (July to August 2013) and intervention periods (September 2013 to March 2014). Data during the intervention period were divided into two types: visits with 60-minute RN appointment and ones with 30-minute RN appointment. For continuous variables, numbers were expressed as average with standard deviation. A two-sample t test was used for two groups and analysis of variance with Turkey’s method was used for more than two groups. For discrete variables, numbers were expressed as percentage, and we used a chi-square test for comparison. All
tests were two-sided, and P < 0.05 was considered significant. For the above analysis, we used the JMP Software, version 9.0 (SAS Institute, Cary, NC).
RESULTS Context of Intervention The environment and culture at our institution strongly support change and improvement efforts. A team-based approach is embedded in our everyday care of patients, which allowed us to establish an interdisciplinary team that functioned cohesively. Nurse team members took initiatives and led some of the changes described below. The actual improvements were implemented primarily by nursing and clerical staff members. The presence of the residents on the floor every day allowed for immediate corrections when glitches
C 2015 American College of Occupational and Environmental Medicine
455
Copyright © 2015 American College of Occupational and Environmental Medicine. Unauthorized reproduction of this article is prohibited.
JOEM r Volume 57, Number 4, April 2015
Sorita et al
and issues with implementation inevitably arose. The support of the division’s leadership allowed this QI initiative to thrive, and ultimately, to be successful.
Intervention In the planning phase, the team identified the most important waste as waiting time between the RN examination and the provider examination. RN appointment was set at 60 minutes, whereas RNs finished their examinations in 32 minutes on average. The patient was escorted back to the waiting area after the RN examination to wait for the provider examination, creating a rework for front desk staff (Fig. 1). A total of seven PDSA cycles were conducted in the 4-month period. The timeline, rationales, details, and perceived success and impact of each intervention in the PDSA cycles are described in Table 1. Overall, the interventions were primarily aimed at eliminating waiting time for the patient. The team also aimed at standardizing the process of the DOT examination, eliminating non–value-added steps, and maximizing utilization of the expertise of each profession, specifically increasing the responsibilities of the secretaries and clinical assistants and delegating some of the responsibilities of the RNs and providers to them. Several changes were abandoned or modified after testing. Institutional policy was the barrier for implementing drastic changes, such as eliminating the RN visit entirely or at least eliminating the documentation requirements for the RN. The team was dedicated to providing potential changes that would be implementable given the institutional constraints. The provider examination component was left untouched because of the perceived difficulty in implementing any changes to this aspect of the examination. The team discovered after additional conversation with the Appointment Coordination Office that the proposed reduction of RN appointments from 60 to 30 minutes had not entirely occurred until January 2014. This provided a concurrent control of 60 minutes versus 30 minutes for the RN visit. This led to the
Outcomes The entire process for the DOT examination was simplified after the interventions (Fig. 1). Overall, 32 and the 101 DOT examinations were conducted at baseline (July to August, 2013) and during the intervention period (September 2013 to March 2014), respectively. After excluding visits with a gap in time between RN and provider appointments, the authors included in the analysis 25 visits at the baseline and 80 visits during the intervention period (38 visits with 30-minute and 42 visits with 60-minute RN appointment). The control chart showed a stable process at baseline with a significant shift after the intervention in visits with 30-minute RN appointment (Fig. 2). Compared with the baseline, the average total visit time decreased by 28 minutes (102 minutes vs 130 minutes; P < 0.001) for 30-minute RN visits, with a relative reduction by 21%. The average time for those with 60-minute RN appointment did not change significantly from the baseline (137 minutes vs 130 minutes; P = 0.51). When comparing each step of the three types of visits (baseline, 30-minute RN, and 60-minute RN visits), no difference was observed for check-in or provider examination, whereas the waiting time between RN and provider visit was significantly shortened in 30-minute RN visits (Fig. 3). It was estimated that costs were reduced by approximately 25% because the nurse examination was cut in half and the equipment usage was standardized. Despite the numerous changes that occurred, RN 30 minutes (intervention)
RN 60 minutes (control)
Baseline 250
Total Visit Time (in minutes)
discovery that the most significant intervention was the reduction of RN responsibilities, which enabled them to perform all essential components of their examination in the shorter appointment times. All proposed changes to the DOT examination process were discussed and approved by the Practice Committee in January 2014. The success of the QI effort was recognized, and the plan was made to further improve the quality by standardizing the examination for providers.
200 Upper Control Limit
150
Average
100
50 Lower Control Limit 10
20
30
40
50
60
70
80
90
100
Number of Exams
FIGURE 2. Control chart for total visit time. The control chart shows changes in total visit time before and after implementation of the changes. The visit time was stable at the baseline. No significant improvement in visit time was seen in visits, with 60-minute RN appointment (in the middle). There was a significant shift (shortening of visit time) in visits with 30-minute RN appointment during the project. RN, registered nurse. 456
C 2015 American College of Occupational and Environmental Medicine
Copyright © 2015 American College of Occupational and Environmental Medicine. Unauthorized reproduction of this article is prohibited.
JOEM r Volume 57, Number 4, April 2015
Teaching QI to Improve Efficiency of DOT Examination
Nurse Exam
Baseline
Check-in 21
RN 60 minutes (control)
Check-in 30
RN 30 minutes (intervention)
Check-in 23
Nurse Appointment
Provider Exam
28
Nurse Appointment
Nurse Appointment
Provider Appointment
Check-out 32
Provider Appointment
60
30
Provider Appointment
Check-out 49
Check-out 47
Check-out 49
FIGURE 3. Visit time by step (minutes). The figure shows visit time stratified by components of the visit. At baseline, there were 21 minutes from check-in to nurse appointment, 32 minutes for nurse visit, 28 minutes between nurse visit and provider visit (waiting time), and 49 minutes for provider visit. The waiting time was shortened significantly in visits with 30-minute nurse appointment as shown at the bottom. There were no significant changes in time from check-in to nurse appointment, nor for provider visit. PDSA #1
25
PDSA #5
PDSA #2
PDSA #6
PDSA #3
FIGURE 4. Accurate completion of certificates. The figure shows the number of accurately and inaccurately completed certificates for institutional employees per month with the timing of PDSA cycles. There were substantial number of errors in certificates until October to November 2013, and then the proportion of inaccurately completed certificates became smaller. PDSA, Plan-DoStudy-Act.
Number of Exams
20
PDSA #7
PDSA #4
15 Inaccurate Accurate
10
5
0 Jul-13
certification errors occurred less frequently after these interventions (Fig. 4). This was likely due to addition of the quality check step by front desk staff that aimed at reducing the work of providers.
DISCUSSION Summary We described an exemplar QI project led by residents that yielded meaningful process improvement in the DOT medical examination. The project successfully reduced the total visit time for the DOT examination by 28 minutes while improving accuracy of the certificate issuance. The project followed a rigorous QI methodology and the changes were implemented into routine clinical practice.
Relation to Other Evidence The residents learned QI methodology from didactics, workshops, online modules, and experiential learning. They also had dedicated time to complete a QI project and had sufficient support by the faculty. The institutional leadership and environment also supported the QI efforts undertaken by residents. These factors influenced successful completion of the QI project by residents.16 Shortening visit time by 21% (28 minutes) is robust and comparable to other QI projects that successfully improved the efficiency of outpatient clinic visits.6,17–20
Aug-13
Sep-13
Oct-13
Nov-13
Dec-13
Jan-14
Feb-14
Mar-14
Month
Limitations This report has inherent limitations. First, the project did not randomize patients to interventions and thus the quality of evidence is limited. Nevertheless, the patients were assigned to 30- or 60minute RN appointment inadvertently because of incomplete implementation, providing the concurrent control. Therefore, the significant reduction in total visit time has a modest strength of evidence. Second, visit time was approximated by available time stamps in the electronic medical record, and thus the time estimates may not be accurate. Nevertheless, the degree of bias should have remained the same over time and thus this limitation is unlikely to have affected the overall results. Third, we assume that faculty expertise and mentoring were instrumental for the success of the project. If this is true, the described project may not be replicated in institutions with limited QI expertise in the faculty. Nevertheless, with increased emphasis on developing competency in QI for trainee physicians, the lack of expertise in QI by the faculty, if any, should be considered a quality gap that needs to be closed.
Interpretation Success factors perceived from the team members were early involvement and engagement of key stakeholders and SMART (specific, measurable, attainable, relevant, and time-bound) goal setting in the aim statement. Although shortening nursing visit time to
C 2015 American College of Occupational and Environmental Medicine
457
Copyright © 2015 American College of Occupational and Environmental Medicine. Unauthorized reproduction of this article is prohibited.
JOEM r Volume 57, Number 4, April 2015
Sorita et al
30 minutes put more work on nursing staff by opening up their schedule, this change was proposed and implemented by nursing staff themselves. This likely contributed to the success of implementation of this particular intervention. Even with the overall success of the project, we were not able to shorten the total visit time by 25% as specified in the aim statement. This was mainly due to factors that were out of control of the team: check-in time of the patient and delay of providers due to other patient appointments. On the basis of lessons learned, the team has already started discussion to standardize the provider examination to further improve quality and efficiency. As the volume of the DOT examinations increases in our worksite, additional opportunity exists to further improve the efficiency: creating a dedicated DOT examination clinic or utilizing support staff to the fullest extent. Furthermore, lessons learned through the project could be applied to other types of examinations in the worksite.
CONCLUSIONS Residents successfully improved the efficiency of the DOT examination while maintaining the quality of the examination through the application of rigorous QI methodology.
REFERENCES 1. Wong BM, Levinson W, Shojania KG. Quality improvement in medical education: current state and future directions. Med Educ. 2012;46:107–119. 2. Wong BM, Ackroyd-Stolarz S, Bukowskyj M, et al. The CanMEDS 2015 Patient Safety and Quality Improvement Expert Working Group Report. Ottawa: The Royal College of Physicians and Surgeons of Canada; 2014. 3. Reznek MA, Digiovine B, Kromrei H, Levine D, Wiese-Rometsch W, Schreiber M. Quality education and safe systems training (QuESST): development and assessment of a comprehensive cross-disciplinary resident quality and patient safety curriculum. J Grad Med Educ. 2010;2:222–227. 4. Kerfoot BP, Conlin PR, Travison T, McMahon GT. Web-based education in systems-based practice: a randomized trial. Arch Intern Med. 2007;167: 361–366. 5. Sellers MM, Hanson K, Schuller M, et al. Development and participant assessment of a practical quality improvement educational initiative for surgical residents. J Am Coll Surg. 2013;216:1207–1213, 1213-e1201. 6. Diaz VA, Carek PJ, Dickerson LM, Steyer TE. Teaching quality improvement in a primary care residency. Jt Comm J Qual Patient Saf. 2010;36:454–460.
458
7. Oyler J, Vinci L, Johnson JK, Arora VM. Teaching internal medicine residents to sustain their improvement through the quality assessment and improvement curriculum. J Gen Intern Med. 2011;26:221–225. 8. Ogrinc G, Headrick LA, Mutha S, Coleman MT, O’Donnell J, Miles PV. A framework for teaching medical students and residents about practice-based learning and improvement, synthesized from a literature review. Acad Med. 2003;78:748–756. 9. Taylor MJ, McNicholas C, Nicolay C, Darzi A, Bell D, Reed JE. Systematic review of the application of the plan-do-study-act method to improve quality in healthcare. BMJ Qual Saf. 2014;23:290–298. 10. Ghafur I, Lalloo D, Macdonald EB, Menon M. Development and implementation of clinical algorithms in occupational health practice. J Occup Environ Med. 2013;55:1443–1448. 11. Belk HD. Implementing continuous quality improvement in occupational health programs. J Occup Med. 1990;32:1184–1188. 12. Kuhle CS, Truitt F, Steffen M, et al. Improving patient-centered care: agenda-setting in occupational medicine. J Occup Environ Med. 2013;55: 479–482. 13. The Federal Motor Carrier Safety Administration. Complete Guide to Medical Examiner Certification. Transportation USDo; 2013. 14. Davidoff F, Batalden P, Stevens D, Ogrinc G, Mooney S. Publication guidelines for quality improvement in health care: evolution of the SQUIRE project. Qual Saf Health Care. 2008;17(suppl 1):i3–9. 15. Nabhan M, Elraiyah T, Murad M. Teaching the contrasting paradigms of evidence-based medicine and quality improvement in residency. Newsletter of the International Society for Evidence-Based Health Care. 2011:5–6. 16. Wong BM, Etchells EE, Kuper A, Levinson W, Shojania KG. Teaching quality improvement and patient safety to trainees: a systematic review. Acad Med. 2010;85:1425–1439. 17. Chera BS, Mazur L, Jackson M, et al. Quantification of the impact of multifaceted initiatives intended to improve operational efficiency and the safety culture: a case study from an academic medical center radiation oncology department. Pract Radiat Oncol. 2014;4:e101–e108. 18. Potisek NM, Malone RM, Shilliday BB, et al. Use of patient flow analysis to improve patient visit efficiency by decreasing wait time in a primary care-based disease management programs for anticoagulation and chronic pain: a quality improvement study. BMC Health Serv Res. 2007; 7:8. 19. Skeldon SC, Simmons A, Hersey K, et al. Lean methodology improves efficiency in outpatient academic uro-oncology clinics. Urology. 2014;83:992– 997. 20. Smith C, Wood S, Beauvais B. Thinking lean: implementing DMAIC methods to improve efficiency within a cystic fibrosis clinic. J Healthc Qual. 2011;33:37–46.
C 2015 American College of Occupational and Environmental Medicine
Copyright © 2015 American College of Occupational and Environmental Medicine. Unauthorized reproduction of this article is prohibited.
Teaching Quality Improvement in Occupational Medicine Improving the Efficiency of Medical Evaluation for Commercial Drivers Atsushi Sorita, MD, MPH, David Raslau, MD, MPH, Mohammad Hassan Murad, MD, MPH, and Mark W. Steffen, MD, MPH
Objective: To describe a successful, resident-led quality improvement (QI) project that improved the efficiency of the Department of Transportation (DOT) medical examination process. Methods: After learning QI principles through didactics, workshops, and online modules, residents led a QI project to streamline the process of the DOT examination. An interdisciplinary group of key stakeholders collaborated to analyze the process and to design and implement interventions. Results: Following the Model for Improvement and Lean concepts, residents ran seven Plan-Do-Study-Act cycles over a 4-month period with multiple iteration and testing changes. Compared with the baseline, the team successfully reduced the total visit time (from check-in to check-out) by 28 minutes (102 minutes vs. 130 minutes; P < 0.001). The accuracy of certificate issuance, as proxy for quality of the examinations, improved after the interventions. Conclusions: Residents successfully improved the efficiency of the DOT examination process.
T
he Accreditation Council for Graduate Medical Education and the Royal College of Physicians and Surgeons of Canada have emphasized that trainees in medical education should become competent in quality improvement (QI).1,2 In response, many training programs developed QI curricula for trainee physicians utilizing didactics and group sessions,3 online modules,4 and experiential learning;5 and some used multipronged approaches.6,7 Nevertheless, the framework for effective curriculum in QI has not been well established, and the integration of QI education in graduate medical training is still challenging.1,8 Although many training programs require that their trainees conduct a QI project, the application of an established QI framework is often not rigorous in actual projects.9 Moreover, the literature on the application of and the education on QI principles in occupational medicine remains sparse.10–12 In this study, we describe a demonstrable QI project, led by residents in Preventive, Occupational, and Aerospace Medicine, which followed formal QI framework and methodology, resulting in a meaningful process improvement in the Department of Transportation (DOT) medical examination.
CONTEXT AND BACKGROUND Effective in May 2014, the Federal Motor Carriers Safety Administration in the United States created the National Registry of Certified Medical Examiners and required that all DOT examiners receive training on the Federal Motor Carriers Safety Administration’s physical standards and pass an examination to verify their understanding.13 Because of the expected shortage of certified medical examiners, the division’s leadership expected an increase in the number of examinations performed in our setting. Nevertheless, From the Division of Preventive, Occupational, and Aerospace Medicine, Mayo Clinic, Rochester, Minn. The authors declare no conflicts of interest. Address correspondence to: Mark W. Steffen, MD, MPH, Division of Preventive, Occupational, and Aerospace Medicine, Mayo Clinic, 200 1st St SW, Rochester, MN 55905 ([email protected]). C 2015 by American College of Occupational and Environmental Copyright Medicine DOI: 10.1097/JOM.0000000000000394
JOEM r Volume 57, Number 4, April 2015
benchmarking identified a quality gap in the efficiency of our practice: the total visit time (from check-in to check-out) was perceived much longer than average local practice. The practice leadership prioritized a project to improve the efficiency of these examinations. A preliminary meeting led to the formation of an interdisciplinary working group in August 2013 that included administrators, providers, residents, nurses, clinical assistants, and secretaries. This group developed an aim statement: to decrease the total visit time and internal costs by an average of 25% within 6 months while maintaining the accuracy of certificate completion.
METHODS This study is reported according to the Standards for Quality Improvement Reporting Excellence guidelines.14 Following institutional guidelines, ethical review was not required for this QI initiative.
Setting This project was performed by residents in Preventive, Occupational, and Aerospace Medicine at the Mayo Clinic in Rochester, Minnesota. The program requires residents to complete a QI project during residency to meet the competency in practice-based learning and improvement set by the Accreditation Council for Graduate Medical Education. The faculty receives training in QI methodology through the Quality Academy, an institutional group staffed by QI experts that provides structured QI education for the staff. The faculty also completes modules in communication, leadership, and teamwork principles.
Curriculum Components A curriculum in QI has been implemented in the residency program since 2005 that combines didactics, workshop, online modules, and experiential learning (ie, learning from direct experience).15 Residents received didactics by faculty and through the Quality Academy about evidence-based medicine, critical appraisal of evidence, conducting a literature search, study design, and an overview of QI methods. Residents also attended workshop-format classes on value stream mapping, change management, and QI methodology (which covered identifying, measuring, and prioritizing opportunities for improvement, selecting and applying appropriate quality tools and methods, and understanding how to sustain long-term improvement). Furthermore, residents completed online modules that covered quality measurement, patient safety, principles of leadership, and health policy. A total of 3 weeks of protected time (three 1-week periods) was provided during one academic year for this project, as well as flexibility for attending classes and meetings. The faculty met frequently with residents during the project and debriefed about progress and planned for the next step. Faculty communicated with the practice leadership and with administrative and support staff to secure buy-in and support for change.
Planning the Intervention The initial step was to form an interdisciplinary team consisting of representatives of all those who would be impacted by changes to the current DOT process, including secretaries, clinical assistants, 453
Copyright © 2015 American College of Occupational and Environmental Medicine. Unauthorized reproduction of this article is prohibited.
JOEM r Volume 57, Number 4, April 2015
Sorita et al
registered nurses (RNs), and providers (nurse practitioners, physician assistants, and physicians). During the first multidisciplinary team meeting, roles were identified: leaders (the residents), a physician champion, project members, and supporters. A brainstorming session was held and input from all members regarding potential interventions was entertained. The entire process was mapped out by shadowing patients throughout their visit (Fig. 1). Next, value stream mapping was used to describe the current and future processes. The team calculated total wait time, total process time, lead time, value percentage added, and first-time quality. Using these tools and lean concepts, the team identified areas of waste, which primarily included waiting, rework, motion, and overprocessing types of waste. Multiple possible interventions were discussed to reduce the waste and improve the efficiency. These were grouped into several categories using an affinity diagram. The team made the decision to move forward using the Model for Improvement by the Institute for Healthcare Improvement and rapid Plan-DoStudy-Act (PDSA) cycles. This allowed multiple, rapid changes over a short period while at the same time minimizing interruptions to the flow of the clinical day. Monthly meetings were held to discuss the changes that were made and to make corrections or plan additional interventions as necessary.
To determine the effect of the QI efforts, several measures were assessed, including total visit time (check-in to check-out) and total time spent in each step (check-in, RN exam, provider examination, and check-out). All DOT examinations were included except those that had a gap in time between RN and provider appointments. Measurements were obtained by reviewing patient tracking logs in the electronic medical record. As a countermeasure to ensure that the examination process maintained the level of quality, the team measured the number of certification errors for the DOT examinations for clinic employees. This was accomplished by having the fleet service at the institution provide feedback if any errors were identified in the certificates that were being provided by our practice. These measures were collected and reviewed at every team meeting. The degree of error in the patient tracking logs was considered to be constant throughout the project period because the process of keeping the logs did not change. The team agreed with the validity and reliability of these measures. Data collection was done by the team leaders (residents). The team also evaluated the effectiveness and implementation of each individual intervention by conducting staff interviews and group discussions.
PROCESS MAP - NEW
PROCESS MAP - BASELINE Check-in
Exam for employees?
Methods of Evaluation
Secretary prepares the certificate* Yes
Patient fills out release of information form
Check-in
No RN checks vitals and rooms the patient
CA checks vitals and rooms the patient*
RN EXAM: Blood pressure Vision by Titmus or Snelling Hearing test Urine test
CA prefills the certificate*
RN EXAM: Vision by Titmus* Hearing test Urine test
RN returns the patient to waiting area
PROVIDER EXAM: History taking Physical exam Complete certificate
CA rooms the patient for provider exam
PROVIDER EXAM: History taking Physical exam Complete certificate
The patient goes to the front desk
Provider laminates the DOT certificate
CA checks accuracy and laminates the certificate*
The patient goes to the front desk
Check-out
Exam for employees?
Yes
Certificate retained at the front desk
*New processes implemented during the project RN: registered nurse
No Check-out
454
CA: clinical assistants
FIGURE 1. Baseline and new process maps. The figure shows the high-level baseline process map on the left, which was created in the planning phase of the project. At baseline, there were separate steps for institutional employees, nurse examinations were not standardized, and providers filled out the certificate. The new process map on the right shows simplified processes with changes indicated with asterisk. In the new process, secretary and CA prefill the certificate, CA rooms the patient instead of RN, RN examinations were standardized, and CA laminates the certificate for providers. CA, clinical assistant; RN, registered nurse.
C 2015 American College of Occupational and Environmental Medicine
Copyright © 2015 American College of Occupational and Environmental Medicine. Unauthorized reproduction of this article is prohibited.
JOEM r Volume 57, Number 4, April 2015
Teaching QI to Improve Efficiency of DOT Examination
TABLE 1. PDSA Cycles PDSA Cycle
Interventions
#1 9/18/2013
CAs provided verbal instructions to patients reminding them of the urine examination and filling out the DOT form RN examination was standardized to use the Titmus machine for visual acuity checks CAs take vitals and room the patients, which were previously done by RNs RN examination time was shortened from 60 to 30 min
#2 10/3
#3 10/14
#4 10/25
#5 11/18
#6 12/30
#7 1/22/2014
CAs handed folders to patients, which contained written instructions for them and the providers regarding important aspects of the process Patients were kept in the same room for RN and provider visits CAs began performing a quality check on the certificates and laminating them in place of the providers CAs no longer waited for patients to complete the DOT form before rooming RNs began directly prompting the providers to see the patient as soon as the RNs finish their visits Secretaries and CAs prefilled certificates with information specific to patients and providers Redundant documents for the visit were eliminated Process for DOT examinations for institutional employees and others was standardized RN examination time was made 30 min for all appointments
Rationale for Intervention Examination was delayed waiting for patients who had urinated before examination or had not filled out the form Not all RNs were using the Titmus machine, which allows for a more expedited examination CAs normally do these for other clinic visits and this will free up RN time
Success of Implementation Fair (not all CAs were consistent in reminding)
Minimal
Good
Moderate
Good
Moderate
Most RNs did not need the full 60min with the reduced responsibilities
Fair (not all examinations were made 30 min) To save CAs’ time of verbally conveying Fair (felt cumbersome the information and to provide the by some CAs) providers with information about the changes To eliminate waste of having patients Fair (not all providers return to waiting area to be roomed complied with it) again Checking and laminating the certificates Good was taking time for providers
Patients can complete the form during waiting periods between examination components RNs were asking CAs to prompt the providers, which created waiting and waste This would reduce work for providers
Impact
Large
Moderate
Moderate
Moderate
Good
Minimal
Good
Moderate
Good
Moderate
These documents were not necessary for Good a DOT examination Good Slight nuanced differences between examinations for institutional employees and others created confusion Not all RN visits were 30 min because of Good system issue
Minimal Moderate
Large
CA, clinical assistant; DOT, Department of Transportation; RN, registered nurse.
Analysis Data were compiled to determine the total visit time before and after the intervention. A control chart (Individual-chart) for total visit time was constructed to determine whether the process was under control and whether the visit time changed significantly before and after the intervention. Comparisons were made between baseline (July to August 2013) and intervention periods (September 2013 to March 2014). Data during the intervention period were divided into two types: visits with 60-minute RN appointment and ones with 30-minute RN appointment. For continuous variables, numbers were expressed as average with standard deviation. A two-sample t test was used for two groups and analysis of variance with Turkey’s method was used for more than two groups. For discrete variables, numbers were expressed as percentage, and we used a chi-square test for comparison. All
tests were two-sided, and P < 0.05 was considered significant. For the above analysis, we used the JMP Software, version 9.0 (SAS Institute, Cary, NC).
RESULTS Context of Intervention The environment and culture at our institution strongly support change and improvement efforts. A team-based approach is embedded in our everyday care of patients, which allowed us to establish an interdisciplinary team that functioned cohesively. Nurse team members took initiatives and led some of the changes described below. The actual improvements were implemented primarily by nursing and clerical staff members. The presence of the residents on the floor every day allowed for immediate corrections when glitches
C 2015 American College of Occupational and Environmental Medicine
455
Copyright © 2015 American College of Occupational and Environmental Medicine. Unauthorized reproduction of this article is prohibited.
JOEM r Volume 57, Number 4, April 2015
Sorita et al
and issues with implementation inevitably arose. The support of the division’s leadership allowed this QI initiative to thrive, and ultimately, to be successful.
Intervention In the planning phase, the team identified the most important waste as waiting time between the RN examination and the provider examination. RN appointment was set at 60 minutes, whereas RNs finished their examinations in 32 minutes on average. The patient was escorted back to the waiting area after the RN examination to wait for the provider examination, creating a rework for front desk staff (Fig. 1). A total of seven PDSA cycles were conducted in the 4-month period. The timeline, rationales, details, and perceived success and impact of each intervention in the PDSA cycles are described in Table 1. Overall, the interventions were primarily aimed at eliminating waiting time for the patient. The team also aimed at standardizing the process of the DOT examination, eliminating non–value-added steps, and maximizing utilization of the expertise of each profession, specifically increasing the responsibilities of the secretaries and clinical assistants and delegating some of the responsibilities of the RNs and providers to them. Several changes were abandoned or modified after testing. Institutional policy was the barrier for implementing drastic changes, such as eliminating the RN visit entirely or at least eliminating the documentation requirements for the RN. The team was dedicated to providing potential changes that would be implementable given the institutional constraints. The provider examination component was left untouched because of the perceived difficulty in implementing any changes to this aspect of the examination. The team discovered after additional conversation with the Appointment Coordination Office that the proposed reduction of RN appointments from 60 to 30 minutes had not entirely occurred until January 2014. This provided a concurrent control of 60 minutes versus 30 minutes for the RN visit. This led to the
Outcomes The entire process for the DOT examination was simplified after the interventions (Fig. 1). Overall, 32 and the 101 DOT examinations were conducted at baseline (July to August, 2013) and during the intervention period (September 2013 to March 2014), respectively. After excluding visits with a gap in time between RN and provider appointments, the authors included in the analysis 25 visits at the baseline and 80 visits during the intervention period (38 visits with 30-minute and 42 visits with 60-minute RN appointment). The control chart showed a stable process at baseline with a significant shift after the intervention in visits with 30-minute RN appointment (Fig. 2). Compared with the baseline, the average total visit time decreased by 28 minutes (102 minutes vs 130 minutes; P < 0.001) for 30-minute RN visits, with a relative reduction by 21%. The average time for those with 60-minute RN appointment did not change significantly from the baseline (137 minutes vs 130 minutes; P = 0.51). When comparing each step of the three types of visits (baseline, 30-minute RN, and 60-minute RN visits), no difference was observed for check-in or provider examination, whereas the waiting time between RN and provider visit was significantly shortened in 30-minute RN visits (Fig. 3). It was estimated that costs were reduced by approximately 25% because the nurse examination was cut in half and the equipment usage was standardized. Despite the numerous changes that occurred, RN 30 minutes (intervention)
RN 60 minutes (control)
Baseline 250
Total Visit Time (in minutes)
discovery that the most significant intervention was the reduction of RN responsibilities, which enabled them to perform all essential components of their examination in the shorter appointment times. All proposed changes to the DOT examination process were discussed and approved by the Practice Committee in January 2014. The success of the QI effort was recognized, and the plan was made to further improve the quality by standardizing the examination for providers.
200 Upper Control Limit
150
Average
100
50 Lower Control Limit 10
20
30
40
50
60
70
80
90
100
Number of Exams
FIGURE 2. Control chart for total visit time. The control chart shows changes in total visit time before and after implementation of the changes. The visit time was stable at the baseline. No significant improvement in visit time was seen in visits, with 60-minute RN appointment (in the middle). There was a significant shift (shortening of visit time) in visits with 30-minute RN appointment during the project. RN, registered nurse. 456
C 2015 American College of Occupational and Environmental Medicine
Copyright © 2015 American College of Occupational and Environmental Medicine. Unauthorized reproduction of this article is prohibited.
JOEM r Volume 57, Number 4, April 2015
Teaching QI to Improve Efficiency of DOT Examination
Nurse Exam
Baseline
Check-in 21
RN 60 minutes (control)
Check-in 30
RN 30 minutes (intervention)
Check-in 23
Nurse Appointment
Provider Exam
28
Nurse Appointment
Nurse Appointment
Provider Appointment
Check-out 32
Provider Appointment
60
30
Provider Appointment
Check-out 49
Check-out 47
Check-out 49
FIGURE 3. Visit time by step (minutes). The figure shows visit time stratified by components of the visit. At baseline, there were 21 minutes from check-in to nurse appointment, 32 minutes for nurse visit, 28 minutes between nurse visit and provider visit (waiting time), and 49 minutes for provider visit. The waiting time was shortened significantly in visits with 30-minute nurse appointment as shown at the bottom. There were no significant changes in time from check-in to nurse appointment, nor for provider visit. PDSA #1
25
PDSA #5
PDSA #2
PDSA #6
PDSA #3
FIGURE 4. Accurate completion of certificates. The figure shows the number of accurately and inaccurately completed certificates for institutional employees per month with the timing of PDSA cycles. There were substantial number of errors in certificates until October to November 2013, and then the proportion of inaccurately completed certificates became smaller. PDSA, Plan-DoStudy-Act.
Number of Exams
20
PDSA #7
PDSA #4
15 Inaccurate Accurate
10
5
0 Jul-13
certification errors occurred less frequently after these interventions (Fig. 4). This was likely due to addition of the quality check step by front desk staff that aimed at reducing the work of providers.
DISCUSSION Summary We described an exemplar QI project led by residents that yielded meaningful process improvement in the DOT medical examination. The project successfully reduced the total visit time for the DOT examination by 28 minutes while improving accuracy of the certificate issuance. The project followed a rigorous QI methodology and the changes were implemented into routine clinical practice.
Relation to Other Evidence The residents learned QI methodology from didactics, workshops, online modules, and experiential learning. They also had dedicated time to complete a QI project and had sufficient support by the faculty. The institutional leadership and environment also supported the QI efforts undertaken by residents. These factors influenced successful completion of the QI project by residents.16 Shortening visit time by 21% (28 minutes) is robust and comparable to other QI projects that successfully improved the efficiency of outpatient clinic visits.6,17–20
Aug-13
Sep-13
Oct-13
Nov-13
Dec-13
Jan-14
Feb-14
Mar-14
Month
Limitations This report has inherent limitations. First, the project did not randomize patients to interventions and thus the quality of evidence is limited. Nevertheless, the patients were assigned to 30- or 60minute RN appointment inadvertently because of incomplete implementation, providing the concurrent control. Therefore, the significant reduction in total visit time has a modest strength of evidence. Second, visit time was approximated by available time stamps in the electronic medical record, and thus the time estimates may not be accurate. Nevertheless, the degree of bias should have remained the same over time and thus this limitation is unlikely to have affected the overall results. Third, we assume that faculty expertise and mentoring were instrumental for the success of the project. If this is true, the described project may not be replicated in institutions with limited QI expertise in the faculty. Nevertheless, with increased emphasis on developing competency in QI for trainee physicians, the lack of expertise in QI by the faculty, if any, should be considered a quality gap that needs to be closed.
Interpretation Success factors perceived from the team members were early involvement and engagement of key stakeholders and SMART (specific, measurable, attainable, relevant, and time-bound) goal setting in the aim statement. Although shortening nursing visit time to
C 2015 American College of Occupational and Environmental Medicine
457
Copyright © 2015 American College of Occupational and Environmental Medicine. Unauthorized reproduction of this article is prohibited.
JOEM r Volume 57, Number 4, April 2015
Sorita et al
30 minutes put more work on nursing staff by opening up their schedule, this change was proposed and implemented by nursing staff themselves. This likely contributed to the success of implementation of this particular intervention. Even with the overall success of the project, we were not able to shorten the total visit time by 25% as specified in the aim statement. This was mainly due to factors that were out of control of the team: check-in time of the patient and delay of providers due to other patient appointments. On the basis of lessons learned, the team has already started discussion to standardize the provider examination to further improve quality and efficiency. As the volume of the DOT examinations increases in our worksite, additional opportunity exists to further improve the efficiency: creating a dedicated DOT examination clinic or utilizing support staff to the fullest extent. Furthermore, lessons learned through the project could be applied to other types of examinations in the worksite.
CONCLUSIONS Residents successfully improved the efficiency of the DOT examination while maintaining the quality of the examination through the application of rigorous QI methodology.
REFERENCES 1. Wong BM, Levinson W, Shojania KG. Quality improvement in medical education: current state and future directions. Med Educ. 2012;46:107–119. 2. Wong BM, Ackroyd-Stolarz S, Bukowskyj M, et al. The CanMEDS 2015 Patient Safety and Quality Improvement Expert Working Group Report. Ottawa: The Royal College of Physicians and Surgeons of Canada; 2014. 3. Reznek MA, Digiovine B, Kromrei H, Levine D, Wiese-Rometsch W, Schreiber M. Quality education and safe systems training (QuESST): development and assessment of a comprehensive cross-disciplinary resident quality and patient safety curriculum. J Grad Med Educ. 2010;2:222–227. 4. Kerfoot BP, Conlin PR, Travison T, McMahon GT. Web-based education in systems-based practice: a randomized trial. Arch Intern Med. 2007;167: 361–366. 5. Sellers MM, Hanson K, Schuller M, et al. Development and participant assessment of a practical quality improvement educational initiative for surgical residents. J Am Coll Surg. 2013;216:1207–1213, 1213-e1201. 6. Diaz VA, Carek PJ, Dickerson LM, Steyer TE. Teaching quality improvement in a primary care residency. Jt Comm J Qual Patient Saf. 2010;36:454–460.
458
7. Oyler J, Vinci L, Johnson JK, Arora VM. Teaching internal medicine residents to sustain their improvement through the quality assessment and improvement curriculum. J Gen Intern Med. 2011;26:221–225. 8. Ogrinc G, Headrick LA, Mutha S, Coleman MT, O’Donnell J, Miles PV. A framework for teaching medical students and residents about practice-based learning and improvement, synthesized from a literature review. Acad Med. 2003;78:748–756. 9. Taylor MJ, McNicholas C, Nicolay C, Darzi A, Bell D, Reed JE. Systematic review of the application of the plan-do-study-act method to improve quality in healthcare. BMJ Qual Saf. 2014;23:290–298. 10. Ghafur I, Lalloo D, Macdonald EB, Menon M. Development and implementation of clinical algorithms in occupational health practice. J Occup Environ Med. 2013;55:1443–1448. 11. Belk HD. Implementing continuous quality improvement in occupational health programs. J Occup Med. 1990;32:1184–1188. 12. Kuhle CS, Truitt F, Steffen M, et al. Improving patient-centered care: agenda-setting in occupational medicine. J Occup Environ Med. 2013;55: 479–482. 13. The Federal Motor Carrier Safety Administration. Complete Guide to Medical Examiner Certification. Transportation USDo; 2013. 14. Davidoff F, Batalden P, Stevens D, Ogrinc G, Mooney S. Publication guidelines for quality improvement in health care: evolution of the SQUIRE project. Qual Saf Health Care. 2008;17(suppl 1):i3–9. 15. Nabhan M, Elraiyah T, Murad M. Teaching the contrasting paradigms of evidence-based medicine and quality improvement in residency. Newsletter of the International Society for Evidence-Based Health Care. 2011:5–6. 16. Wong BM, Etchells EE, Kuper A, Levinson W, Shojania KG. Teaching quality improvement and patient safety to trainees: a systematic review. Acad Med. 2010;85:1425–1439. 17. Chera BS, Mazur L, Jackson M, et al. Quantification of the impact of multifaceted initiatives intended to improve operational efficiency and the safety culture: a case study from an academic medical center radiation oncology department. Pract Radiat Oncol. 2014;4:e101–e108. 18. Potisek NM, Malone RM, Shilliday BB, et al. Use of patient flow analysis to improve patient visit efficiency by decreasing wait time in a primary care-based disease management programs for anticoagulation and chronic pain: a quality improvement study. BMC Health Serv Res. 2007; 7:8. 19. Skeldon SC, Simmons A, Hersey K, et al. Lean methodology improves efficiency in outpatient academic uro-oncology clinics. Urology. 2014;83:992– 997. 20. Smith C, Wood S, Beauvais B. Thinking lean: implementing DMAIC methods to improve efficiency within a cystic fibrosis clinic. J Healthc Qual. 2011;33:37–46.
C 2015 American College of Occupational and Environmental Medicine
Copyright © 2015 American College of Occupational and Environmental Medicine. Unauthorized reproduction of this article is prohibited.
