Practice Quality Improvement During Residency: Where Do We Stand and Where Can We Improve? Sadia Choudhery, MD, Michael Richter, MD, Alvin Anene, MD, Yin Xi, MS, Travis Browning, MD, David Chason, MD, Michael Craig Morriss, MD Rationale and Objectives: Completing a systems-based practice project, equivalent to a practice quality improvement project (PQI), is a residency requirement by the Accreditation Council for Graduate Medical Education and an American Board of Radiology milestone. The aim of this study was to assess the residents’ perspectives on quality improvement projects in radiology. Materials and Methods: Survey data were collected from 154 trainee members of the Association of University Radiologists to evaluate the residents’ views on PQI. Results: Most residents were aware of the requirement of completing a PQI project and had faculty mentors for their projects. Residents who thought it was difficult to find a mentor were more likely to start their project later in residency (P < .0001). Publication rates were low overall, and lack of time was considered the greatest obstacle. Having dedicated time for a PQI project was associated with increased likelihood of publishing or presenting the data (P = .0091). Residents who rated the five surveyed PQI steps (coming up with an idea, finding a mentor, designing a project, finding resources, and finding time) as difficult steps were more likely to not have initiated a PQI project (P < .0001 for the first four and P = .0046 for time). Conclusion: We present five practical areas of improvement to make PQI a valuable learning experience: 1) Increasing awareness of PQI and providing ideas for projects, 2) encouraging faculty mentorship and publication, 3) educating residents about project design and implementation, 4) providing resources such as books and funds, and 5) allowing dedicated time. Key Words: Practice quality improvement; systems-based practice; core competency; maintenance of certification; milestones. ªAUR, 2014
T
he world of health care is rapidly changing, and a significant component of that change is the emphasis on accountability to the public for the quality of medical practice. Quality improvement (QI) in medicine has become vital to the daily practice of physicians and the training of future physicians. QI in radiology encompasses improving the effectiveness of diagnostic and therapeutic procedures, selecting the appropriate imaging services, ensuring safety and quality of services delivered, and overseeing the efficiency and management of all imaging services (1). To highlight the importance of QI, the American Board of Radiology made Practice Quality Improvement (PQI) one of the four core components for Maintenance of Certification (MOC). At least 23 other medical specialty boards besides the American Board of Radiology (ABR) have a requirement for some form of practice improvement under the MOC plan
Acad Radiol 2014; -:1–8 From the Department of Radiology, UT Southwestern Medical Center, 5323 Harry Hines Blvd. Dallas, TX 75390 (S.C., M.R., A.A., Y.X., T.B., D.C., M.C.M.). Received October 8, 2013; accepted November 24, 2013. Address correspondence to: S.C. e-mail: [email protected] ªAUR, 2014 http://dx.doi.org/10.1016/j.acra.2013.11.021
developed by the American Board of Medical Specialties in 2000 (2). When MOC was officially introduced in 2007 by the ABR, radiologists were only required to perform one PQI project every 10 years (3). The requirements were then adjusted to three PQI projects every 10 years (4). Recently, with the introduction of ‘‘continuous certification,’’ the expected change in requirement to one project every 3 years occurred (3). All the current residents in radiology programs nationally will be certified under this new ABR requirement, and preparing residents for the future has become a vital component of training across the country. Additionally, one of the six core competencies of the Accreditation Council for Graduate Medical Education (ACGME) during residency is systems-based practice (SBP) (5). Systems-based practice is defined by the ACGME as ‘‘awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value’’ (5). Since 2008, the ACGME has required participation of radiology residents in at least one systems-based practice project with documentation of participation in such an activity (6). The ABR has formally incorporated this requirement as one of the 12 required milestones to be achieved by radiology residents during their training (7). SBP is a challenging 1
Academic Radiology, Vol -, No -, - 2014
CHOUDHERY ET AL
TABLE 1. American Board of Radiology Practice Quality Improvement Categories of Resident-Led Projects by PGY
PGY-2 PGY-3 PGY-4 PGY-5
Patient Safety
Accuracy of Interpretation
Report Timeliness
Practice Guidelines and Technical Standards
Referring Physician Surveys
25* 24.53 20 21.43
37.5 16.98 16.67 21.43
0 13.21 0 28.57
37.5 39.62 56.67 21.43
0 5.66 6.67 7.14
PGY, postgraduate year. *% of survey respondents in the respective PGY.
competency to define, teach, and evaluate (8) and was reported to be the most difficult competency to implement in radiology residency programs in the 2005 annual survey of the Association of Program Directors in Radiology (9). A systems-based practice project performed during residency is essentially equivalent to a PQI project performed after board certification. This study examined radiology residents’ perspectives on QI and explored the obstacles faced by them during the process of performing QI or systems-based practice projects. The aim of this study was to determine the awareness of residents of the PQI requirement, hindrances faced by residents in the process of project development and completion, and the percentage of projects published in the scientific literature or presented at meetings. A secondary goal was to provide an insight into steps that may be taken within academic radiology departments to facilitate the process of completing PQI projects in a way that would make PQI a valuable learning experience for residents.
MATERIALS AND METHODS An eight-question electronic survey using SurveyMonkey (www.surveymonkey.com) was sent out to 51 residents at our institution and 2039 registered trainee members of the Association of University Radiologists with responses collected for approximately 8 weeks (see Appendix). This survey was distributed in May 2013, near the end of the academic year, to avoid bias related to having initiated a new residency program or year. The survey initially queried the participants about their postgraduate year (PGY) level and their awareness of the requirement of doing a SBP or QI project during residency. Next, a series of questions was asked to learn about the PQI policy at the residents’ institutions including availability of dedicated time and a formal process to assist in development and completion of QI projects. The subsequent sequence of questions was targeted toward gauging the time frame of initiation and completion of QI projects during residency. The participants were additionally asked to state which of the five ABR PQI categories (patient safety, accuracy of interpretation, report timeliness, practice guidelines and technical standards, and referring physician surveys) best described their QI projects. They were also questioned regarding faculty mentorship and publication and presentation of their projects. 2
Last, they were asked to rate the hardship they faced on a scale of 5 ranging from very easy to very difficult in completing these five PQI steps: coming up with an idea, finding a mentor, designing and implementing the project, finding resources such as books and funds, and finding time. Participants were also provided an opportunity to leave comments about the QI process at the end. The data were collected anonymously and analyzed using descriptive statistics and chi-square and Fisher exact tests with statistical significance defined as P < .05. This survey was institutional review board exempt.
RESULTS Overview of Participants, Awareness of QI Projects, and Topics Covered
A total of 154 residents nationally responded to the survey, of which approximately 11% were PGY-2, 54% were PGY-3, 25% were PGY-4, and 10% were PGY-5 residents. Ninetyone percent of the residents were aware of the requirement to complete a systems-based practice project or a QI project during residency, whereas 9% were not aware of this requirement. All PGY-5 respondents knew about the PQI requirement for graduation, but 12% of PGY-2, 11% of PGY-3, and 8% of PGY-4 participants were not aware of this requirement. The ABR classifies QI projects into five categories: patient safety, accuracy of interpretation, report timeliness, practice guidelines, and technical standards. The category of practice guidelines and technical standards accounted for the highest number of projects at 42%. Twenty-three percent of projects fit the category of patient safety, whereas 19% were related to accuracy of interpretation. The categories of report timeliness and referring physician surveys defined the remaining 16% of the projects. The distribution of these five QI project categories by PGY is illustrated in Table 1. Timeframe of QI Projects
The majority of the survey participants had begun their PQI project in their first 2 years of residency, with 52% initiating them during PGY-2 and 40% during PGY-3 (Fig 1). Of the remaining 8% that had started their projects during their last
Academic Radiology, Vol -, No -, - 2014
QUALITY IMPROVEMENT DURING RESIDENCY
Figure 1. Postgraduate year (PGY) of survey respondents at initiation of a quality improvement project by percentage of residents.
Figure 2. Status of quality improvement projects among the different postgraduate year (PGY) residents by percentage of residents.
2 years of residency, 1% of them were PGY-5. Forty-three percent of survey respondents had completed their QI projects, whereas 27% had currently active projects and 30% had not initiated a project. For PGY-2 residents, 0% had completed a project, 41% had an active project, and 59% had yet to begin a project (Fig 2). For PGY-3 residents, 37% had completed a project, 26% had an active project, and 37% had yet to begin a project. For PGY-4 residents, 58% had completed a project, 26% had an active project, and 16% had yet to begin a project. For PGY-5 residents, 87% had completed a project, 13% had an active project, and 0% had yet to begin a project. Institutional QI Policy, Faculty Mentorship, and Publication of QI Projects
A total of 71% of residents stated that they had a process for organizing a QI project at their institution, whereas 29% indicated that they did not (Table 2). Having a QI process at one’s institution was significantly associated with having initiated or completed a QI project at the time of this survey administration (P = .0007). Additionally, 81% of the survey participants performed QI projects under faculty mentorship, whereas 19% of the respondents did not have faculty mentors
TABLE 2. Availability of Dedicated QI Process and Time, Faculty Mentorship, and Publication/Presentation Rates of QI Projects among Surveyed Residents
Availability of a QI process Availability of time for QI Faculty mentorship Publication/presentation of a QI project or plans to do so in the future
Yes
No
71* 26 81 39
29 74 19 61
QI, quality improvement. *% of total survey respondents.
(Table 2). Only 26% of respondents had dedicated time during residency to work on a QI project and 74% did not (Table 2). Forty-seven percent of PGY-5 respondents indicated that they had dedicated time for performing QI, whereas only 24% of PGY-2, 29% of PGY-3, and 13% of PGY-4 participants stated the same. About 19% of the trainees had presented or published their project results and 20% had plans to do so (Table 2). The remaining residents indicated that they did not have plans to publish or present their projects at a meeting. There was a 3
Academic Radiology, Vol -, No -, - 2014
CHOUDHERY ET AL
Figure 3. Publication and presentation rates of quality improvement projects among residents who have dedicated time versus those who do not by percentage of residents.
TABLE 3. Resident Rating of Each of the Five Surveyed Quality Improvement Steps by PGY
PGY-2 Very easy Easy Somewhat difficult Difficult Very difficult PGY-3 Very easy Easy Somewhat difficult Difficult Very difficult PGY-4 Very easy Easy Somewhat difficult Difficult Very difficult PGY-5 Very easy Easy Somewhat difficult Difficult Very difficult
Coming up with an Idea
Finding a Mentor
Designing and Implementing a Project
Finding Resources
Finding Time
0* 20 70 10 0
10 70 20 0 0
0 30 70 0 0
0 50 50 0 0
0 10 70 10 10
19 46 25 5 5
32 54 5 5 4
7 44 33 9 7
7 49 26 9 9
3 19 32 23 23
21 38 29 12 0
16 36 36 6 6
6 26 47 21 0
6 41 25 22 6
0 15 37 21 27
13 60 20 7 0
20 60 7 13 0
7 73 13 7 0
7 53 26 7 7
13 34 20 20 13
PGY, postgraduate year. *% of survey respondents in respective post-graduate year (PGY).
significant relationship between having dedicated time for a QI project and the likelihood of presentation or publication (P = .0091) (Fig 3). Hindrances Encountered during QI Initiatives
The survey respondents were asked about the obstacles they faced in the process of performing QI projects. Table 3 illustrates the surveyed difficulty ratings given by each PGY for the five PQI steps: coming up with an idea, finding a mentor, designing and implementing a project, finding resources, and finding time. Finding a mentor was considered overall the least challenging step in performing a QI project, receiving ‘‘very easy’’ or ‘‘easy’’ rating by 75% of the partici4
pants. Lack of time was thought to be the greatest obstacle and was rated either ‘‘somewhat difficult,’’ ‘‘difficult,’’ or ‘‘very difficult’’ by 78% of participants. We further looked at relationships between these hindrances and the status of the QI project at the time of the survey. For this, we categorized ‘‘easy’’ as including both ‘‘very easy’’ and ‘‘easy’’ and ‘‘difficult’’ as including ‘‘somewhat difficult,’’ ‘‘difficult,’’ and ‘‘very difficult.’’ The status of the QI project was defined as having been initiated (either completed or in progress) or not initiated. There was a significant relationship between the rating given, either easy or difficult, to each of the five QI steps (coming up with an idea, finding a mentor, designing a project, finding resources, and finding time given clinical and research workload) and the status of
Academic Radiology, Vol -, No -, - 2014
the QI project (P < .0001 for the first four listed hindrances and P = .0046 for the hindrance of finding time). Residents who thought these PQI steps were easy were more likely to have initiated a QI project at the time of this survey administration. Additionally, residents who thought it was difficult to find a mentor in comparison to those who thought it was easy were more likely to start their project during their last 2 years of residency versus the first 2 years of residency (P < .0001). DISCUSSION With increasing demands by the public for physicians to be held accountable for errors in health care delivery (10), demand for QI in medicine will only continue to increase. As a result of these demands for accountability in the quality of medicine, the ABR and at least 23 other medical specialty boards have made QI projects in some form a requirement for recertification (2). The concept of requiring physicians to play an active role in improving the quality of care they provide has also influenced the education of trainee physicians. SBP, considered equivalent to PQI, is one of the ACGME’s required six core competencies for resident education. From developing SBP curriculum and electives (11–13) to establishing dedicated departmental PQI systems (3), radiology departments have incorporated SBP and PQI into their resident education in a variety of ways. Although studies have examined the views of practicing radiologists regarding QI (14), few data exist about radiology residents’ perspectives and practice of QI. Our study provides insight into the radiology trainees’ views of performing QI projects and highlights important areas for improvement in QI education. In our study, most of the residents were aware of the requirement of a PQI or a SBP project. About 9% of the residents did not know about this requisite, and a vast majority of these residents, not surprisingly, were in their first 2 years of residency. However, up to 8% of the PGY-4 respondents were not cognizant of the requirement to perform a QI project. This translated to about 16% of PGY-4 residents not having started a project near the beginning of their last year of residency and about 13% of PGY-5 residents not having completed a project near the completion of their residency. With the institution of the ABR Milestone Project in July 2013, initiation of a PQI project is required for level 3 of the SBP Quality Improvement Milestone and completion of a project is required for attaining level 4 (7). Thus, it is the expectation of the ABR that PGY-4 residents are identifying and working on QI projects that should then be completed no later than the PGY-5 year. There is an important need for increasing awareness of the requirement of PQI and for encouraging earlier initiation and fulfillment of this requisite for progress along the milestones pathway. Given also the institution of the ABR Core Exam, which will be taken at the end of the PGY-4 year, it will be advantageous for residents to begin these projects earlier in their residency so that they do not conflict with time spent studying for the
QUALITY IMPROVEMENT DURING RESIDENCY
board examination. Launching a project earlier would allow residents more time to design and complete a meaningful project suitable for publication or presentation. The question remains regarding how to teach and increase awareness of PQI. In their systematic review, Boonyasai et al. demonstrated that dedicated QI curricula can increase physicians’ knowledge or confidence to perform QI (15). Several radiology departments have experimented with various QI teaching methodologies. One department developed a program that introduced PGY-2 residents to QI and SBP through lectures (9). The PGY-2 residents were then required to submit a written proposal and present it to their peers 6 months after the beginning of their residency. They were given 12 to 18 months to work on their projects with a mandatory presentation of results at the end of their PGY3 year. This approach resulted in timely completion of QI projects and increased scholarly activities. This is just one example of how hands-on QI teaching can be performed and reinforced in residency. Our results indicate that a majority of residents are at institutions with some form of a departmental QI process in place. On the other hand, only about 26% of residents have any dedicated time to work on a QI project. This indicates a discrepancy in the number of institutions that have a formally organized QI process and that allow protected QI time for residents. Residents with an institutional QI process are more likely to have initiated a QI project. Although only 39% of respondents had or intended to publish or present their work, residents with dedicated time for a QI project were more likely to do so. It can therefore be inferred that although residents are more likely to initiate their QI efforts at institutions with a QI process, they are less likely to publish or present their work if they do not have dedicated time for their projects. Our study is consistent with the results of systematic reviews in which lack of time (16) and competing educational demands (17) have been shown to be major barriers to resident participation in QI initiatives. Studies have also shown that residents have a low publication rate in general during residency and that lack of time, funding, and research experience contribute to this low research productivity (18). It can, therefore, be concluded that to make PQI a truly worthwhile learning experience for residents, it is not enough to just have a QI system. It is also vital to allow residents dedicated time for PQI, which would increase research productivity, and to mentor them in their publication efforts. A senior research elective is one possibility in which a resident may have dedicated time to complete a QI project and prepare it for publication or presentation at a scientific meeting. The questions regarding obstacles faced during QI projects revealed that residents who thought it was easy to come up with an idea, find a mentor, design and implement a project, find resources such as books and funds, and find time for project completion were more likely to have initiated a QI initiative in comparison to those who thought it was difficult to do so. Although finding a mentor was rated difficult less frequently than any of the other QI steps, up to 19% of 5
CHOUDHERY ET AL
resident respondents did not have faculty mentors and those who thought it was difficult to find a mentor were more likely to start a QI project later in residency. The difficulty level of these five surveyed PQI steps presents an opportunity for radiology departments to improve their QI process. To introduce QI project ideas and to assist in design and implementation, an organized elective or curriculum may be used. Additionally, everyday clinical practice may be used to help residents discover prospective QI initiatives (8). To facilitate the process of finding a mentor, departments may operate a central database of proposed and ongoing projects by residents and faculty members. Furthermore, radiology departments can make teaching QI more valuable by allowing dedicated time to residents for performing PQI initiatives, offering helpful books, and providing or suggesting sources for funding. Additional obstacles to consider that have been identified in prior studies include residents’ lack of knowledge of data collection and analysis techniques (19,20), absence of previous QI training (21), hierarchical culture of medicine (22), and faculty skepticism about resident-led QI projects (23). Solutions to these problems might include collaboration with successful mentors, assistance by department statisticians, and coaching by the residency program director. Because most faculty members themselves are now required to complete PQI projects, it is reasonable to coach faculty in the value of collaborating with residents to accomplish PQI requirements for the faculty and trainee. Several limitations of our study must be considered. Our sample size was relatively small, and this was a survey study, introducing response bias. Lack of reminders and incentives to take the survey may have contributed to the small sample size. We also had significant representation from PGY-2 and PGY-3 residents, with only 10% representation from PGY-5 residents, who were likely occupied with studying for oral boards at the time of this survey administration. With a higher number of younger residents in our study who are expected to have more difficulty in the QI process, we might have overestimated the value of hindrances faced by residents during QI initiatives. Additionally, some of our questions used abstract terminology, such as ‘‘project design and implementation,’’ which added subjectivity to our data. An additional limitation of our survey design was our use of two categories for defining easy—‘‘very easy’’ and ‘‘easy’’—and three categories for designing difficult—‘‘somewhat difficult,’’ ‘‘difficult,’’ and ‘‘very difficult’’—for rating of a QI step. Retrospective adjustment of these categories is challenging because one does not know if participants who picked ‘‘somewhat difficult’’ were aiming for the middle of the scale or truly thought there was some difficulty involved with the respective QI step and would have picked ‘‘difficult’’ or ‘‘very difficult’’ if the category of ‘‘somewhat difficult’’ did not exist. The radiology residents of today will be expected to participate in QI initiatives throughout the rest of their careers. The ACGME requirement of completing a systems-based practice project as well as the ABR Milestone requirement can be used to prepare trainee physicians for their future QI endeavors. 6
Academic Radiology, Vol -, No -, - 2014
Training in QI is currently an underrepresented part of resident education. QI needs to be taught so that our future radiologists are not only prepared to fulfill their ABR QI requirements, but also are trained to be transformers of the quality of health care they provide to their patients. Increasing awareness of PQI and providing residents with ideas for projects, encouraging faculty mentorship and publication of QI projects, educating residents about project design and implementation, providing resources such as books and funds, and allowing dedicated time for QI initiatives are some ways we can make QI a valuable educational experience during residency.
REFERENCES 1. Kruskal JB, Eisenberg R, Sosna J, et al. Quality initiatives: quality improvement in radiology: basic principles and tools required to achieve success. Radiographics 2011; 31:1499–1509. 2. ABMS Maintenance of Certification. Available at: http://www.abms.org/ maintenance_of_certification/ABMS_MOC.aspx. Accessed October 1, 2013. 3. Kouo T. Experience with a practice quality improvement system in a university radiology department. J Am Coll Radiol 2012; 9:814–819. 4. Maintenance of Certification Part IV: ABR Guide to Practice Quality Improvement 2012. Available at: http://www.theabr.org/sites/all/themes/ abr-media/pdf/PQI_2012.pdf. Accessed October 1, 2013. 5. Dyne PL, Strauss RW, Rinnert S. Systems-based practice: the sixth core competency. Acad Emerg Med 2002; 9:1270–1277. 6. Amis ES, Jr. New program requirements for diagnostic radiology: update and discussion of the more complex requirements. AJR Am J Roentgenol 2008; 190:2–4. 7. The Diagnostic Radiology Milestone Project. Available at: http://www. acgme-nas.org/assets/pdf/Milestones/DiagnosticRadiologyMilestones.pdf. Accessed October 7, 2013. 8. Johnson JK, Miller SH, Horowitz SD. Systems-based practice: improving the safety and quality of patient care by recognizing and improving the systems in which we work. In: Henriksen K, Battles JB, Keyes MA, et al., eds. Advances in patient safety: new directions and alternative approaches. Rockville, MD: Agency for Healthcare Research and Quality. Available at: http://www.ncbi.nlm.nih.gov/books/NBK43731/; 2008. 9. Buchmann RF, Deloney LA, Donepudi SK, et al. Development and implementation of a systems-based practice project requirement for radiology residents. Acad Radiol 2008; 15:1040–1045. 10. Thrall JH. Quality and safety revolution in health care. Radiology 2004; 233: 3–6. 11. Krajewski K, Siewert B, Yam S, et al. A quality assurance elective for radiology residents. Acad Radiol 2007; 14:239–245. 12. Relyea-Chew A, Talner LB. A dedicated general competencies curriculum for radiology residents development and implementation. Acad Radiol 2011; 18:650–654. 13. Brandon CJ, Mullan PB. Teaching medical management and operations engineering for systems-based practice to radiology residents. Acad Radiol 2013; 20:345–350. 14. Deitch CH, Chan WC, Sunshine JH, et al. Quality assessment and improvement: what radiologists do and think. AJR Am J Roentgenol 1994; 163:1245–1254. 15. Boonyasai RT, Windish DM, Chakraborti C, et al. Effectiveness of teaching quality improvement to clinicians: a systematic review. JAMA 2007; 298: 1023–1037. 16. Patow CA, Karpovich K, Riesenberg LA, et al. Residents’ engagement in quality improvement: a systematic review of the literature. Acad Med 2009; 84:1757–1764. 17. Wong BM, Etchells EE, Kuper A, et al. Teaching quality improvement and patient safety to trainees: a systematic review. Acad Med 2010; 85: 1425–1439. 18. Gaught AM, Cleveland CA, Hill JJ, 3rd. Publish or perish? Physician research productivity during residency training. Am J Phys Med Rehabil 2013; 92:710–714. 19. Eliastam M, Mizrahi T. Quality improvement, housestaff, and the role of chief residents. Acad Med 1996; 71:670–674.
Academic Radiology, Vol -, No -, - 2014
20. Francis MD, Varney AJ. Learning by doing: use of resident-led quality improvement projects to teach clinical practice improvement. Hosp Physician 2006; 42:52–57. 21. Weingart SN, Tess A, Driver J, et al. Creating a quality improvement elective for medical house officers. J Gen Intern Med 2004; 19:861–867.
QUALITY IMPROVEMENT DURING RESIDENCY
22. Weingart SN. A house officer-sponsored quality improvement initiative: Leadership lessons and liabilities. Jt Comm J Qual Improv 1998; 24: 371–378. 23. Headrick LA, Richardson A, Priebe GP. Continuous improvement learning for residents. Pediatrics 1998; 101:768–773.
7
CHOUDHERY ET AL
APPENDIX 1. Please indicate your PGY [postgraduate] level: A. PGY- 2 B. PGY- 3 C. PGY- 4 D. PGY -5 2. Please answer yes or no to the following questions: Completing a quality improvement (QI) project is an Accreditation Council for Graduate Medical Education requirement. Are you aware of this requirement? A. Yes B. No Do you have a process for organizing a QI project at your institution? A. Yes B. No Do you have dedicated time to work on your QI project in your residency? A. Yes B. No 3. What is the status of your QI project? A. Have not started B. In progress C. Completed 4. Which year did you begin your QI project? A. PGY-2 B. PGY-3 C. PGY-4 D. PGY-5
8
Academic Radiology, Vol -, No -, - 2014
5. Which of the following ABR quality improvement project categories best describes your project? A. Patient safety B. Accuracy of interpretation C. Report timeliness D. Practice guidelines and technical standards E. Referring physician surveys 6. Did you have a faculty mentor for your project? A. Yes B. No 7. Was your QI project presented at a meeting or published in an academic journal? A. Yes; please indicate where it was presented or published B. Plan to present or publish in the future C. No 8. Please rate the difficulty level of each of the following QI steps during your QI initiative: A. Coming up with an idea: very easy, easy, somewhat difficult, difficult, very difficult B. Finding a mentor: very easy, easy, somewhat difficult, difficult, very difficult C. Project design and implementation: very easy, easy, somewhat difficult, difficult, very difficult D. Lack of resources (funds, books, etc.): very easy, easy, somewhat difficult, difficult, very difficult E. Lack of time given your clinical and research workload: very easy, easy, somewhat difficult, difficult, very difficult
T
he world of health care is rapidly changing, and a significant component of that change is the emphasis on accountability to the public for the quality of medical practice. Quality improvement (QI) in medicine has become vital to the daily practice of physicians and the training of future physicians. QI in radiology encompasses improving the effectiveness of diagnostic and therapeutic procedures, selecting the appropriate imaging services, ensuring safety and quality of services delivered, and overseeing the efficiency and management of all imaging services (1). To highlight the importance of QI, the American Board of Radiology made Practice Quality Improvement (PQI) one of the four core components for Maintenance of Certification (MOC). At least 23 other medical specialty boards besides the American Board of Radiology (ABR) have a requirement for some form of practice improvement under the MOC plan
Acad Radiol 2014; -:1–8 From the Department of Radiology, UT Southwestern Medical Center, 5323 Harry Hines Blvd. Dallas, TX 75390 (S.C., M.R., A.A., Y.X., T.B., D.C., M.C.M.). Received October 8, 2013; accepted November 24, 2013. Address correspondence to: S.C. e-mail: [email protected] ªAUR, 2014 http://dx.doi.org/10.1016/j.acra.2013.11.021
developed by the American Board of Medical Specialties in 2000 (2). When MOC was officially introduced in 2007 by the ABR, radiologists were only required to perform one PQI project every 10 years (3). The requirements were then adjusted to three PQI projects every 10 years (4). Recently, with the introduction of ‘‘continuous certification,’’ the expected change in requirement to one project every 3 years occurred (3). All the current residents in radiology programs nationally will be certified under this new ABR requirement, and preparing residents for the future has become a vital component of training across the country. Additionally, one of the six core competencies of the Accreditation Council for Graduate Medical Education (ACGME) during residency is systems-based practice (SBP) (5). Systems-based practice is defined by the ACGME as ‘‘awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value’’ (5). Since 2008, the ACGME has required participation of radiology residents in at least one systems-based practice project with documentation of participation in such an activity (6). The ABR has formally incorporated this requirement as one of the 12 required milestones to be achieved by radiology residents during their training (7). SBP is a challenging 1
Academic Radiology, Vol -, No -, - 2014
CHOUDHERY ET AL
TABLE 1. American Board of Radiology Practice Quality Improvement Categories of Resident-Led Projects by PGY
PGY-2 PGY-3 PGY-4 PGY-5
Patient Safety
Accuracy of Interpretation
Report Timeliness
Practice Guidelines and Technical Standards
Referring Physician Surveys
25* 24.53 20 21.43
37.5 16.98 16.67 21.43
0 13.21 0 28.57
37.5 39.62 56.67 21.43
0 5.66 6.67 7.14
PGY, postgraduate year. *% of survey respondents in the respective PGY.
competency to define, teach, and evaluate (8) and was reported to be the most difficult competency to implement in radiology residency programs in the 2005 annual survey of the Association of Program Directors in Radiology (9). A systems-based practice project performed during residency is essentially equivalent to a PQI project performed after board certification. This study examined radiology residents’ perspectives on QI and explored the obstacles faced by them during the process of performing QI or systems-based practice projects. The aim of this study was to determine the awareness of residents of the PQI requirement, hindrances faced by residents in the process of project development and completion, and the percentage of projects published in the scientific literature or presented at meetings. A secondary goal was to provide an insight into steps that may be taken within academic radiology departments to facilitate the process of completing PQI projects in a way that would make PQI a valuable learning experience for residents.
MATERIALS AND METHODS An eight-question electronic survey using SurveyMonkey (www.surveymonkey.com) was sent out to 51 residents at our institution and 2039 registered trainee members of the Association of University Radiologists with responses collected for approximately 8 weeks (see Appendix). This survey was distributed in May 2013, near the end of the academic year, to avoid bias related to having initiated a new residency program or year. The survey initially queried the participants about their postgraduate year (PGY) level and their awareness of the requirement of doing a SBP or QI project during residency. Next, a series of questions was asked to learn about the PQI policy at the residents’ institutions including availability of dedicated time and a formal process to assist in development and completion of QI projects. The subsequent sequence of questions was targeted toward gauging the time frame of initiation and completion of QI projects during residency. The participants were additionally asked to state which of the five ABR PQI categories (patient safety, accuracy of interpretation, report timeliness, practice guidelines and technical standards, and referring physician surveys) best described their QI projects. They were also questioned regarding faculty mentorship and publication and presentation of their projects. 2
Last, they were asked to rate the hardship they faced on a scale of 5 ranging from very easy to very difficult in completing these five PQI steps: coming up with an idea, finding a mentor, designing and implementing the project, finding resources such as books and funds, and finding time. Participants were also provided an opportunity to leave comments about the QI process at the end. The data were collected anonymously and analyzed using descriptive statistics and chi-square and Fisher exact tests with statistical significance defined as P < .05. This survey was institutional review board exempt.
RESULTS Overview of Participants, Awareness of QI Projects, and Topics Covered
A total of 154 residents nationally responded to the survey, of which approximately 11% were PGY-2, 54% were PGY-3, 25% were PGY-4, and 10% were PGY-5 residents. Ninetyone percent of the residents were aware of the requirement to complete a systems-based practice project or a QI project during residency, whereas 9% were not aware of this requirement. All PGY-5 respondents knew about the PQI requirement for graduation, but 12% of PGY-2, 11% of PGY-3, and 8% of PGY-4 participants were not aware of this requirement. The ABR classifies QI projects into five categories: patient safety, accuracy of interpretation, report timeliness, practice guidelines, and technical standards. The category of practice guidelines and technical standards accounted for the highest number of projects at 42%. Twenty-three percent of projects fit the category of patient safety, whereas 19% were related to accuracy of interpretation. The categories of report timeliness and referring physician surveys defined the remaining 16% of the projects. The distribution of these five QI project categories by PGY is illustrated in Table 1. Timeframe of QI Projects
The majority of the survey participants had begun their PQI project in their first 2 years of residency, with 52% initiating them during PGY-2 and 40% during PGY-3 (Fig 1). Of the remaining 8% that had started their projects during their last
Academic Radiology, Vol -, No -, - 2014
QUALITY IMPROVEMENT DURING RESIDENCY
Figure 1. Postgraduate year (PGY) of survey respondents at initiation of a quality improvement project by percentage of residents.
Figure 2. Status of quality improvement projects among the different postgraduate year (PGY) residents by percentage of residents.
2 years of residency, 1% of them were PGY-5. Forty-three percent of survey respondents had completed their QI projects, whereas 27% had currently active projects and 30% had not initiated a project. For PGY-2 residents, 0% had completed a project, 41% had an active project, and 59% had yet to begin a project (Fig 2). For PGY-3 residents, 37% had completed a project, 26% had an active project, and 37% had yet to begin a project. For PGY-4 residents, 58% had completed a project, 26% had an active project, and 16% had yet to begin a project. For PGY-5 residents, 87% had completed a project, 13% had an active project, and 0% had yet to begin a project. Institutional QI Policy, Faculty Mentorship, and Publication of QI Projects
A total of 71% of residents stated that they had a process for organizing a QI project at their institution, whereas 29% indicated that they did not (Table 2). Having a QI process at one’s institution was significantly associated with having initiated or completed a QI project at the time of this survey administration (P = .0007). Additionally, 81% of the survey participants performed QI projects under faculty mentorship, whereas 19% of the respondents did not have faculty mentors
TABLE 2. Availability of Dedicated QI Process and Time, Faculty Mentorship, and Publication/Presentation Rates of QI Projects among Surveyed Residents
Availability of a QI process Availability of time for QI Faculty mentorship Publication/presentation of a QI project or plans to do so in the future
Yes
No
71* 26 81 39
29 74 19 61
QI, quality improvement. *% of total survey respondents.
(Table 2). Only 26% of respondents had dedicated time during residency to work on a QI project and 74% did not (Table 2). Forty-seven percent of PGY-5 respondents indicated that they had dedicated time for performing QI, whereas only 24% of PGY-2, 29% of PGY-3, and 13% of PGY-4 participants stated the same. About 19% of the trainees had presented or published their project results and 20% had plans to do so (Table 2). The remaining residents indicated that they did not have plans to publish or present their projects at a meeting. There was a 3
Academic Radiology, Vol -, No -, - 2014
CHOUDHERY ET AL
Figure 3. Publication and presentation rates of quality improvement projects among residents who have dedicated time versus those who do not by percentage of residents.
TABLE 3. Resident Rating of Each of the Five Surveyed Quality Improvement Steps by PGY
PGY-2 Very easy Easy Somewhat difficult Difficult Very difficult PGY-3 Very easy Easy Somewhat difficult Difficult Very difficult PGY-4 Very easy Easy Somewhat difficult Difficult Very difficult PGY-5 Very easy Easy Somewhat difficult Difficult Very difficult
Coming up with an Idea
Finding a Mentor
Designing and Implementing a Project
Finding Resources
Finding Time
0* 20 70 10 0
10 70 20 0 0
0 30 70 0 0
0 50 50 0 0
0 10 70 10 10
19 46 25 5 5
32 54 5 5 4
7 44 33 9 7
7 49 26 9 9
3 19 32 23 23
21 38 29 12 0
16 36 36 6 6
6 26 47 21 0
6 41 25 22 6
0 15 37 21 27
13 60 20 7 0
20 60 7 13 0
7 73 13 7 0
7 53 26 7 7
13 34 20 20 13
PGY, postgraduate year. *% of survey respondents in respective post-graduate year (PGY).
significant relationship between having dedicated time for a QI project and the likelihood of presentation or publication (P = .0091) (Fig 3). Hindrances Encountered during QI Initiatives
The survey respondents were asked about the obstacles they faced in the process of performing QI projects. Table 3 illustrates the surveyed difficulty ratings given by each PGY for the five PQI steps: coming up with an idea, finding a mentor, designing and implementing a project, finding resources, and finding time. Finding a mentor was considered overall the least challenging step in performing a QI project, receiving ‘‘very easy’’ or ‘‘easy’’ rating by 75% of the partici4
pants. Lack of time was thought to be the greatest obstacle and was rated either ‘‘somewhat difficult,’’ ‘‘difficult,’’ or ‘‘very difficult’’ by 78% of participants. We further looked at relationships between these hindrances and the status of the QI project at the time of the survey. For this, we categorized ‘‘easy’’ as including both ‘‘very easy’’ and ‘‘easy’’ and ‘‘difficult’’ as including ‘‘somewhat difficult,’’ ‘‘difficult,’’ and ‘‘very difficult.’’ The status of the QI project was defined as having been initiated (either completed or in progress) or not initiated. There was a significant relationship between the rating given, either easy or difficult, to each of the five QI steps (coming up with an idea, finding a mentor, designing a project, finding resources, and finding time given clinical and research workload) and the status of
Academic Radiology, Vol -, No -, - 2014
the QI project (P < .0001 for the first four listed hindrances and P = .0046 for the hindrance of finding time). Residents who thought these PQI steps were easy were more likely to have initiated a QI project at the time of this survey administration. Additionally, residents who thought it was difficult to find a mentor in comparison to those who thought it was easy were more likely to start their project during their last 2 years of residency versus the first 2 years of residency (P < .0001). DISCUSSION With increasing demands by the public for physicians to be held accountable for errors in health care delivery (10), demand for QI in medicine will only continue to increase. As a result of these demands for accountability in the quality of medicine, the ABR and at least 23 other medical specialty boards have made QI projects in some form a requirement for recertification (2). The concept of requiring physicians to play an active role in improving the quality of care they provide has also influenced the education of trainee physicians. SBP, considered equivalent to PQI, is one of the ACGME’s required six core competencies for resident education. From developing SBP curriculum and electives (11–13) to establishing dedicated departmental PQI systems (3), radiology departments have incorporated SBP and PQI into their resident education in a variety of ways. Although studies have examined the views of practicing radiologists regarding QI (14), few data exist about radiology residents’ perspectives and practice of QI. Our study provides insight into the radiology trainees’ views of performing QI projects and highlights important areas for improvement in QI education. In our study, most of the residents were aware of the requirement of a PQI or a SBP project. About 9% of the residents did not know about this requisite, and a vast majority of these residents, not surprisingly, were in their first 2 years of residency. However, up to 8% of the PGY-4 respondents were not cognizant of the requirement to perform a QI project. This translated to about 16% of PGY-4 residents not having started a project near the beginning of their last year of residency and about 13% of PGY-5 residents not having completed a project near the completion of their residency. With the institution of the ABR Milestone Project in July 2013, initiation of a PQI project is required for level 3 of the SBP Quality Improvement Milestone and completion of a project is required for attaining level 4 (7). Thus, it is the expectation of the ABR that PGY-4 residents are identifying and working on QI projects that should then be completed no later than the PGY-5 year. There is an important need for increasing awareness of the requirement of PQI and for encouraging earlier initiation and fulfillment of this requisite for progress along the milestones pathway. Given also the institution of the ABR Core Exam, which will be taken at the end of the PGY-4 year, it will be advantageous for residents to begin these projects earlier in their residency so that they do not conflict with time spent studying for the
QUALITY IMPROVEMENT DURING RESIDENCY
board examination. Launching a project earlier would allow residents more time to design and complete a meaningful project suitable for publication or presentation. The question remains regarding how to teach and increase awareness of PQI. In their systematic review, Boonyasai et al. demonstrated that dedicated QI curricula can increase physicians’ knowledge or confidence to perform QI (15). Several radiology departments have experimented with various QI teaching methodologies. One department developed a program that introduced PGY-2 residents to QI and SBP through lectures (9). The PGY-2 residents were then required to submit a written proposal and present it to their peers 6 months after the beginning of their residency. They were given 12 to 18 months to work on their projects with a mandatory presentation of results at the end of their PGY3 year. This approach resulted in timely completion of QI projects and increased scholarly activities. This is just one example of how hands-on QI teaching can be performed and reinforced in residency. Our results indicate that a majority of residents are at institutions with some form of a departmental QI process in place. On the other hand, only about 26% of residents have any dedicated time to work on a QI project. This indicates a discrepancy in the number of institutions that have a formally organized QI process and that allow protected QI time for residents. Residents with an institutional QI process are more likely to have initiated a QI project. Although only 39% of respondents had or intended to publish or present their work, residents with dedicated time for a QI project were more likely to do so. It can therefore be inferred that although residents are more likely to initiate their QI efforts at institutions with a QI process, they are less likely to publish or present their work if they do not have dedicated time for their projects. Our study is consistent with the results of systematic reviews in which lack of time (16) and competing educational demands (17) have been shown to be major barriers to resident participation in QI initiatives. Studies have also shown that residents have a low publication rate in general during residency and that lack of time, funding, and research experience contribute to this low research productivity (18). It can, therefore, be concluded that to make PQI a truly worthwhile learning experience for residents, it is not enough to just have a QI system. It is also vital to allow residents dedicated time for PQI, which would increase research productivity, and to mentor them in their publication efforts. A senior research elective is one possibility in which a resident may have dedicated time to complete a QI project and prepare it for publication or presentation at a scientific meeting. The questions regarding obstacles faced during QI projects revealed that residents who thought it was easy to come up with an idea, find a mentor, design and implement a project, find resources such as books and funds, and find time for project completion were more likely to have initiated a QI initiative in comparison to those who thought it was difficult to do so. Although finding a mentor was rated difficult less frequently than any of the other QI steps, up to 19% of 5
CHOUDHERY ET AL
resident respondents did not have faculty mentors and those who thought it was difficult to find a mentor were more likely to start a QI project later in residency. The difficulty level of these five surveyed PQI steps presents an opportunity for radiology departments to improve their QI process. To introduce QI project ideas and to assist in design and implementation, an organized elective or curriculum may be used. Additionally, everyday clinical practice may be used to help residents discover prospective QI initiatives (8). To facilitate the process of finding a mentor, departments may operate a central database of proposed and ongoing projects by residents and faculty members. Furthermore, radiology departments can make teaching QI more valuable by allowing dedicated time to residents for performing PQI initiatives, offering helpful books, and providing or suggesting sources for funding. Additional obstacles to consider that have been identified in prior studies include residents’ lack of knowledge of data collection and analysis techniques (19,20), absence of previous QI training (21), hierarchical culture of medicine (22), and faculty skepticism about resident-led QI projects (23). Solutions to these problems might include collaboration with successful mentors, assistance by department statisticians, and coaching by the residency program director. Because most faculty members themselves are now required to complete PQI projects, it is reasonable to coach faculty in the value of collaborating with residents to accomplish PQI requirements for the faculty and trainee. Several limitations of our study must be considered. Our sample size was relatively small, and this was a survey study, introducing response bias. Lack of reminders and incentives to take the survey may have contributed to the small sample size. We also had significant representation from PGY-2 and PGY-3 residents, with only 10% representation from PGY-5 residents, who were likely occupied with studying for oral boards at the time of this survey administration. With a higher number of younger residents in our study who are expected to have more difficulty in the QI process, we might have overestimated the value of hindrances faced by residents during QI initiatives. Additionally, some of our questions used abstract terminology, such as ‘‘project design and implementation,’’ which added subjectivity to our data. An additional limitation of our survey design was our use of two categories for defining easy—‘‘very easy’’ and ‘‘easy’’—and three categories for designing difficult—‘‘somewhat difficult,’’ ‘‘difficult,’’ and ‘‘very difficult’’—for rating of a QI step. Retrospective adjustment of these categories is challenging because one does not know if participants who picked ‘‘somewhat difficult’’ were aiming for the middle of the scale or truly thought there was some difficulty involved with the respective QI step and would have picked ‘‘difficult’’ or ‘‘very difficult’’ if the category of ‘‘somewhat difficult’’ did not exist. The radiology residents of today will be expected to participate in QI initiatives throughout the rest of their careers. The ACGME requirement of completing a systems-based practice project as well as the ABR Milestone requirement can be used to prepare trainee physicians for their future QI endeavors. 6
Academic Radiology, Vol -, No -, - 2014
Training in QI is currently an underrepresented part of resident education. QI needs to be taught so that our future radiologists are not only prepared to fulfill their ABR QI requirements, but also are trained to be transformers of the quality of health care they provide to their patients. Increasing awareness of PQI and providing residents with ideas for projects, encouraging faculty mentorship and publication of QI projects, educating residents about project design and implementation, providing resources such as books and funds, and allowing dedicated time for QI initiatives are some ways we can make QI a valuable educational experience during residency.
REFERENCES 1. Kruskal JB, Eisenberg R, Sosna J, et al. Quality initiatives: quality improvement in radiology: basic principles and tools required to achieve success. Radiographics 2011; 31:1499–1509. 2. ABMS Maintenance of Certification. Available at: http://www.abms.org/ maintenance_of_certification/ABMS_MOC.aspx. Accessed October 1, 2013. 3. Kouo T. Experience with a practice quality improvement system in a university radiology department. J Am Coll Radiol 2012; 9:814–819. 4. Maintenance of Certification Part IV: ABR Guide to Practice Quality Improvement 2012. Available at: http://www.theabr.org/sites/all/themes/ abr-media/pdf/PQI_2012.pdf. Accessed October 1, 2013. 5. Dyne PL, Strauss RW, Rinnert S. Systems-based practice: the sixth core competency. Acad Emerg Med 2002; 9:1270–1277. 6. Amis ES, Jr. New program requirements for diagnostic radiology: update and discussion of the more complex requirements. AJR Am J Roentgenol 2008; 190:2–4. 7. The Diagnostic Radiology Milestone Project. Available at: http://www. acgme-nas.org/assets/pdf/Milestones/DiagnosticRadiologyMilestones.pdf. Accessed October 7, 2013. 8. Johnson JK, Miller SH, Horowitz SD. Systems-based practice: improving the safety and quality of patient care by recognizing and improving the systems in which we work. In: Henriksen K, Battles JB, Keyes MA, et al., eds. Advances in patient safety: new directions and alternative approaches. Rockville, MD: Agency for Healthcare Research and Quality. Available at: http://www.ncbi.nlm.nih.gov/books/NBK43731/; 2008. 9. Buchmann RF, Deloney LA, Donepudi SK, et al. Development and implementation of a systems-based practice project requirement for radiology residents. Acad Radiol 2008; 15:1040–1045. 10. Thrall JH. Quality and safety revolution in health care. Radiology 2004; 233: 3–6. 11. Krajewski K, Siewert B, Yam S, et al. A quality assurance elective for radiology residents. Acad Radiol 2007; 14:239–245. 12. Relyea-Chew A, Talner LB. A dedicated general competencies curriculum for radiology residents development and implementation. Acad Radiol 2011; 18:650–654. 13. Brandon CJ, Mullan PB. Teaching medical management and operations engineering for systems-based practice to radiology residents. Acad Radiol 2013; 20:345–350. 14. Deitch CH, Chan WC, Sunshine JH, et al. Quality assessment and improvement: what radiologists do and think. AJR Am J Roentgenol 1994; 163:1245–1254. 15. Boonyasai RT, Windish DM, Chakraborti C, et al. Effectiveness of teaching quality improvement to clinicians: a systematic review. JAMA 2007; 298: 1023–1037. 16. Patow CA, Karpovich K, Riesenberg LA, et al. Residents’ engagement in quality improvement: a systematic review of the literature. Acad Med 2009; 84:1757–1764. 17. Wong BM, Etchells EE, Kuper A, et al. Teaching quality improvement and patient safety to trainees: a systematic review. Acad Med 2010; 85: 1425–1439. 18. Gaught AM, Cleveland CA, Hill JJ, 3rd. Publish or perish? Physician research productivity during residency training. Am J Phys Med Rehabil 2013; 92:710–714. 19. Eliastam M, Mizrahi T. Quality improvement, housestaff, and the role of chief residents. Acad Med 1996; 71:670–674.
Academic Radiology, Vol -, No -, - 2014
20. Francis MD, Varney AJ. Learning by doing: use of resident-led quality improvement projects to teach clinical practice improvement. Hosp Physician 2006; 42:52–57. 21. Weingart SN, Tess A, Driver J, et al. Creating a quality improvement elective for medical house officers. J Gen Intern Med 2004; 19:861–867.
QUALITY IMPROVEMENT DURING RESIDENCY
22. Weingart SN. A house officer-sponsored quality improvement initiative: Leadership lessons and liabilities. Jt Comm J Qual Improv 1998; 24: 371–378. 23. Headrick LA, Richardson A, Priebe GP. Continuous improvement learning for residents. Pediatrics 1998; 101:768–773.
7
CHOUDHERY ET AL
APPENDIX 1. Please indicate your PGY [postgraduate] level: A. PGY- 2 B. PGY- 3 C. PGY- 4 D. PGY -5 2. Please answer yes or no to the following questions: Completing a quality improvement (QI) project is an Accreditation Council for Graduate Medical Education requirement. Are you aware of this requirement? A. Yes B. No Do you have a process for organizing a QI project at your institution? A. Yes B. No Do you have dedicated time to work on your QI project in your residency? A. Yes B. No 3. What is the status of your QI project? A. Have not started B. In progress C. Completed 4. Which year did you begin your QI project? A. PGY-2 B. PGY-3 C. PGY-4 D. PGY-5
8
Academic Radiology, Vol -, No -, - 2014
5. Which of the following ABR quality improvement project categories best describes your project? A. Patient safety B. Accuracy of interpretation C. Report timeliness D. Practice guidelines and technical standards E. Referring physician surveys 6. Did you have a faculty mentor for your project? A. Yes B. No 7. Was your QI project presented at a meeting or published in an academic journal? A. Yes; please indicate where it was presented or published B. Plan to present or publish in the future C. No 8. Please rate the difficulty level of each of the following QI steps during your QI initiative: A. Coming up with an idea: very easy, easy, somewhat difficult, difficult, very difficult B. Finding a mentor: very easy, easy, somewhat difficult, difficult, very difficult C. Project design and implementation: very easy, easy, somewhat difficult, difficult, very difficult D. Lack of resources (funds, books, etc.): very easy, easy, somewhat difficult, difficult, very difficult E. Lack of time given your clinical and research workload: very easy, easy, somewhat difficult, difficult, very difficult
