Special Topic The New Accreditation Council for Graduate Medical Education Next Accreditation System Milestones Evaluation System: What Is Expected and How Are Plastic Surgery Residency Programs Preparing? Nyama M. Sillah, M.D. Ahmed M. S. Ibrahim, M.D., Ph.D. Frank H. Lau, M.D. Jinesh Shah, B.S. Caroline Medin, B.S. Bernard T. Lee, M.D., M.B.A. Samuel J. Lin, M.D. Boston, Mass.; and New Orleans, La.
Background: The Accreditation Council for Graduate Medical Education Next Accreditation System milestones were implemented for plastic surgery programs in July of 2014. Forward progress through the milestones is an indicator of trainee-appropriate development, whereas regression or stalling may indicate the need for concentrated, targeted training. Methods: Online software at www.surveymonkey.com was used to create a survey about the program’s approaches to milestones and was distributed to program directors and administrators of 96 Accreditation Council for Graduate Medical Education–approved plastic surgery programs. Results: The authors had a 63.5 percent response rate (61 of 96 plastic surgery programs). Most programs report some level of readiness, only 22 percent feel completely prepared for the Next Accreditation System milestones, and only 23 percent are completely satisfied with their planned approach for compliance. Seventy-five percent of programs claim to be using some form of electronic tracking system. Programs plan to use multiple tools to capture and report milestone data. Most programs (44.4 percent) plan to administer evaluations at the end of each rotation. Over 70 percent of respondents believe that the milestones approach would improve the quality of resident training. However, programs were less than confident that their current compliance systems would live up to their full potential. Conclusions: The Next Accreditation System has been implemented nationwide for plastic surgery training programs. Milestone-based resident training is a new paradigm for residency training evaluation; programs are in the process of making this transition to find ways to make milestone data meaningful for faculty and residents. (Plast. Reconstr. Surg. 136: 181, 2015.)
T
he Accreditation Council for Graduate Medical Education was established in 1981 and has sought to accentuate program structure, to strengthen the amount and quality of resident education by striking a balance between service and education, and to provide constructive feedback to trainees.1 Over the past three decades, From the Division of Plastic Surgery, Beth Israel Deaconess Medical Center, Harvard Medical School; and the Division of Plastic Surgery, Louisiana State University. Received for publication October 4, 2014; accepted January 27, 2015. Copyright © 2015 by the American Society of Plastic Surgeons DOI: 10.1097/PRS.0000000000001368
Disclosure: The authors have no financial interest to declare in relation to the content of this article. There was no internal or external financial support for this study.
Supplemental digital content is available for this article. Direct URL citations appear in the text; simply type the URL address into any Web browser to access this content. Clickable links to the material are provided in the HTML text of this article on the Journal’s Web site (www. PRSJournal.com).
www.PRSJournal.com
181
Copyright © 2015 American Society of Plastic Surgeons. Unauthorized reproduction of this article is prohibited.
Plastic and Reconstructive Surgery • July 2015 the Accreditation Council for Graduate Medical Education has worked and evolved to achieve the aforementioned goals. In 1999, the Accreditation Council for Graduate Medical Education Outcome Project introduced six “core competencies” (i.e., patient care, medical knowledge, professionalism, systems-based practice, practiced-based learning and improvement, and interpersonal and communications skills) that were to be incorporated into resident training and education to emphasize a more outcomes-based education.2 However, the core competencies were limited in their ability to use outcomes data in accreditation and public accountability because of various factors such as lack of clear conceptual meaning of the competencies, failure to develop a uniform assessment approach, and lack of adequate training of the individuals expected to evaluate residents on the competencies, namely, faculty.3 In addition, compliance was assessed only every 4 to 5 years.1 Therefore, in 2009, the Accreditation Council for Graduate Medical Education began the process of restructuring the accreditation system by instituting several changes, including a clinical competency committee, institution-wide clinical learning environment review visit, and a new evaluation process of trainees. This evaluation process is known as the “milestones” framework, which has five levels to better evaluate residents. All these changes put together are known as the Next Accreditation System.1,3 The purpose of the new system is to better prepare training physicians for future practice through the peer-review system, base accreditation on educational outcomes, and decrease the cumbersome nature related to the current process-based approach. The milestones are one of nine program performance indicators in the Next Accreditation System.4 Milestones for trainees are collected on a semiannual basis.4 Instead of compliance being assessed every 4 to 5 years, milestone data will be used in annual accreditation review, therefore allowing the period between formal accreditation visits to be increased to every 10 years.1,4 Milestones are competency-based developmental outcomes of core specialty-specific knowledge, skills, attitudes, and beliefs that residents should be able to progress through from the beginning of training through graduation, with the overall goal of producing safe, competent, ethical physicians.3–5 The milestones were developed in each specialty through workgroups that consisted of American Board of Medical Specialties certifying boards, review committees, medical specialty organizations, program-director
associations, and residents.1 Subcompetencies were identified within the six core competencies with the goal of developing approximately 36 (or other number deemed appropriate by the workgroups) subcompetencies for each American Board of Medical Specialties specialty.3 A five-level model for the milestones was developed based on concepts adapted from the Dreyfus model of expertise. The Dreyfus model of expertise was initially developed after evaluation of jet pilots in the Air Force and was a model of professional expertise in which individuals’ progress through various stages: novice, advanced beginner, competent, proficient, and expert.3,6 Since its development, this model has been expanded from the acquisition of skills by jet pilots and modified to apply to other disciplines. The Accreditation Council for Graduate Medical Education adaptation of the model includes milestones that residents entering a specialty who have recently graduated from medical school should be able to attain (level 1), progressively advancing to goal competencies for graduation (level 4), to highly ambitious goals for extraordinary residents or physicians already in practice (level 5).3 In addition to the milestones, other information to be collected for annual surveillance will be Accreditation Council for Graduate Medical Education surgery forms for faculty and residents, and operative and case logs.1 Early on, the milestone data will be used to evaluate the quality of resident and fellow training based on outcomes.4,5 The first phase of the Next Accreditation System was initiated in July of 2013 in seven specialties: emergency medicine, internal medicine, neurologic surgery, orthopedic surgery, pediatrics, diagnostic radiology, and urology. Initial data for the validity and utility of the milestones in these initial specialties is starting to emerge.3 In July of 2014, plastic surgery programs began implementation of the Next Accreditation System for evaluation of resident and fellows by use of the milestone criteria. 5,7 (See Figure, Supplemental Digital Content 1, which shows the Accreditation Council for Graduate Medical Education plastic surgery milestones, http://links.lww.com/PRS/B337.) This paradigm shift represents a substantial change in evaluation of resident and fellows from what most plastic surgery programs are accustomed to. Although the milestones have been used in other specialties for a little over 1 year, they are very new for plastic surgery programs; therefore, we sought to evaluate how the Next Accreditation System is being implemented across plastic surgery programs nationwide.
182 Copyright © 2015 American Society of Plastic Surgeons. Unauthorized reproduction of this article is prohibited.
Volume 136, Number 1 • Plastic Surgery Accreditation Milestones METHODS Recruitment Institutional review board exemption was obtained. The online software available at www. surveymonkey.com was used to create the survey. (See Figure, Supplemental Digital Content 2, which shows the Accreditation Council for Graduate Medical Education plastic surgery milestones survey, http://links.lww.com/PRS/B338.) An e-mail invitation containing a generic link to the survey, which also ensured anonymity and prevented tracking, was distributed to 96 Accreditation Council for Graduate Medical Education– approved plastic surgery programs (integrated and independent programs). The e-mail was sent to both the program directors and their administrators. Following collection of all responses, the data were processed. No incentives were given for participation in this study. There was no preselection based on anticipated responses. Survey The Web-based survey gathered basic information on program demographics, including how a program would characterize itself and the number of residents enrolled per year. Inquiries were made about the prevalence of electronic tracking systems to record milestones, how programs capture and report data, and what strategies were used in the individual program’s approach to milestones. There were two types of approaches that were assessed: the bottom-up approach, where each milestone was broken into multiple component parts (e.g., evaluate specific skills such as wound débridement within the “patient care for tissue transfer” milestone); and the top-down approach, which used Accreditation Council for Graduate Medical Education milestones as primary evaluation points (e.g., rate patient care on a scale of 1 to 5, or other strategies), and the anticipated ongoing expenditures to comply with the new Accreditation Council for Graduate Medical Education milestones requirements. Furthermore, questions were asked about planning and rollout for Accreditation Council for Graduate Medical Education milestone compliance, hours per month spent on milestone compliance, readiness of programs for milestone compliance, frequency of evaluations, and involvement of other hospital staff for data acquisition. Finally, participating programs were asked about their overall perception of the Accreditation Council for Graduate Medical Education milestones approach. To ensure validation of the survey, it was presented to a focus
group of students at the Massachusetts Institute of Technology Sloan School of Management. This was done because only 96 Accreditation Council for Graduate Medical Education–accredited plastic surgery programs currently exist; as such, forming a smaller subset of that population as a test group for the survey was not feasible because the data provided from this subset would be pertinent to the findings of the study. Although the background of the focus group differs from that of the program directors, this approach has been previously described in the validation process of surveys.8–10 The survey was revised until the final draft was complete.
RESULTS Plastic Surgery Program Demographics The data for this study were ascertained through 61 plastic surgery program responses (response rate, 63.5 percent). A majority (72 percent) of survey respondents were affiliated with a university- and/or academic-based program. Most programs (44.3 percent) had one or two residents enrolled per year, which correlates with national data,11 and approximately 30 percent reported having five or more residents enrolled on an annual basis (most likely, those programs with both integrated and independent residents). Plastic Surgery Programs and Accreditation Council for Graduate Medical Education Milestones Of the 61 plastic surgery programs participating in this survey, three-quarters were noted to be using some form of an electronic tracking system; similarly, three-quarters of those tracking milestones electronically plan to use a third-party system. Programs plan to use multiple tools to capture and report milestone data: 66.8 percent expect to use Web-based surveys, 46.3 percent anticipate that they will use some form of paper-based survey, and 63 percent expect to conduct interviews. In terms of compliance with milestones, 64.1 percent of participating programs plan to use a top-down approach, 22.6 percent plan to use a bottom-up approach, and the remainder had reported either not yet finalizing their approach or using a combination of both. Most programs do not anticipate upfront expenditures (74 percent) or ongoing expenditures (71 percent) to comply with the new Accreditation Council for Graduate Medical Education milestones requirement. Of those who anticipate
183 Copyright © 2015 American Society of Plastic Surgeons. Unauthorized reproduction of this article is prohibited.
Plastic and Reconstructive Surgery • July 2015 expenditures, there was a wide array of cost estimates ranging from $5000 to $20,000 for both, which were attributed to uncertainty, variability related to time and effort, and nominal costs to plastic surgery departments, in addition to large costs to the institution and significant labor costs related to an increase in staff and technology expenses. The program director and the program administrator are the most involved individuals in the planning and rollout for Accreditation Council for Graduate Medical Education milestones compliance (96 percent and 86 percent, respectively). Less than half of participating programs plan to engage residents, hospital administrators, or the graduate medical education coordinator (Fig. 1). On average, programs expect their teams to commit 34 collective hours per month to milestone compliance. Program directors and faculty are each expected to spend 8 to 9 hours per month, and program administrators are expected to spend approximately 12 hours per month. The vast majority of responding programs report some level of readiness, but only 22 percent of programs feel completely prepared for milestone compliance. Similarly, only 23 percent of program respondents were completely satisfied with their planned approach for compliance (Fig. 2). Most programs plan to administer evaluations at the end of each rotation (44.4 percent), whereas some (26.7 percent) are planning evaluations every 6 months (Fig. 3). Many programs plan to involve nursing staff (70.5 percent) and midlevel providers such as physician assistants and nurse practitioners (59 percent) in their milestones compliance systems. Fifty percent of programs
Fig. 1. Compliance team. PD, program director; PA, program administrator; PF, program faculty; R, residents; HA/GME lead, hospital administrator/graduate medical education coordinator.
will engage physicians from other specialties, whereas 22.7 percent do not plan to involve anyone outside of plastic surgery. Over 70 percent of respondents believe that the milestones approach would improve the quality of the residency program to some extent in terms of both the depth and the quality of a resident’s training. However, programs appeared less than confident that their compliance systems as currently planned would be compliant with these new guidelines (Fig. 4).
DISCUSSION The Accreditation Council for Graduate Medical Education Next Accreditation System is an extension of the Outcomes Project previously developed with the emphasis on data-based outcomes rather than process-based outcomes, with the use of milestones for acquisition of these data.4 Forward progress through the milestones will be an indicator of resident- or fellow-appropriate development, whereas regression or stalling may indicate the need for concentrated, targeted education/training. Overall, the progression of trainees through the milestones will be an indicator of the effectiveness of residency and fellowship training programs of producing high-quality and dependable physicians to take care of the greater public. One stated goal of the Next Accreditation System was to identify high-functioning programs that could then be given the latitude to be innovative regarding resident trainee education; conversely, this system may be used to identify programs not as high functioning, allowing identification of areas of improvement rather than citation.1,4 Milestone data will be collected about each resident by a clinical competency committee composed of staff on a semiannual basis and entered electronically to the Accreditation Council for Graduate Medical Education. The role of the clinical competency committee is to determine a resident’s acquisition and progression or regression through the milestones and make appropriate recommendations of advancement, graduation, or remediation accordingly.5 The data will provide residents with tangible feedback on an individual level, and the collective resident acquisition of milestones will be used annually for accreditation review of overall program performance.4 In our study, we found that most plastic surgery programs are planning to involve the program director, program administrator, and faculty on a clinical competency committee, but less than half plan to engage residents or other hospital administrators. In addition, on average, programs expect their clinical competency
184 Copyright © 2015 American Society of Plastic Surgeons. Unauthorized reproduction of this article is prohibited.
Volume 136, Number 1 • Plastic Surgery Accreditation Milestones
Fig. 2. Readiness assessment.
Fig. 3. Frequency of evaluations.
committee to spend 34 collective hours monthly to be compliant with the milestones. The University of New Mexico General Surgery Residency program was a test site for the milestones from 2012
to 2013. Their clinical competency committee was composed of six faculty “competency champions” appointed by the program director who were each assigned one of the six Accreditation Council for Graduate Medical Education competencies based on expertise for the competency. In addition to the six champions, the clinical competency committee also included the program director, associate program director, chair of the department, and two atlarge faculty members. In addition, the two chief residents were included as nonvoting members. The clinical competency committee met monthly, with rotating competency topics, with the goal of each competency being discussed twice by the end of the year. In addition, an entire class was discussed at each meeting. As the champions became more knowledgeable about the competencies and more efficient, an entire class could be reviewed in only 1 hour. Because of frequent meetings, struggling residents were better identified and placed on remediation programs headed by the champion for the particular competency. After 1 year, this program noted that faculty were better able to provide constructive and tangible feedback; at-risk residents
185 Copyright © 2015 American Society of Plastic Surgeons. Unauthorized reproduction of this article is prohibited.
Plastic and Reconstructive Surgery • July 2015
Fig. 4. Overall perception of Next Accreditation System milestones. ACGME, Accreditation Council for Graduate Medical Education.
were able to be identified and, if needed, put in a remediation program to work on improving a specific milestone; residents had clearer knowledge of what was expected of them; and, overall, rotations were better structured to comply with attainment of the milestones.12 Diverse composition of a clinical competency committee with specifically designated roles and frequent meetings eventually may also provide further viewpoints. These early data demonstrate that diverse composition of a clinical competency committee with specifically designated roles and frequent meetings may be a model that is beneficial to both staff and residents; however, further prospective data collection on the milestones is needed. Implementation of the Next Accreditation System and collection of milestones began nationally for all plastic surgery programs in July of 2014; however, seven specialties initiated the program over 1 year ago and have been using the milestones for resident and fellow outcomes-based assessment. Emergency medicine was one of the first specialties to develop and implement use of the milestones in 2013. During the developmental process
before implementation, the proposed milestones were sent in a survey to all accredited emergency medicine residency program directors and key faculty for validation.13 The internal medicine community developed milestones through numerous workshops. To comply with the Next Accreditation System, the internal medicine residency program at the University of Kansas restructured their curricula to better align with the milestones and created templates for each rotation to collect the outcome data. Once developed, the internal medicine department worked with the graduate medical education office and technical support team to make the templates electronic to be more available to faculty and residents. Currently, the developers of this program are performing objective data collection of their system but believe sharing of data nationally on collaborative portals of how programs are implementing the new milestone framework will be vital.14 Survey of the internal medicine community with regard to the utility of the milestones demonstrated that 85 percent of 253 faculty, residents, and others thought the milestones would enhance
186 Copyright © 2015 American Society of Plastic Surgeons. Unauthorized reproduction of this article is prohibited.
Volume 136, Number 1 • Plastic Surgery Accreditation Milestones resident education, feedback to trainees, and faculty development.3 The orthopedic surgery milestones underwent pilot testing before implementation, and 80 percent of the 75 participants thought milestones allowed for meaningful resident evaluation.3 In the current study, over 70 percent of respondents feel the milestones approach will improve the quality of plastic surgery training to some extent. Overall, from our study, it appears that most programs report some level of readiness, but only 22 percent of programs feel completely prepared for milestone compliance and only 23 percent were completely satisfied with their planned approach for compliance. The new milestone method of acquiring data about residents poses a significant change from what programs are currently doing. The development of clinical competency committee training; the milestones assessment; restructuring of rotations to fit the milestone framework; identifying and remediating at-risk residents; and development of an efficient, reproducible, unbiased, and user-friendly template to collect data will be vital for programs to ease into the transition of the Next Accreditation System. Sharing of information between programs and prospective collection of data specific to plastic surgery will also be vital for continuation of using the milestones to produce proficient plastic surgeons ready to start practice in an unsupervised environment. The education of trainees in plastic surgery is of great importance to all plastic surgeons in both academic and private practice settings. The Next Accreditation System will affect the future of the plastic surgery workforce as it relates to the general public and patient care and interactions among colleagues. This article brings awareness about the complete change that is occurring in trainee education and promotes discussion and further research on the use of this model of assessment.
CONCLUSIONS The Next Accreditation System has been implemented nationwide for all plastic surgery training programs. Twenty-two percent of plastic surgery programs feel completely prepared for Next Accreditation System compliance. Programs are making this transition to find ways to make acquisition and distribution of the data efficient and meaningful for faculty and residents. The stated purpose of Next Accreditation System is to ensure that proficient and trustworthy physicians from all specialties will be entering the workforce. Preliminary data from the first seven specialties that instituted the Next Accreditation System last year demonstrate that physician educators feel
it is beneficial in improving resident education. Future prospective research on the utility and effectiveness of this new model is needed. Samuel J. Lin, M.D. 110 Francis Street, Suite 5A Boston, Mass. 02215 [email protected]
references 1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system: Rationale and benefits. N Engl J Med. 2012;366:1051–1056. 2. Bancroft GN, Basu CB, Leong M, Mateo C, Hollier LH Jr, Stal S. Outcome-based residency education: Teaching and evaluating the core competencies in plastic surgery. Plast Reconstr Surg. 2008;121:441e–448e. 3. Swing SR, Beeson MS, Carraccio C, et al. Educational milestone development in the first 7 specialties to enter the next accreditation system. J Grad Med Educ. 2013;5:98–106. 4. Beeson MS, Carter WA, Christopher TA, et al. The development of the emergency medicine milestones. Acad Emerg Med. 2013;20:724–729. 5. Accreditation Council for Graduate Medical Education. Milestones. Available at: https://www.acgme.org/acgmeweb/ tabid/430/ProgramandInstitutionalAccreditation/ NextAccreditationSystem/Milestones.aspx. Accessed September 28, 2014. 6. Peña A. The Dreyfus model of clinical problem-solving skills acquisition: A critical perspective. Med Educ Online 2010;15. 7. Accreditation Council for Graduate Medical Education; American Board of Plastic Surgery. The plastic surgery milestone project: A joint initiative of the Accreditation Council for Graduate Medical Education and the American Board of Plastic Surgery, Inc. Available at: http://www. acgme.org/acgmeweb/Portals/0/PDFs/Milestones/ PlasticSurgeryMilestones.pdf. Accessed December 6, 2014 8. Bauman LJ, Adair EG. The use of ethnographic interviewing to inform questionnaire construction. Health Educ Q. 1992;19:9–23. 9. Hughes D, DuMont K. Using focus groups to facilitate culturally anchored research. Am J Community Psychol. 1993;21:775–806. 10. Fuller TD, Edwards JN, Vorakitphokatorn S, et al. Using focus groups to adapt survey instruments to new populations. In: Morgan DL, ed. Successful Focus Groups: Advancing the State of the Art. Newbury Park, Calif: Sage; 1993. 11. Accreditation Council for Graduate Medical Education. Data Resource Book: Academic Year 2011–2012. Available at: https://www.acgme.org/acgmeweb/Portals/0/PFAssets/ PublicationsBooks/2011-2012_ACGME_DATABOOK_ DOCUMENT_Final.pdf. Accessed December 6, 2014. 12. Ketteler ER, Auyang ED, Beard KE, et al. Competency champions in the clinical competency committee: A successful strategy to implement milestone evaluations and competency coaching. J Surg Educ. 2014;71:36–38. 13. Korte RC, Beeson MS, Russ CM, Carter WA, Reisdorff EJ; Emergency Medicine Milestones Working Group. The emergency medicine milestones: A validation study. Acad Emerg Med. 2013;20:730–735. 14. Lowry BN, Vansaghi LM, Rigler SK, Stites SW. Applying the milestones in an internal medicine residency program curriculum: A foundation for outcomes-based learner assessment under the next accreditation system. Acad Med. 2013;88:1665–1669.
187 Copyright © 2015 American Society of Plastic Surgeons. Unauthorized reproduction of this article is prohibited.
Background: The Accreditation Council for Graduate Medical Education Next Accreditation System milestones were implemented for plastic surgery programs in July of 2014. Forward progress through the milestones is an indicator of trainee-appropriate development, whereas regression or stalling may indicate the need for concentrated, targeted training. Methods: Online software at www.surveymonkey.com was used to create a survey about the program’s approaches to milestones and was distributed to program directors and administrators of 96 Accreditation Council for Graduate Medical Education–approved plastic surgery programs. Results: The authors had a 63.5 percent response rate (61 of 96 plastic surgery programs). Most programs report some level of readiness, only 22 percent feel completely prepared for the Next Accreditation System milestones, and only 23 percent are completely satisfied with their planned approach for compliance. Seventy-five percent of programs claim to be using some form of electronic tracking system. Programs plan to use multiple tools to capture and report milestone data. Most programs (44.4 percent) plan to administer evaluations at the end of each rotation. Over 70 percent of respondents believe that the milestones approach would improve the quality of resident training. However, programs were less than confident that their current compliance systems would live up to their full potential. Conclusions: The Next Accreditation System has been implemented nationwide for plastic surgery training programs. Milestone-based resident training is a new paradigm for residency training evaluation; programs are in the process of making this transition to find ways to make milestone data meaningful for faculty and residents. (Plast. Reconstr. Surg. 136: 181, 2015.)
T
he Accreditation Council for Graduate Medical Education was established in 1981 and has sought to accentuate program structure, to strengthen the amount and quality of resident education by striking a balance between service and education, and to provide constructive feedback to trainees.1 Over the past three decades, From the Division of Plastic Surgery, Beth Israel Deaconess Medical Center, Harvard Medical School; and the Division of Plastic Surgery, Louisiana State University. Received for publication October 4, 2014; accepted January 27, 2015. Copyright © 2015 by the American Society of Plastic Surgeons DOI: 10.1097/PRS.0000000000001368
Disclosure: The authors have no financial interest to declare in relation to the content of this article. There was no internal or external financial support for this study.
Supplemental digital content is available for this article. Direct URL citations appear in the text; simply type the URL address into any Web browser to access this content. Clickable links to the material are provided in the HTML text of this article on the Journal’s Web site (www. PRSJournal.com).
www.PRSJournal.com
181
Copyright © 2015 American Society of Plastic Surgeons. Unauthorized reproduction of this article is prohibited.
Plastic and Reconstructive Surgery • July 2015 the Accreditation Council for Graduate Medical Education has worked and evolved to achieve the aforementioned goals. In 1999, the Accreditation Council for Graduate Medical Education Outcome Project introduced six “core competencies” (i.e., patient care, medical knowledge, professionalism, systems-based practice, practiced-based learning and improvement, and interpersonal and communications skills) that were to be incorporated into resident training and education to emphasize a more outcomes-based education.2 However, the core competencies were limited in their ability to use outcomes data in accreditation and public accountability because of various factors such as lack of clear conceptual meaning of the competencies, failure to develop a uniform assessment approach, and lack of adequate training of the individuals expected to evaluate residents on the competencies, namely, faculty.3 In addition, compliance was assessed only every 4 to 5 years.1 Therefore, in 2009, the Accreditation Council for Graduate Medical Education began the process of restructuring the accreditation system by instituting several changes, including a clinical competency committee, institution-wide clinical learning environment review visit, and a new evaluation process of trainees. This evaluation process is known as the “milestones” framework, which has five levels to better evaluate residents. All these changes put together are known as the Next Accreditation System.1,3 The purpose of the new system is to better prepare training physicians for future practice through the peer-review system, base accreditation on educational outcomes, and decrease the cumbersome nature related to the current process-based approach. The milestones are one of nine program performance indicators in the Next Accreditation System.4 Milestones for trainees are collected on a semiannual basis.4 Instead of compliance being assessed every 4 to 5 years, milestone data will be used in annual accreditation review, therefore allowing the period between formal accreditation visits to be increased to every 10 years.1,4 Milestones are competency-based developmental outcomes of core specialty-specific knowledge, skills, attitudes, and beliefs that residents should be able to progress through from the beginning of training through graduation, with the overall goal of producing safe, competent, ethical physicians.3–5 The milestones were developed in each specialty through workgroups that consisted of American Board of Medical Specialties certifying boards, review committees, medical specialty organizations, program-director
associations, and residents.1 Subcompetencies were identified within the six core competencies with the goal of developing approximately 36 (or other number deemed appropriate by the workgroups) subcompetencies for each American Board of Medical Specialties specialty.3 A five-level model for the milestones was developed based on concepts adapted from the Dreyfus model of expertise. The Dreyfus model of expertise was initially developed after evaluation of jet pilots in the Air Force and was a model of professional expertise in which individuals’ progress through various stages: novice, advanced beginner, competent, proficient, and expert.3,6 Since its development, this model has been expanded from the acquisition of skills by jet pilots and modified to apply to other disciplines. The Accreditation Council for Graduate Medical Education adaptation of the model includes milestones that residents entering a specialty who have recently graduated from medical school should be able to attain (level 1), progressively advancing to goal competencies for graduation (level 4), to highly ambitious goals for extraordinary residents or physicians already in practice (level 5).3 In addition to the milestones, other information to be collected for annual surveillance will be Accreditation Council for Graduate Medical Education surgery forms for faculty and residents, and operative and case logs.1 Early on, the milestone data will be used to evaluate the quality of resident and fellow training based on outcomes.4,5 The first phase of the Next Accreditation System was initiated in July of 2013 in seven specialties: emergency medicine, internal medicine, neurologic surgery, orthopedic surgery, pediatrics, diagnostic radiology, and urology. Initial data for the validity and utility of the milestones in these initial specialties is starting to emerge.3 In July of 2014, plastic surgery programs began implementation of the Next Accreditation System for evaluation of resident and fellows by use of the milestone criteria. 5,7 (See Figure, Supplemental Digital Content 1, which shows the Accreditation Council for Graduate Medical Education plastic surgery milestones, http://links.lww.com/PRS/B337.) This paradigm shift represents a substantial change in evaluation of resident and fellows from what most plastic surgery programs are accustomed to. Although the milestones have been used in other specialties for a little over 1 year, they are very new for plastic surgery programs; therefore, we sought to evaluate how the Next Accreditation System is being implemented across plastic surgery programs nationwide.
182 Copyright © 2015 American Society of Plastic Surgeons. Unauthorized reproduction of this article is prohibited.
Volume 136, Number 1 • Plastic Surgery Accreditation Milestones METHODS Recruitment Institutional review board exemption was obtained. The online software available at www. surveymonkey.com was used to create the survey. (See Figure, Supplemental Digital Content 2, which shows the Accreditation Council for Graduate Medical Education plastic surgery milestones survey, http://links.lww.com/PRS/B338.) An e-mail invitation containing a generic link to the survey, which also ensured anonymity and prevented tracking, was distributed to 96 Accreditation Council for Graduate Medical Education– approved plastic surgery programs (integrated and independent programs). The e-mail was sent to both the program directors and their administrators. Following collection of all responses, the data were processed. No incentives were given for participation in this study. There was no preselection based on anticipated responses. Survey The Web-based survey gathered basic information on program demographics, including how a program would characterize itself and the number of residents enrolled per year. Inquiries were made about the prevalence of electronic tracking systems to record milestones, how programs capture and report data, and what strategies were used in the individual program’s approach to milestones. There were two types of approaches that were assessed: the bottom-up approach, where each milestone was broken into multiple component parts (e.g., evaluate specific skills such as wound débridement within the “patient care for tissue transfer” milestone); and the top-down approach, which used Accreditation Council for Graduate Medical Education milestones as primary evaluation points (e.g., rate patient care on a scale of 1 to 5, or other strategies), and the anticipated ongoing expenditures to comply with the new Accreditation Council for Graduate Medical Education milestones requirements. Furthermore, questions were asked about planning and rollout for Accreditation Council for Graduate Medical Education milestone compliance, hours per month spent on milestone compliance, readiness of programs for milestone compliance, frequency of evaluations, and involvement of other hospital staff for data acquisition. Finally, participating programs were asked about their overall perception of the Accreditation Council for Graduate Medical Education milestones approach. To ensure validation of the survey, it was presented to a focus
group of students at the Massachusetts Institute of Technology Sloan School of Management. This was done because only 96 Accreditation Council for Graduate Medical Education–accredited plastic surgery programs currently exist; as such, forming a smaller subset of that population as a test group for the survey was not feasible because the data provided from this subset would be pertinent to the findings of the study. Although the background of the focus group differs from that of the program directors, this approach has been previously described in the validation process of surveys.8–10 The survey was revised until the final draft was complete.
RESULTS Plastic Surgery Program Demographics The data for this study were ascertained through 61 plastic surgery program responses (response rate, 63.5 percent). A majority (72 percent) of survey respondents were affiliated with a university- and/or academic-based program. Most programs (44.3 percent) had one or two residents enrolled per year, which correlates with national data,11 and approximately 30 percent reported having five or more residents enrolled on an annual basis (most likely, those programs with both integrated and independent residents). Plastic Surgery Programs and Accreditation Council for Graduate Medical Education Milestones Of the 61 plastic surgery programs participating in this survey, three-quarters were noted to be using some form of an electronic tracking system; similarly, three-quarters of those tracking milestones electronically plan to use a third-party system. Programs plan to use multiple tools to capture and report milestone data: 66.8 percent expect to use Web-based surveys, 46.3 percent anticipate that they will use some form of paper-based survey, and 63 percent expect to conduct interviews. In terms of compliance with milestones, 64.1 percent of participating programs plan to use a top-down approach, 22.6 percent plan to use a bottom-up approach, and the remainder had reported either not yet finalizing their approach or using a combination of both. Most programs do not anticipate upfront expenditures (74 percent) or ongoing expenditures (71 percent) to comply with the new Accreditation Council for Graduate Medical Education milestones requirement. Of those who anticipate
183 Copyright © 2015 American Society of Plastic Surgeons. Unauthorized reproduction of this article is prohibited.
Plastic and Reconstructive Surgery • July 2015 expenditures, there was a wide array of cost estimates ranging from $5000 to $20,000 for both, which were attributed to uncertainty, variability related to time and effort, and nominal costs to plastic surgery departments, in addition to large costs to the institution and significant labor costs related to an increase in staff and technology expenses. The program director and the program administrator are the most involved individuals in the planning and rollout for Accreditation Council for Graduate Medical Education milestones compliance (96 percent and 86 percent, respectively). Less than half of participating programs plan to engage residents, hospital administrators, or the graduate medical education coordinator (Fig. 1). On average, programs expect their teams to commit 34 collective hours per month to milestone compliance. Program directors and faculty are each expected to spend 8 to 9 hours per month, and program administrators are expected to spend approximately 12 hours per month. The vast majority of responding programs report some level of readiness, but only 22 percent of programs feel completely prepared for milestone compliance. Similarly, only 23 percent of program respondents were completely satisfied with their planned approach for compliance (Fig. 2). Most programs plan to administer evaluations at the end of each rotation (44.4 percent), whereas some (26.7 percent) are planning evaluations every 6 months (Fig. 3). Many programs plan to involve nursing staff (70.5 percent) and midlevel providers such as physician assistants and nurse practitioners (59 percent) in their milestones compliance systems. Fifty percent of programs
Fig. 1. Compliance team. PD, program director; PA, program administrator; PF, program faculty; R, residents; HA/GME lead, hospital administrator/graduate medical education coordinator.
will engage physicians from other specialties, whereas 22.7 percent do not plan to involve anyone outside of plastic surgery. Over 70 percent of respondents believe that the milestones approach would improve the quality of the residency program to some extent in terms of both the depth and the quality of a resident’s training. However, programs appeared less than confident that their compliance systems as currently planned would be compliant with these new guidelines (Fig. 4).
DISCUSSION The Accreditation Council for Graduate Medical Education Next Accreditation System is an extension of the Outcomes Project previously developed with the emphasis on data-based outcomes rather than process-based outcomes, with the use of milestones for acquisition of these data.4 Forward progress through the milestones will be an indicator of resident- or fellow-appropriate development, whereas regression or stalling may indicate the need for concentrated, targeted education/training. Overall, the progression of trainees through the milestones will be an indicator of the effectiveness of residency and fellowship training programs of producing high-quality and dependable physicians to take care of the greater public. One stated goal of the Next Accreditation System was to identify high-functioning programs that could then be given the latitude to be innovative regarding resident trainee education; conversely, this system may be used to identify programs not as high functioning, allowing identification of areas of improvement rather than citation.1,4 Milestone data will be collected about each resident by a clinical competency committee composed of staff on a semiannual basis and entered electronically to the Accreditation Council for Graduate Medical Education. The role of the clinical competency committee is to determine a resident’s acquisition and progression or regression through the milestones and make appropriate recommendations of advancement, graduation, or remediation accordingly.5 The data will provide residents with tangible feedback on an individual level, and the collective resident acquisition of milestones will be used annually for accreditation review of overall program performance.4 In our study, we found that most plastic surgery programs are planning to involve the program director, program administrator, and faculty on a clinical competency committee, but less than half plan to engage residents or other hospital administrators. In addition, on average, programs expect their clinical competency
184 Copyright © 2015 American Society of Plastic Surgeons. Unauthorized reproduction of this article is prohibited.
Volume 136, Number 1 • Plastic Surgery Accreditation Milestones
Fig. 2. Readiness assessment.
Fig. 3. Frequency of evaluations.
committee to spend 34 collective hours monthly to be compliant with the milestones. The University of New Mexico General Surgery Residency program was a test site for the milestones from 2012
to 2013. Their clinical competency committee was composed of six faculty “competency champions” appointed by the program director who were each assigned one of the six Accreditation Council for Graduate Medical Education competencies based on expertise for the competency. In addition to the six champions, the clinical competency committee also included the program director, associate program director, chair of the department, and two atlarge faculty members. In addition, the two chief residents were included as nonvoting members. The clinical competency committee met monthly, with rotating competency topics, with the goal of each competency being discussed twice by the end of the year. In addition, an entire class was discussed at each meeting. As the champions became more knowledgeable about the competencies and more efficient, an entire class could be reviewed in only 1 hour. Because of frequent meetings, struggling residents were better identified and placed on remediation programs headed by the champion for the particular competency. After 1 year, this program noted that faculty were better able to provide constructive and tangible feedback; at-risk residents
185 Copyright © 2015 American Society of Plastic Surgeons. Unauthorized reproduction of this article is prohibited.
Plastic and Reconstructive Surgery • July 2015
Fig. 4. Overall perception of Next Accreditation System milestones. ACGME, Accreditation Council for Graduate Medical Education.
were able to be identified and, if needed, put in a remediation program to work on improving a specific milestone; residents had clearer knowledge of what was expected of them; and, overall, rotations were better structured to comply with attainment of the milestones.12 Diverse composition of a clinical competency committee with specifically designated roles and frequent meetings eventually may also provide further viewpoints. These early data demonstrate that diverse composition of a clinical competency committee with specifically designated roles and frequent meetings may be a model that is beneficial to both staff and residents; however, further prospective data collection on the milestones is needed. Implementation of the Next Accreditation System and collection of milestones began nationally for all plastic surgery programs in July of 2014; however, seven specialties initiated the program over 1 year ago and have been using the milestones for resident and fellow outcomes-based assessment. Emergency medicine was one of the first specialties to develop and implement use of the milestones in 2013. During the developmental process
before implementation, the proposed milestones were sent in a survey to all accredited emergency medicine residency program directors and key faculty for validation.13 The internal medicine community developed milestones through numerous workshops. To comply with the Next Accreditation System, the internal medicine residency program at the University of Kansas restructured their curricula to better align with the milestones and created templates for each rotation to collect the outcome data. Once developed, the internal medicine department worked with the graduate medical education office and technical support team to make the templates electronic to be more available to faculty and residents. Currently, the developers of this program are performing objective data collection of their system but believe sharing of data nationally on collaborative portals of how programs are implementing the new milestone framework will be vital.14 Survey of the internal medicine community with regard to the utility of the milestones demonstrated that 85 percent of 253 faculty, residents, and others thought the milestones would enhance
186 Copyright © 2015 American Society of Plastic Surgeons. Unauthorized reproduction of this article is prohibited.
Volume 136, Number 1 • Plastic Surgery Accreditation Milestones resident education, feedback to trainees, and faculty development.3 The orthopedic surgery milestones underwent pilot testing before implementation, and 80 percent of the 75 participants thought milestones allowed for meaningful resident evaluation.3 In the current study, over 70 percent of respondents feel the milestones approach will improve the quality of plastic surgery training to some extent. Overall, from our study, it appears that most programs report some level of readiness, but only 22 percent of programs feel completely prepared for milestone compliance and only 23 percent were completely satisfied with their planned approach for compliance. The new milestone method of acquiring data about residents poses a significant change from what programs are currently doing. The development of clinical competency committee training; the milestones assessment; restructuring of rotations to fit the milestone framework; identifying and remediating at-risk residents; and development of an efficient, reproducible, unbiased, and user-friendly template to collect data will be vital for programs to ease into the transition of the Next Accreditation System. Sharing of information between programs and prospective collection of data specific to plastic surgery will also be vital for continuation of using the milestones to produce proficient plastic surgeons ready to start practice in an unsupervised environment. The education of trainees in plastic surgery is of great importance to all plastic surgeons in both academic and private practice settings. The Next Accreditation System will affect the future of the plastic surgery workforce as it relates to the general public and patient care and interactions among colleagues. This article brings awareness about the complete change that is occurring in trainee education and promotes discussion and further research on the use of this model of assessment.
CONCLUSIONS The Next Accreditation System has been implemented nationwide for all plastic surgery training programs. Twenty-two percent of plastic surgery programs feel completely prepared for Next Accreditation System compliance. Programs are making this transition to find ways to make acquisition and distribution of the data efficient and meaningful for faculty and residents. The stated purpose of Next Accreditation System is to ensure that proficient and trustworthy physicians from all specialties will be entering the workforce. Preliminary data from the first seven specialties that instituted the Next Accreditation System last year demonstrate that physician educators feel
it is beneficial in improving resident education. Future prospective research on the utility and effectiveness of this new model is needed. Samuel J. Lin, M.D. 110 Francis Street, Suite 5A Boston, Mass. 02215 [email protected]
references 1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system: Rationale and benefits. N Engl J Med. 2012;366:1051–1056. 2. Bancroft GN, Basu CB, Leong M, Mateo C, Hollier LH Jr, Stal S. Outcome-based residency education: Teaching and evaluating the core competencies in plastic surgery. Plast Reconstr Surg. 2008;121:441e–448e. 3. Swing SR, Beeson MS, Carraccio C, et al. Educational milestone development in the first 7 specialties to enter the next accreditation system. J Grad Med Educ. 2013;5:98–106. 4. Beeson MS, Carter WA, Christopher TA, et al. The development of the emergency medicine milestones. Acad Emerg Med. 2013;20:724–729. 5. Accreditation Council for Graduate Medical Education. Milestones. Available at: https://www.acgme.org/acgmeweb/ tabid/430/ProgramandInstitutionalAccreditation/ NextAccreditationSystem/Milestones.aspx. Accessed September 28, 2014. 6. Peña A. The Dreyfus model of clinical problem-solving skills acquisition: A critical perspective. Med Educ Online 2010;15. 7. Accreditation Council for Graduate Medical Education; American Board of Plastic Surgery. The plastic surgery milestone project: A joint initiative of the Accreditation Council for Graduate Medical Education and the American Board of Plastic Surgery, Inc. Available at: http://www. acgme.org/acgmeweb/Portals/0/PDFs/Milestones/ PlasticSurgeryMilestones.pdf. Accessed December 6, 2014 8. Bauman LJ, Adair EG. The use of ethnographic interviewing to inform questionnaire construction. Health Educ Q. 1992;19:9–23. 9. Hughes D, DuMont K. Using focus groups to facilitate culturally anchored research. Am J Community Psychol. 1993;21:775–806. 10. Fuller TD, Edwards JN, Vorakitphokatorn S, et al. Using focus groups to adapt survey instruments to new populations. In: Morgan DL, ed. Successful Focus Groups: Advancing the State of the Art. Newbury Park, Calif: Sage; 1993. 11. Accreditation Council for Graduate Medical Education. Data Resource Book: Academic Year 2011–2012. Available at: https://www.acgme.org/acgmeweb/Portals/0/PFAssets/ PublicationsBooks/2011-2012_ACGME_DATABOOK_ DOCUMENT_Final.pdf. Accessed December 6, 2014. 12. Ketteler ER, Auyang ED, Beard KE, et al. Competency champions in the clinical competency committee: A successful strategy to implement milestone evaluations and competency coaching. J Surg Educ. 2014;71:36–38. 13. Korte RC, Beeson MS, Russ CM, Carter WA, Reisdorff EJ; Emergency Medicine Milestones Working Group. The emergency medicine milestones: A validation study. Acad Emerg Med. 2013;20:730–735. 14. Lowry BN, Vansaghi LM, Rigler SK, Stites SW. Applying the milestones in an internal medicine residency program curriculum: A foundation for outcomes-based learner assessment under the next accreditation system. Acad Med. 2013;88:1665–1669.
187 Copyright © 2015 American Society of Plastic Surgeons. Unauthorized reproduction of this article is prohibited.
