The Journal of Emergency Medicine, Vol. 47, No. 4, pp. 432–440, 2014 Copyright Ó 2014 Elsevier Inc. Printed in the USA. All rights reserved 0736-4679/$ - see front matter
http://dx.doi.org/10.1016/j.jemermed.2014.04.032
Education PRACTICING EMERGENCY PHYSICIANS REPORT PERFORMING WELL ON MOST EMERGENCY MEDICINE MILESTONES Timothy C. Peck, MD,*† Nicole Dubosh, MD,*† Carlo Rosen, MD,*† Carrie Tibbles, MD,*† Jennifer Pope, MD,‡ and Jonathan Fisher, MD, MPH*† *Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts, †Harvard Medical School, Boston, Massachusetts, and ‡Emergency Medicine Department, St Luke’s Hospital, New Bedford, Massachusetts Reprint Address: Timothy C. Peck, MD, Department of Emergency Medicine, Beth Israel Deaconess Medical Center, One Deaconess Road, West Campus Clinical Center, 2nd Floor, Boston, MA 02215
, Abstract—Background: The Accreditation Council for Graduate Medical Education’s Next Accreditation System endorsed specialty-specific milestones as the foundation of an outcomes-based resident evaluation process. These milestones represent five competency levels (entry level to expert), and graduating residents will be expected to meet Level 4 on all 23 milestones. Limited validation data on these milestones exist. It is unclear if higher levels represent true competencies of practicing emergency medicine (EM) attendings. Objective: Our aim was to examine how practicing EM attendings in academic and community settings selfevaluate on the new EM milestones. Methods: An electronic self-evaluation survey outlining 9 of the 23 EM milestones was sent to a sample of practicing EM attendings in academic and community settings. Attendings were asked to identify which level was appropriate for them. Results: Seventy-nine attendings were surveyed, with an 89% response rate. Sixty-one percent were academic. Twenty-three percent (95% confidence interval [CI] 20% 27%) of all responses were Levels 1, 2, or 3; 38% (95% CI 34% 42%) were Level 4; and 39% (95% CI 35% 43%) were Level 5. Seventy-seven percent of attendings found themselves to be Level 4 or 5 in eight of nine milestones. Only 47% found themselves to be Level 4 or 5 in ultrasound skills (p = 0.0001). Conclusions: Although a majority of EM attendings reported meeting Level 4 milestones, many felt they did not
meet Level 4 criteria. Attendings report less perceived competence in ultrasound skills than other milestones. It is unclear if self-assessments reflect the true competency of practicing attendings. The study design can be useful to define the accuracy, precision, and validity of milestones for any medical field. Ó 2014 Elsevier Inc. , Keywords—graduate medical education; emergency medicine milestones; ACGME; Next Accreditation System; resident evaluation
INTRODUCTION As part of its Next Accreditation System, the Accreditation Council for Graduate Medical Education (ACGME) has recently endorsed and implemented specialty-specific milestones as the foundation of a new outcomes-based resident evaluation process (1). The ACGME has explicitly stated the following intentions behind its actions: basing residency program accreditation on educational outcomes, demonstrating to the public the effectiveness of competence-based education, and changing a system that does not encourage innovation and has become prescriptive (1). Empowered by this directive, the emergency medicine (EM) community completed development of the EM milestones in 2012 after an intensive process (2,3).
This study was exempt from our hospital’s Institutional Review Board.
RECEIVED: 2 October 2013; FINAL SUBMISSION RECEIVED: 30 January 2014; ACCEPTED: 28 April 2014 432
EM Milestone Self-Assessment
The EM milestones will assess resident performance within the following six core competency domains as defined by the ACGME Outcomes Project: Patient Care, Medical Knowledge, Practice-Based Learning and Improvement, Interpersonal and Communication Skills, Professionalism, and Systems-Based Process (4). Each milestone consists of five levels of resident competence ranging from entry level (medical school graduate/ incoming resident) to expert (5). Level 4 is the expected level of competency of an individual at the time of graduation from residency, i.e., the American Board of Emergency Medicine (ABEM) certification standard. In addition, program directors will be expected to file biannual reports on individual resident progress (4). The EM milestones are a substantial achievement, however, the milestones are based on expert consensus and the completed milestones as written have not been studied extensively or validated. For example, although Level 4 is meant to represent the needed competency for a graduating resident to successfully practice EM, it is unclear without the benefit of prospective validation data if the Level 4 milestones as they are currently written truly represent the current skill set of independent practitioners of EM. Our objective was to examine how practicing EM physicians in academic and community settings evaluate their own performance on the new EM milestones. We hypothesized that if Level 4 milestones truly represent a graduating resident’s competency, then nearly all attendings would self-evaluate at Level 4 or 5. METHODS We performed a prospective evaluation of practicing EM physicians in both community and academic settings. The study was conducted from November 1, 2012 through November 24, 2012. A self-evaluation survey was compiled outlining the EM milestones and was sent electronically to a convenience sample of attending physicians at four different institutions, including one academic site and three community sites. The academic institution was a Level I trauma center with an ACGME-accredited EM residency and an annual volume of 57,000 emergency department visits. The three community sites had volumes of 63,000, 62,000, and 25,000 visits per year. A subset of 9 of the 23 milestones was selected and included in the survey. One milestone was selected from each of the six core competencies and three additional milestones were selected due to their interesting characteristics, such as Ultrasound. Demographic information was obtained, including academic vs. community, years of practice, and board certification.
433
Participants were asked to rate themselves on each of the nine milestones using the 1 5 scale outlined by the EM Milestone Task Force (see Figure 1). Data were collected using Surveymonkey (Surveymonkey.com, LLC 2010) and responses were deidentified to ensure that the study investigators were blinded to individual responses. Data were analyzed using Microsoft Excel (2003, Microsoft, Redmond, WA) to calculate proportions and confidence intervals (CIs). Participants were informed by e-mail that participation in this survey was voluntary and that by submitting their responses, they were consenting to use of their deidentified data for research and publication purposes. The project was submitted to the local Institutional Review Board. It was determined that the study met criteria for exemption under federal guidelines. RESULTS Seventy-nine attendings were surveyed, with an 89% response rate. Sixty-one percent were academic. Thirtyfour percent were practicing for 0 5 years. Ninetythree percent graduated EM residency (see Table 1). The study’s power precluded any significant associations between milestone level and years of practice, academic vs. community setting, or EM boarded vs. not. Overall, 77% (95% CI 74% 80%) of all responses by EM physicians were at the Level 4 or 5. Twenty-three percent (95% CI 20% 27%) of all responses were Level 1, 2, or 3, 38% (95% CI 34% 42%) were Level 4, and 39% (95% CI 35% 43%) were Level 5 (see Table 2). Additionally, 77% of attendings found themselves to be Level 4 or 5 in 8 of 9 milestones, and only 47% found themselves to be Level 4 or 5 in Ultrasound (Patient Care [PC] 10) (p = 0.0001) (see Table 2). DISCUSSION By surveying practicing EM physicians, we have learned that a large proportion of attending-level physicians do see themselves as at least Level 4 competency, i.e., the Milestone Task Force’s definition of the expected competency of graduating residents. However, some EM physicians report they do not meet this expected level of competence on at least one milestone. There are several possible explanations for our findings. This might represent a problem with accuracy and precision in which EM attendings might not be able to correctly self-assess their own competency or they might not understand how to use the milestones. Alternatively, there may be issues with external validity of the current incarnation of the milestones.
434
T. C. Peck et al.
Figure 1. Self-assessment survey questions as they appeared on electronic survey. Each question is designed for practicing emergency medicine (EM) physician self-assessment of one of nine selected EM milestones. Question 5 = Patient Care (PC) 3: Diagnostic Studies; Question 6 = PC6: Observation and Reassessment; Question 7 = PC10: Airway Management; Question 8 = PC12: Goal-Directed Focused Ultrasound; Question 9 = Medical Knowledge; Question 10 = Professionalism 1: Professional Values; Question 11 = Interpersonal and Communication Skills 1: Patient-Centered Communication; Question 12 = Practice-Based Learning and Improvement 1: Practice-Based Performance Improvement; Question 13 = Systems Based Process 1: Patient Safety. ED = emergency department, BVM = bag valve mask, echo = echocardiogram, TEE = transesophageal echocardiogram, ABEM = American Board of Emergency Medicine, M&M = morbidity and mortality.
EM Milestone Self-Assessment
Figure 1. (continued).
435
436
Figure 1. (continued).
T. C. Peck et al.
EM Milestone Self-Assessment
437
Table 1. Demographics of Survey Participants Practice Environment
Percent total of the 79 survey participants*
Years Practicing
Residency Training
Academic Practitioner
Community Practitioner
0 5
6 10
>11
Graduated EM Residency
Did Not Graduate from EM Residency
61
39
35
19
47
93
7
EM = emergency medicine. * All survey participants were practicing EM physicians.
Creation of EM Milestones An understanding of the timeline and process that lead to the creation of the milestones is illustrative. The milestone levels were constructed by a thorough and thoughtful process that involved expert consensus, polling large samples of emergency physicians, and the work of various committees (3,6). The ABEM Initial Certification Task Force (ICTF) was created and tasked with studying the initial EM physician certification process and understanding current standards of practice in EM. The ICTF composed relevant and clear definitions of the true tasks of current practicing EM physicians (7). The ABEM Relevance of Examinations to Physician Practice Task Force then recommended changes to the content and methods of ABEM’s examinations to accurately reflect the newly created definitions (8). Working from the new definitions, an ICTF Advisory Panel then further refined the expectations of an individual pursuing initial certification in EM in the form of Knowledge, Skills, and Abilities (KSAs). ABEM then collected survey responses from >7000 EM diplomats, which queried the importance and frequency of each KSA (9,10). Empowered by the KSAs, whose frequency and importance were defined by actual EM practitioners, a Milestone Working Group then ultimately identified 23 Milestones for the ACGME’s six core competencies (i.e., Patient Care, Medical Knowledge, Professionalism, Interpersonal
Communication Skills, Practice-Based Learning and Improvement, and Systems-Base Practice) (11). Drawing off of ABEM’s work on hierarchical scales of performance, each milestone was broken into five competency levels, with the intent of creating uniformity in milestone reporting. The scale ranges from Level 1, representing the expected competency of a medical school graduate to Level 5, representing the competency of an experienced practitioner (5). ABEM and the Council of Emergency Medicine Residency Directors/Clerkship Directors in Emergency Medicine then made recommendations for methods of milestone level assessment. Beginning in July 2013, the products of this comprehensive process were integrated into EM program requirements as part of the Next Accreditation System pilot (6). Accuracy and Precision of the Milestones Level 4 is meant to reflect the ABEM certification standards, however, 23% of responses did not meet the expected Level 4 competency. We find this percentage remarkably high, and it raises a question about the validity of the milestones. Based on these results, it is unclear if the Level 4 milestones in their current form truly reflect the actual skill set needed to practice EM successfully. There may be consequences to inaccurate milestones. If residents are required to meet Level 4 expectations in order to graduate residency, then it follows that residencies
Table 2. Level of Self-Assessment by Practicing Emergency Medicine Physicians on Selected Emergency Medicine Milestones Milestone*
Level 1
Level 2
Level 3
Level 4
Level 5
PC3: Diagnostic Studies PC6: Observation and Reassessment PC10: Airway Management PC12: Goal-Directed Focused Ultrasound MK: Medical Knowledge PROF1: Professional Values ICS1: Patient-Centered Communication PBL1: Practice-Based Performance Improvement SBP1: Patient Safety Overall
0 3 0 7 1 3 4 1 6 3
0 3 4 13 3 0 0 7 10 4
21 16 10 33 3 16 16 23 14 16
30 59 23 46 9 43 43 40 37 38
49 20 63 1 84 39 39 27 33 39
ICS = Interpersonal and Communication Skills; MK = Medical Knowledge; PBL = Practice-Based Learning and Improvement; PROF = Professionalism; SBP = Systems-Based Process; PC = Patient Care. Values are percentages. * At least one milestone was selected from each of the six Accreditation Council for Graduate Medical Education core competencies.
438
will be obligated to move resources and efforts toward assuring that this occurs. The nature of residency programs dictates that trainees are limited by time and resources. There exists a finite amount of training hours during the course of a residency; there are also a finite amount of funds in the budget of each residency. Asking residents to be competent in one skill or proficiency may result in less time dedicated to other proficiencies that a residency deems important. If the minimum competency level is set higher than that necessary to practice effectively, then the inaccuracy of that milestone may unnecessarily pull resources away from other skills and proficiencies that residency programs and residents may find more valuable. This reality becomes more clear when one considers that ACGME residency accreditation may depend on meeting Level 4 milestones. The majority of attendings surveyed did not meet Level 4 on Goal-Directed Focus Ultrasound (PC12). This may reflect the nature of the advancement of this technology and that attendings already practicing may not have the ultrasound competency that the EM community deems to be important for future attending-level physicians. For the ultrasound milestone, attendings may be asked to train their residents to a level higher than their own competency. This presents a challenge, and more resources may need to be diverted toward ultrasound training. Based on our results, a minority of surveyed attendings met Level 5 milestone competency (39%). Level 5 is meant to represent an expert (5). As stated here, the ACGME will be asking residencies to divert their resources toward achieving Level 4 and they are not overtly asking to train residents to Level 5. However, the very existence of Level 5 competencies may have unintended consequences. A stated goal of the ACGME is to hold residencies accountable to the public (1). If the rates of milestone achievements are reported to the public, residencies will surely be pressured to attempt to train their residents to Level 5. Our culture puts value on rankings, and a system that ranks residents on competency level may translate into publically available reportable metrics in which residencies could be judged and tiered—not unlike colleges or hospitals. The practicing EM physician responses also have revealed that not all milestones have a similar per-level distribution of attending self-evaluation. Level 4 on one milestone may not be equivalent to Level 4 on another milestone. As a group, attendings found themselves to be at competency Level 4 or above at widely different rates, depending on which milestone they were asked to consider. Level 4/5 rates ranged from 47% on Ultrasound to 93% on Medical Knowledge. A wide inter-milestone variation will make some milestones more pertinent than others. As the milestones stand now, Level 4 compe-
T. C. Peck et al.
tency in Medical Knowledge may be too easy to achieve, thus devaluing medical knowledge. Ultrasound competency, however, may be difficult to achieve, which may necessitate more attention to ultrasound skills and less attention to assuring our residents have a sound base of medical knowledge. Limitations There are limitations to our study. It is unclear if anonymous self-assessment is truly an accurate representation of actual practicing physician competency. Attending physicians may not be able to accurately or precisely measure their actual skill level. An often-cited review of physician self-assessment literature by Davis et al. concludes that ‘‘while suboptimal in quality, the preponderance of evidence suggests that physicians have a limited ability to accurately self-assess. The processes currently used to undertake professional development and evaluate competence may need to focus more on external assessment’’ (12). Our lack of external assessment is a limitation. However, we find it difficult to extrapolate Davis’ conclusion to our own study. The Davis review was based on only 20 studies that met their rigorous standards for inclusion. In addition, only seven were of retrospective self-analyses that asked the participants to provide mental representations of themselves over time like that of our survey (13 19). Of these seven studies on retrospective self-assessment, four were deemed to suggest a positive correlation of external and self assessment, one suggested inverse correlation, and two yielded inconclusive results (13 19). Although we had an impressive response rate to our survey, our sample size is relatively small and future investigations should aim to include a larger sample of the practicing EM physician population. It is also unclear if some milestones are easier to self-assess than others. In order to maximize the survey response rate, we only asked for data on nine of the milestones. It is quite possible that these 9 are not an accurate representation of all 23 milestones. Therefore, our results may not apply to the milestones as a complete set, and may only relate to the nine that were tested. To obtain Level 5 in three of the milestones we studied, the physician must be in the position to teach the proficiencies within that milestone. It is possible that community physicians in our study do not have the opportunity to teach and would therefore evaluate themselves at a Level 4 even though they are expert in that milestone. There is the chance that attendings did not understand the milestones themselves, although we did reproduce the language of the milestones word for word. Our results must be interpreted in this context. We had a small number of attendings answer that they were Level 1 for some
EM Milestone Self-Assessment
439
of the milestones; this is an unlikely reality, and can skew the statistical analysis of our milestone assessment. 7.
CONCLUSIONS Although a majority of EM attendings who completed our self-assessment survey reported meeting Level 4 milestones, many attendings felt they did not meet Level 4 criteria. Attendings report less perceived competence in ultrasound skills than other surveyed milestones. It is unclear if self-assessments reflect the true competencies of practicing attendings. Reproducing this study with a larger sample size would likely yield valuable data. A well-powered study could clarify and validate the results of our study, which used a convenience sample of 79 physicians. Furthermore, associations may be found between self-assessed milestone level and various metrics, such as physician familiarity with milestones, years of practice, and academic vs. community setting. Finally, this model of study may be useful for fields other than EM to better define the accuracy, precision, and validity of their own milestones. The model may be especially useful to those fields that have yet to create, or are in the process of creating, their milestones.
8.
9.
10.
11.
12.
13.
REFERENCES 1. Nasca TJ. The Next Accreditation System—rational and benefits. N Engl J Med 2012;366:1051–6. 2. Ling LJ, Beeson MS. Milestones in emergency medicine. J Acute Med 2012;2:65–9. 3. Beeson MS, Carter WA, Christopher TA, et al. The development of the emergency medicine milestones. Acad Emerg Med 2013;20: 724–9. 4. Accreditation Council for Graduate Medical Education. Common program requirements. ACGME Approved Standards. Available at: http://www.acgme.org/acgmeweb/Portals/0/PFAssets/Program Requirements/CPRs2013.pdf. September 26, 2010. Accessed June 15, 2014. 5. Accreditation Council for Graduate Medical Education and American Board of Emergency Medicine. The emergency medicine milestones project. ACGME Next Accreditation System: Rationale and Benefits, Milestones. Available at: https://www.acgme. org/acgmeweb/Portals/0/PDFs/Milestones/EmergencyMedicine Milestones.pdf. December 2012. Accessed June 15, 2014. 6. Beeson M. Emergency medicine milestones. 2012 CORD Academic Assembly Schedule/Handouts/Materials, Day 1. Available
14.
15. 16. 17. 18.
19.
at: http://www.cordem.org/files/DOCUMENTLIBRARY/2012% 20AA%20Handouts/1%20Day/CORD%20AA%20milestones% 20(2).ppt. April 10, 2012. Accessed June 15, 2014. Perina DG, Brunett CP, Caro DA, et al. The 2011 model of the clinical practice of emergency medicine. Acad Emerg Med 2012;19: e19–40. American Board of Emergency Medicine. Diplomates to Participate in the Development of Initial Certification. ABEM Memo 2010; Vol 12(1). https://www.abem.org/public/docs/default-source/ publication-documents/2010-spring-newsletter.pdf?sfvrsn=6. Available at: Accessed on June 15th, 2014. Steele MT. Report at the American Academy of Emergency Medicine Scientific Assembly. American Board of Emergency Medicine Reports at National Meetings. Available at: https://www.abem.org/ public/docs/default-source/publication-documents/aaem-2011final56E286AC8033.pdf. October 10, 2011. Accessed June 15, 2014. American Board of Emergency Medicine. Annual Report, 20112012. Available at: https://www.abem.org/public/docs/defaultsource/publication-documents/2010-11-annual-report.pdf. July 27, 2012. Accessed June 15, 2014. The Accreditation Council for Graduate Medical Education and The American Board of Emergency Medicine. The emergency medicine milestone project. American Board of Emergency Medicine Emergency Medicine Milestones website. Available at: https:// www.abem.org/public/docs/default-source/migrated-documentsand-files/em-milestones.pdf?sfvrsn=4. October 22, 2012. Accessed June 15, 2014. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA 2006;296:1094–102. Claridge JA, Calland JF, Chandrasekhara V, Young JS, Sanfey H, Schirmer BD. Comparing resident measurements to attending surgeon self-perceptions of surgical educators. Am J Surg 2003;185: 323–7. Fernandez A, Schillinger D, Grumbach K, et al. Physician language ability and cultural competence. An exploratory study of communication with Spanish-speaking patients. J Gen Intern Med 2004;19: 167–74. Hoppe RB, Farquhar LJ, Henry R, Stoffelmayr B. Residents’ attitudes towards and skills in counseling: using undetected standardized patients. J Gen Intern Med 1990;5:415–20. Ireton HR, Sherman M. Self-ratings of graduating family practice residents’ psychological medicine abilities. Fam Pract Res J 1988; 7:236–44. Johnson D, Cujec B. Comparison of self, nurse, and physician assessment of residents rotating through an intensive care unit. Crit Care Med 1998;26:1811–6. Paradise JE, Finkel MA, Beiser AS, Berenson AB, Greenberg DB, Winter MR. Assessments of girl’s genital findings and the likelihood of sexual abuse: agreement among physicians self-rated as skilled. Arch Pediatr Adolesc Med 1997;151:883–91. Robbins JM, Kirmayer LJ, Cathe´bras P, Yaffe MJ, Dworkind M. Physician characteristics and the recognition of depression and anxiety in primary care. Med Care 1994;32:795–812.
440
T. C. Peck et al.
ARTICLE SUMMARY 1. Why is this topic important? The emergency medicine milestones are currently being implemented and will directly affect the training of all future emergency physicians in the United States. However, the milestones have not been validated. 2. What does this study attempt to show? The milestones are constructed so that achievement of Level 4 represents the American Board of Emergency Medicine (ABEM) certification standard. The study asks if currently practicing emergency physicians evaluate themselves to be at Level 4 on selected milestones. If Level 4 accurately represents the ABEM standard, then practicing physicians should be Level 4 or above. 3. What are the key findings? A significant portion of practicing emergency physicians evaluate themselves as not achieving Level 4 on selected milestones. This is especially true when these physicians consider their level of competency in ultrasound skills. 4. How is patient care impacted? There are limited time and resources that can be dedicated to resident education. An accurate and precise method of resident evaluation would improve the appropriation of these resources and could improve quality of education and therefore quality of patient care. Although additional investigation is needed, this study suggests the milestones as written might not be accurate or precise and can benefit from further validation studies.
http://dx.doi.org/10.1016/j.jemermed.2014.04.032
Education PRACTICING EMERGENCY PHYSICIANS REPORT PERFORMING WELL ON MOST EMERGENCY MEDICINE MILESTONES Timothy C. Peck, MD,*† Nicole Dubosh, MD,*† Carlo Rosen, MD,*† Carrie Tibbles, MD,*† Jennifer Pope, MD,‡ and Jonathan Fisher, MD, MPH*† *Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Boston, Massachusetts, †Harvard Medical School, Boston, Massachusetts, and ‡Emergency Medicine Department, St Luke’s Hospital, New Bedford, Massachusetts Reprint Address: Timothy C. Peck, MD, Department of Emergency Medicine, Beth Israel Deaconess Medical Center, One Deaconess Road, West Campus Clinical Center, 2nd Floor, Boston, MA 02215
, Abstract—Background: The Accreditation Council for Graduate Medical Education’s Next Accreditation System endorsed specialty-specific milestones as the foundation of an outcomes-based resident evaluation process. These milestones represent five competency levels (entry level to expert), and graduating residents will be expected to meet Level 4 on all 23 milestones. Limited validation data on these milestones exist. It is unclear if higher levels represent true competencies of practicing emergency medicine (EM) attendings. Objective: Our aim was to examine how practicing EM attendings in academic and community settings selfevaluate on the new EM milestones. Methods: An electronic self-evaluation survey outlining 9 of the 23 EM milestones was sent to a sample of practicing EM attendings in academic and community settings. Attendings were asked to identify which level was appropriate for them. Results: Seventy-nine attendings were surveyed, with an 89% response rate. Sixty-one percent were academic. Twenty-three percent (95% confidence interval [CI] 20% 27%) of all responses were Levels 1, 2, or 3; 38% (95% CI 34% 42%) were Level 4; and 39% (95% CI 35% 43%) were Level 5. Seventy-seven percent of attendings found themselves to be Level 4 or 5 in eight of nine milestones. Only 47% found themselves to be Level 4 or 5 in ultrasound skills (p = 0.0001). Conclusions: Although a majority of EM attendings reported meeting Level 4 milestones, many felt they did not
meet Level 4 criteria. Attendings report less perceived competence in ultrasound skills than other milestones. It is unclear if self-assessments reflect the true competency of practicing attendings. The study design can be useful to define the accuracy, precision, and validity of milestones for any medical field. Ó 2014 Elsevier Inc. , Keywords—graduate medical education; emergency medicine milestones; ACGME; Next Accreditation System; resident evaluation
INTRODUCTION As part of its Next Accreditation System, the Accreditation Council for Graduate Medical Education (ACGME) has recently endorsed and implemented specialty-specific milestones as the foundation of a new outcomes-based resident evaluation process (1). The ACGME has explicitly stated the following intentions behind its actions: basing residency program accreditation on educational outcomes, demonstrating to the public the effectiveness of competence-based education, and changing a system that does not encourage innovation and has become prescriptive (1). Empowered by this directive, the emergency medicine (EM) community completed development of the EM milestones in 2012 after an intensive process (2,3).
This study was exempt from our hospital’s Institutional Review Board.
RECEIVED: 2 October 2013; FINAL SUBMISSION RECEIVED: 30 January 2014; ACCEPTED: 28 April 2014 432
EM Milestone Self-Assessment
The EM milestones will assess resident performance within the following six core competency domains as defined by the ACGME Outcomes Project: Patient Care, Medical Knowledge, Practice-Based Learning and Improvement, Interpersonal and Communication Skills, Professionalism, and Systems-Based Process (4). Each milestone consists of five levels of resident competence ranging from entry level (medical school graduate/ incoming resident) to expert (5). Level 4 is the expected level of competency of an individual at the time of graduation from residency, i.e., the American Board of Emergency Medicine (ABEM) certification standard. In addition, program directors will be expected to file biannual reports on individual resident progress (4). The EM milestones are a substantial achievement, however, the milestones are based on expert consensus and the completed milestones as written have not been studied extensively or validated. For example, although Level 4 is meant to represent the needed competency for a graduating resident to successfully practice EM, it is unclear without the benefit of prospective validation data if the Level 4 milestones as they are currently written truly represent the current skill set of independent practitioners of EM. Our objective was to examine how practicing EM physicians in academic and community settings evaluate their own performance on the new EM milestones. We hypothesized that if Level 4 milestones truly represent a graduating resident’s competency, then nearly all attendings would self-evaluate at Level 4 or 5. METHODS We performed a prospective evaluation of practicing EM physicians in both community and academic settings. The study was conducted from November 1, 2012 through November 24, 2012. A self-evaluation survey was compiled outlining the EM milestones and was sent electronically to a convenience sample of attending physicians at four different institutions, including one academic site and three community sites. The academic institution was a Level I trauma center with an ACGME-accredited EM residency and an annual volume of 57,000 emergency department visits. The three community sites had volumes of 63,000, 62,000, and 25,000 visits per year. A subset of 9 of the 23 milestones was selected and included in the survey. One milestone was selected from each of the six core competencies and three additional milestones were selected due to their interesting characteristics, such as Ultrasound. Demographic information was obtained, including academic vs. community, years of practice, and board certification.
433
Participants were asked to rate themselves on each of the nine milestones using the 1 5 scale outlined by the EM Milestone Task Force (see Figure 1). Data were collected using Surveymonkey (Surveymonkey.com, LLC 2010) and responses were deidentified to ensure that the study investigators were blinded to individual responses. Data were analyzed using Microsoft Excel (2003, Microsoft, Redmond, WA) to calculate proportions and confidence intervals (CIs). Participants were informed by e-mail that participation in this survey was voluntary and that by submitting their responses, they were consenting to use of their deidentified data for research and publication purposes. The project was submitted to the local Institutional Review Board. It was determined that the study met criteria for exemption under federal guidelines. RESULTS Seventy-nine attendings were surveyed, with an 89% response rate. Sixty-one percent were academic. Thirtyfour percent were practicing for 0 5 years. Ninetythree percent graduated EM residency (see Table 1). The study’s power precluded any significant associations between milestone level and years of practice, academic vs. community setting, or EM boarded vs. not. Overall, 77% (95% CI 74% 80%) of all responses by EM physicians were at the Level 4 or 5. Twenty-three percent (95% CI 20% 27%) of all responses were Level 1, 2, or 3, 38% (95% CI 34% 42%) were Level 4, and 39% (95% CI 35% 43%) were Level 5 (see Table 2). Additionally, 77% of attendings found themselves to be Level 4 or 5 in 8 of 9 milestones, and only 47% found themselves to be Level 4 or 5 in Ultrasound (Patient Care [PC] 10) (p = 0.0001) (see Table 2). DISCUSSION By surveying practicing EM physicians, we have learned that a large proportion of attending-level physicians do see themselves as at least Level 4 competency, i.e., the Milestone Task Force’s definition of the expected competency of graduating residents. However, some EM physicians report they do not meet this expected level of competence on at least one milestone. There are several possible explanations for our findings. This might represent a problem with accuracy and precision in which EM attendings might not be able to correctly self-assess their own competency or they might not understand how to use the milestones. Alternatively, there may be issues with external validity of the current incarnation of the milestones.
434
T. C. Peck et al.
Figure 1. Self-assessment survey questions as they appeared on electronic survey. Each question is designed for practicing emergency medicine (EM) physician self-assessment of one of nine selected EM milestones. Question 5 = Patient Care (PC) 3: Diagnostic Studies; Question 6 = PC6: Observation and Reassessment; Question 7 = PC10: Airway Management; Question 8 = PC12: Goal-Directed Focused Ultrasound; Question 9 = Medical Knowledge; Question 10 = Professionalism 1: Professional Values; Question 11 = Interpersonal and Communication Skills 1: Patient-Centered Communication; Question 12 = Practice-Based Learning and Improvement 1: Practice-Based Performance Improvement; Question 13 = Systems Based Process 1: Patient Safety. ED = emergency department, BVM = bag valve mask, echo = echocardiogram, TEE = transesophageal echocardiogram, ABEM = American Board of Emergency Medicine, M&M = morbidity and mortality.
EM Milestone Self-Assessment
Figure 1. (continued).
435
436
Figure 1. (continued).
T. C. Peck et al.
EM Milestone Self-Assessment
437
Table 1. Demographics of Survey Participants Practice Environment
Percent total of the 79 survey participants*
Years Practicing
Residency Training
Academic Practitioner
Community Practitioner
0 5
6 10
>11
Graduated EM Residency
Did Not Graduate from EM Residency
61
39
35
19
47
93
7
EM = emergency medicine. * All survey participants were practicing EM physicians.
Creation of EM Milestones An understanding of the timeline and process that lead to the creation of the milestones is illustrative. The milestone levels were constructed by a thorough and thoughtful process that involved expert consensus, polling large samples of emergency physicians, and the work of various committees (3,6). The ABEM Initial Certification Task Force (ICTF) was created and tasked with studying the initial EM physician certification process and understanding current standards of practice in EM. The ICTF composed relevant and clear definitions of the true tasks of current practicing EM physicians (7). The ABEM Relevance of Examinations to Physician Practice Task Force then recommended changes to the content and methods of ABEM’s examinations to accurately reflect the newly created definitions (8). Working from the new definitions, an ICTF Advisory Panel then further refined the expectations of an individual pursuing initial certification in EM in the form of Knowledge, Skills, and Abilities (KSAs). ABEM then collected survey responses from >7000 EM diplomats, which queried the importance and frequency of each KSA (9,10). Empowered by the KSAs, whose frequency and importance were defined by actual EM practitioners, a Milestone Working Group then ultimately identified 23 Milestones for the ACGME’s six core competencies (i.e., Patient Care, Medical Knowledge, Professionalism, Interpersonal
Communication Skills, Practice-Based Learning and Improvement, and Systems-Base Practice) (11). Drawing off of ABEM’s work on hierarchical scales of performance, each milestone was broken into five competency levels, with the intent of creating uniformity in milestone reporting. The scale ranges from Level 1, representing the expected competency of a medical school graduate to Level 5, representing the competency of an experienced practitioner (5). ABEM and the Council of Emergency Medicine Residency Directors/Clerkship Directors in Emergency Medicine then made recommendations for methods of milestone level assessment. Beginning in July 2013, the products of this comprehensive process were integrated into EM program requirements as part of the Next Accreditation System pilot (6). Accuracy and Precision of the Milestones Level 4 is meant to reflect the ABEM certification standards, however, 23% of responses did not meet the expected Level 4 competency. We find this percentage remarkably high, and it raises a question about the validity of the milestones. Based on these results, it is unclear if the Level 4 milestones in their current form truly reflect the actual skill set needed to practice EM successfully. There may be consequences to inaccurate milestones. If residents are required to meet Level 4 expectations in order to graduate residency, then it follows that residencies
Table 2. Level of Self-Assessment by Practicing Emergency Medicine Physicians on Selected Emergency Medicine Milestones Milestone*
Level 1
Level 2
Level 3
Level 4
Level 5
PC3: Diagnostic Studies PC6: Observation and Reassessment PC10: Airway Management PC12: Goal-Directed Focused Ultrasound MK: Medical Knowledge PROF1: Professional Values ICS1: Patient-Centered Communication PBL1: Practice-Based Performance Improvement SBP1: Patient Safety Overall
0 3 0 7 1 3 4 1 6 3
0 3 4 13 3 0 0 7 10 4
21 16 10 33 3 16 16 23 14 16
30 59 23 46 9 43 43 40 37 38
49 20 63 1 84 39 39 27 33 39
ICS = Interpersonal and Communication Skills; MK = Medical Knowledge; PBL = Practice-Based Learning and Improvement; PROF = Professionalism; SBP = Systems-Based Process; PC = Patient Care. Values are percentages. * At least one milestone was selected from each of the six Accreditation Council for Graduate Medical Education core competencies.
438
will be obligated to move resources and efforts toward assuring that this occurs. The nature of residency programs dictates that trainees are limited by time and resources. There exists a finite amount of training hours during the course of a residency; there are also a finite amount of funds in the budget of each residency. Asking residents to be competent in one skill or proficiency may result in less time dedicated to other proficiencies that a residency deems important. If the minimum competency level is set higher than that necessary to practice effectively, then the inaccuracy of that milestone may unnecessarily pull resources away from other skills and proficiencies that residency programs and residents may find more valuable. This reality becomes more clear when one considers that ACGME residency accreditation may depend on meeting Level 4 milestones. The majority of attendings surveyed did not meet Level 4 on Goal-Directed Focus Ultrasound (PC12). This may reflect the nature of the advancement of this technology and that attendings already practicing may not have the ultrasound competency that the EM community deems to be important for future attending-level physicians. For the ultrasound milestone, attendings may be asked to train their residents to a level higher than their own competency. This presents a challenge, and more resources may need to be diverted toward ultrasound training. Based on our results, a minority of surveyed attendings met Level 5 milestone competency (39%). Level 5 is meant to represent an expert (5). As stated here, the ACGME will be asking residencies to divert their resources toward achieving Level 4 and they are not overtly asking to train residents to Level 5. However, the very existence of Level 5 competencies may have unintended consequences. A stated goal of the ACGME is to hold residencies accountable to the public (1). If the rates of milestone achievements are reported to the public, residencies will surely be pressured to attempt to train their residents to Level 5. Our culture puts value on rankings, and a system that ranks residents on competency level may translate into publically available reportable metrics in which residencies could be judged and tiered—not unlike colleges or hospitals. The practicing EM physician responses also have revealed that not all milestones have a similar per-level distribution of attending self-evaluation. Level 4 on one milestone may not be equivalent to Level 4 on another milestone. As a group, attendings found themselves to be at competency Level 4 or above at widely different rates, depending on which milestone they were asked to consider. Level 4/5 rates ranged from 47% on Ultrasound to 93% on Medical Knowledge. A wide inter-milestone variation will make some milestones more pertinent than others. As the milestones stand now, Level 4 compe-
T. C. Peck et al.
tency in Medical Knowledge may be too easy to achieve, thus devaluing medical knowledge. Ultrasound competency, however, may be difficult to achieve, which may necessitate more attention to ultrasound skills and less attention to assuring our residents have a sound base of medical knowledge. Limitations There are limitations to our study. It is unclear if anonymous self-assessment is truly an accurate representation of actual practicing physician competency. Attending physicians may not be able to accurately or precisely measure their actual skill level. An often-cited review of physician self-assessment literature by Davis et al. concludes that ‘‘while suboptimal in quality, the preponderance of evidence suggests that physicians have a limited ability to accurately self-assess. The processes currently used to undertake professional development and evaluate competence may need to focus more on external assessment’’ (12). Our lack of external assessment is a limitation. However, we find it difficult to extrapolate Davis’ conclusion to our own study. The Davis review was based on only 20 studies that met their rigorous standards for inclusion. In addition, only seven were of retrospective self-analyses that asked the participants to provide mental representations of themselves over time like that of our survey (13 19). Of these seven studies on retrospective self-assessment, four were deemed to suggest a positive correlation of external and self assessment, one suggested inverse correlation, and two yielded inconclusive results (13 19). Although we had an impressive response rate to our survey, our sample size is relatively small and future investigations should aim to include a larger sample of the practicing EM physician population. It is also unclear if some milestones are easier to self-assess than others. In order to maximize the survey response rate, we only asked for data on nine of the milestones. It is quite possible that these 9 are not an accurate representation of all 23 milestones. Therefore, our results may not apply to the milestones as a complete set, and may only relate to the nine that were tested. To obtain Level 5 in three of the milestones we studied, the physician must be in the position to teach the proficiencies within that milestone. It is possible that community physicians in our study do not have the opportunity to teach and would therefore evaluate themselves at a Level 4 even though they are expert in that milestone. There is the chance that attendings did not understand the milestones themselves, although we did reproduce the language of the milestones word for word. Our results must be interpreted in this context. We had a small number of attendings answer that they were Level 1 for some
EM Milestone Self-Assessment
439
of the milestones; this is an unlikely reality, and can skew the statistical analysis of our milestone assessment. 7.
CONCLUSIONS Although a majority of EM attendings who completed our self-assessment survey reported meeting Level 4 milestones, many attendings felt they did not meet Level 4 criteria. Attendings report less perceived competence in ultrasound skills than other surveyed milestones. It is unclear if self-assessments reflect the true competencies of practicing attendings. Reproducing this study with a larger sample size would likely yield valuable data. A well-powered study could clarify and validate the results of our study, which used a convenience sample of 79 physicians. Furthermore, associations may be found between self-assessed milestone level and various metrics, such as physician familiarity with milestones, years of practice, and academic vs. community setting. Finally, this model of study may be useful for fields other than EM to better define the accuracy, precision, and validity of their own milestones. The model may be especially useful to those fields that have yet to create, or are in the process of creating, their milestones.
8.
9.
10.
11.
12.
13.
REFERENCES 1. Nasca TJ. The Next Accreditation System—rational and benefits. N Engl J Med 2012;366:1051–6. 2. Ling LJ, Beeson MS. Milestones in emergency medicine. J Acute Med 2012;2:65–9. 3. Beeson MS, Carter WA, Christopher TA, et al. The development of the emergency medicine milestones. Acad Emerg Med 2013;20: 724–9. 4. Accreditation Council for Graduate Medical Education. Common program requirements. ACGME Approved Standards. Available at: http://www.acgme.org/acgmeweb/Portals/0/PFAssets/Program Requirements/CPRs2013.pdf. September 26, 2010. Accessed June 15, 2014. 5. Accreditation Council for Graduate Medical Education and American Board of Emergency Medicine. The emergency medicine milestones project. ACGME Next Accreditation System: Rationale and Benefits, Milestones. Available at: https://www.acgme. org/acgmeweb/Portals/0/PDFs/Milestones/EmergencyMedicine Milestones.pdf. December 2012. Accessed June 15, 2014. 6. Beeson M. Emergency medicine milestones. 2012 CORD Academic Assembly Schedule/Handouts/Materials, Day 1. Available
14.
15. 16. 17. 18.
19.
at: http://www.cordem.org/files/DOCUMENTLIBRARY/2012% 20AA%20Handouts/1%20Day/CORD%20AA%20milestones% 20(2).ppt. April 10, 2012. Accessed June 15, 2014. Perina DG, Brunett CP, Caro DA, et al. The 2011 model of the clinical practice of emergency medicine. Acad Emerg Med 2012;19: e19–40. American Board of Emergency Medicine. Diplomates to Participate in the Development of Initial Certification. ABEM Memo 2010; Vol 12(1). https://www.abem.org/public/docs/default-source/ publication-documents/2010-spring-newsletter.pdf?sfvrsn=6. Available at: Accessed on June 15th, 2014. Steele MT. Report at the American Academy of Emergency Medicine Scientific Assembly. American Board of Emergency Medicine Reports at National Meetings. Available at: https://www.abem.org/ public/docs/default-source/publication-documents/aaem-2011final56E286AC8033.pdf. October 10, 2011. Accessed June 15, 2014. American Board of Emergency Medicine. Annual Report, 20112012. Available at: https://www.abem.org/public/docs/defaultsource/publication-documents/2010-11-annual-report.pdf. July 27, 2012. Accessed June 15, 2014. The Accreditation Council for Graduate Medical Education and The American Board of Emergency Medicine. The emergency medicine milestone project. American Board of Emergency Medicine Emergency Medicine Milestones website. Available at: https:// www.abem.org/public/docs/default-source/migrated-documentsand-files/em-milestones.pdf?sfvrsn=4. October 22, 2012. Accessed June 15, 2014. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA 2006;296:1094–102. Claridge JA, Calland JF, Chandrasekhara V, Young JS, Sanfey H, Schirmer BD. Comparing resident measurements to attending surgeon self-perceptions of surgical educators. Am J Surg 2003;185: 323–7. Fernandez A, Schillinger D, Grumbach K, et al. Physician language ability and cultural competence. An exploratory study of communication with Spanish-speaking patients. J Gen Intern Med 2004;19: 167–74. Hoppe RB, Farquhar LJ, Henry R, Stoffelmayr B. Residents’ attitudes towards and skills in counseling: using undetected standardized patients. J Gen Intern Med 1990;5:415–20. Ireton HR, Sherman M. Self-ratings of graduating family practice residents’ psychological medicine abilities. Fam Pract Res J 1988; 7:236–44. Johnson D, Cujec B. Comparison of self, nurse, and physician assessment of residents rotating through an intensive care unit. Crit Care Med 1998;26:1811–6. Paradise JE, Finkel MA, Beiser AS, Berenson AB, Greenberg DB, Winter MR. Assessments of girl’s genital findings and the likelihood of sexual abuse: agreement among physicians self-rated as skilled. Arch Pediatr Adolesc Med 1997;151:883–91. Robbins JM, Kirmayer LJ, Cathe´bras P, Yaffe MJ, Dworkind M. Physician characteristics and the recognition of depression and anxiety in primary care. Med Care 1994;32:795–812.
440
T. C. Peck et al.
ARTICLE SUMMARY 1. Why is this topic important? The emergency medicine milestones are currently being implemented and will directly affect the training of all future emergency physicians in the United States. However, the milestones have not been validated. 2. What does this study attempt to show? The milestones are constructed so that achievement of Level 4 represents the American Board of Emergency Medicine (ABEM) certification standard. The study asks if currently practicing emergency physicians evaluate themselves to be at Level 4 on selected milestones. If Level 4 accurately represents the ABEM standard, then practicing physicians should be Level 4 or above. 3. What are the key findings? A significant portion of practicing emergency physicians evaluate themselves as not achieving Level 4 on selected milestones. This is especially true when these physicians consider their level of competency in ultrasound skills. 4. How is patient care impacted? There are limited time and resources that can be dedicated to resident education. An accurate and precise method of resident evaluation would improve the appropriation of these resources and could improve quality of education and therefore quality of patient care. Although additional investigation is needed, this study suggests the milestones as written might not be accurate or precise and can benefit from further validation studies.
