The Journal of Emergency Medicine, Vol. 47, No. 4, pp. 441–452, 2014 Copyright Ó 2014 Elsevier Inc. Printed in the USA. All rights reserved 0736-4679/$ - see front matter

http://dx.doi.org/10.1016/j.jemermed.2014.06.014

Education ENTRUSTABLE PROFESSIONAL ACTIVITIES: MAKING SENSE OF THE EMERGENCY MEDICINE MILESTONES Michael S. Beeson, MD, MBA,* Steven Warrington, MD,* Amber Bradford-Saffles, DO,* and Danielle Hart, MD† *Akron General Medical Center, Akron, Ohio and †Hennepin County Medical Center, Minneapolis, Minnesota Reprint Address: Michael S. Beeson, MD, MBA, Akron General Medical Center, 444 Wabash Ave., Akron, OH 44307

, Abstract—Background: The Next Accreditation System (NAS) is being implemented by the Accreditation Council for Graduate Medical Education with seven specialties, including Emergency Medicine (EM), which began in July 2013. The NAS represents a more structured method of accreditation, with dependence on outcomes and less emphasis on educational process. A key component of the NAS is the individual resident semiannual reporting of the Milestone proficiency levels for all sub-competencies, which are more specific areas of domain for the general competencies. All specialties are struggling to some extent with developing assessment mechanisms for the Milestones. At the heart of this struggle is the conceptualization of the Milestones themselves—descriptors of the individual. In practice, faculty assess clinical care provided to the patient by the resident. This creates difficulty for faculty to assign a resident to a specific sub-competency proficiency level when their focus has been on assessment of clinical care. Objectives: The objectives of this article include the discussion of whether Entrustable Professional Activities (EPAs) could be defined and linked to milestones in a way that, once implemented, could inform Clinical Competency Committees of the Milestone proficiency reporting. Discussion: EPAs are units of professional work, or clinical care that may help translate aspects of clinical care into Milestone proficiencies. This article explores EPAs in depth, and discusses how EPAs may be used within EM as one method of assigning proficiency levels to residents. Conclusions: EPAs may be a useful tool to inform Milestone proficiency placement of residents. Because EPAs are based on clinical descriptions rather than individual physician descriptions,

there may be less faculty development needed for Milestone sub-competency assessment. Ó 2014 Elsevier Inc. , Keywords—Milestones; Entrustable Professional Activities; EPAs; education; resident education

INTRODUCTION The Accreditation Council for Graduate Medical Education (ACGME) has introduced the Next Accreditation System (NAS) with implementation for seven specialties, including Emergency Medicine (EM), which began in July 2013. The rest will follow in July 2014. The core of the NAS involves Milestones, which are stages of development within identified sub-competencies (Figure 1). The definition and specifics of these subcompetencies and milestones are different for each postgraduate medical education residency program. Significant time at major EM educational meetings is now being spent as residencies struggle with how to measure a resident’s proficiency in each of the 23 EM Milestone sub-competencies. Part of this struggle involves recognition that a competency is a descriptor of a physician. The difficulty is in translating a resident’s specific behaviors during patient care into this descriptor of the resident. Entrustable Professional Activities (EPAs) are units of physician practice in which the goal is unsupervised

RECEIVED: 24 June 2013; FINAL SUBMISSION RECEIVED: 31 January 2014; ACCEPTED: 29 June 2014 441

442

M. S. Beeson et al.

Figure 1. Milestones nomenclature.

competent practice by a trainee (1). EPAs consist of descriptors of the tasks or work involved in the practice of medicine. EPAs may provide a means to translate actual practice into the stages of resident development, or Milestones. EPAs, because they are clinically based, may be more readily accepted by residents and faculty as understandable and assessable. This article explores the concept of EPAs, their relationship to the general competencies and EM sub-competencies, and potential methods to measure EPAs and translate them into sub-competency proficiency-level descriptors (Milestones). DEVELOPMENT OF THE NAS AND THE GOAL OF OUTCOMES MEASUREMENT The NAS is a culmination of the Outcomes Project in which the ACGME attempted to change its focus from the educational process and structure to more of an emphasis on outcomes (2,3). The core competencies of patient care, medical knowledge, professionalism, interpersonal communication skills, practice-based learning, and systems-based practice were defined to provide a structure for professional development (4). By themselves there was difficulty in defining specific outcomes within such broad general competencies (5). This created the need for more specific ‘‘Milestones’’ of

competency development, with the Milestones Project announced in 2008. Milestones of competency development in each specialty were defined based on the Dreyfus and Dreyfus model of proficiency acquisition (4,6). Milestone development for each specialty was sponsored by the ACGME and the representative American Board of Medical Specialties Board (7). The EM Milestone Working Group (EMMWG) that developed the EM Milestones was made up of representatives from the ACGME, ACGME Residency Review Committee for EM, American Board of Emergency Medicine (ABEM), and the various organizations within EM (Table 1) (8). The EMMWG benefitted from previous work that ABEM had completed in its own internal Initial Certification Task Force. This task force identified the knowledge, skills, and attitudes (KSAs) necessary for the practice of EM, and critical to the initial certification process. These KSAs were extensions of the physician tasks necessary for the practice of EM, identified in the Model of the Clinical Practice of EM (the Model) (9). The ABEM-identified KSAs were scrutinized through a survey process of diplomates in terms of frequency and perceived importance. The process was identical to the process used to identify the frequency and importance of the Model’s Conditions and Components (10).

EPAs: Making Sense of the EM Milestones Table 1. Contributing Organizations to the EM Milestones The Association of Academic Chairs of Emergency Medicine (AACEM) The American Academy of Emergency Medicine (AAEM) The American Board of Emergency Medicine (ABEM) The American College of Emergency Physicians (ACEP) The Council of Emergency Medicine Residency Directors (CORD) The Emergency Medicine Residents’ Association (EMRA) The Resident Review Committee for Emergency Medicine (RRC-EM) The Society of Academic Emergency Medicine (SAEM) EM = emergency medicine.

The EMMWG identified 23 sub-competencies within the six core competencies, and within each subcompetency, five different levels of proficiency. Each level has one or more Milestones of competency that mark that level of proficiency. Most of the Milestones have as their basis a foundation within the ABEM-identified KSAs. Hence, Milestones are KSAs—measurable knowledge, skills, and attitudes within a sub-competency. As a large component of the NAS, each Milestone sub-competency will have to be reported for each resident at 6-month intervals in terms of level of proficiency. THE DIFFICULTY WITH MEASURING MILESTONES A requirement of the NAS is the reporting of the proficiency level of a resident for each Milestone sub-competency. Each sub-competency will have to be measured in some way to fulfill this requirement. The difficulty lies with how to measure each sub-competency. Although specialty-wide solutions are being sought and discussed, ultimately, each residency will need to decide how it will measure each sub-competency proficiency level for each resident. EM is no different from other specialties in this regard. Where EM is different is in the episodic contact that supervisors have with individual residents. In specialties such as general surgery and internal medicine, it is common to rotate on a service for an entire month, with the same supervisory senior resident(s) and attending(s). A monthly evaluation in terms of that specialty’s Milestones may be appropriate due to working closely together for that period of time. However, in busy emergency departments (EDs) it may be common for a resident to work with different attending staff most of their shifts, or to have overlapping shifts with different attendings the same day. A monthly EM Milestone evaluation in this situation would not be realistic due to the infrequent contact that a resident might have with different attendings. A solution of a resident working primarily with one to four same attendings in a month would facilitate monthly Milestone evaluation, but is not realistic due to the required 24/7 ED coverage demands.

443

The EM Milestones from the EMMWG published on the ABEM Web site include suggested potential methods of evaluation (11). These vary with sub-competency but may include direct observation, simulation, chart review, standardized patients, global ratings, multi-source feedback, and end-of-shift evaluations. End-of-month Milestone evaluations of residents would involve global ratings scales with the Milestones, and would not be specific to direct patient care. This raises the issue of validity: is what is being measured really a reflection of that resident’s sub-competency proficiency level? Issues of cost occur with the use of standardized patients and simulation modalities. Many EM residencies are left trying to develop tools of assessment related to within-shift direct observation. This also raises concern that limited observations within a shift may jeopardize the validity of reported proficiency level. The result is a medical education community trying to assign proficiency level based on limited clinical encounter assessments. The real frustration lies in the fact that competencies are descriptors of physicians (1). Any assessment tools developed to measure sub-competencies or the general competencies must be based on data related to patient care and how the other general competencies interact with patient care. The assessment tool must then extrapolate specific actions that occur during patient care (KSAs) to a descriptor of the physician, their proficiency or competency level. It is not a direct assessment. THE CONCEPT OF EPAS EPAs are entrustable professional activities that are descriptors of the work that physicians do in providing patient care. ‘‘EPA’’ is a relatively recent term first described in 2005 by ten Cate (12). Although the level of granularity of EM EPAs is not yet defined, an example for EM might be the overall evaluation and management of a patient who presents to the ED with pharyngitis. The physician must assess their vital sign stability, obtain a limited history, perform a limited but focused physical examination, and determine management based on clinical impression or additional testing. This is an EPA that most educators would expect an EM resident to be able to perform within their first year of residency. In other words, an entrustable activity in which the resident can be trusted to be able to perform. It may take multiple encounters with patients with pharyngitis to actually be able to consider the entire spectrum of diagnostic possibilities, and hence be able to manage this patient unsupervised. More advanced EPAs might involve the evaluation and management of a middle-aged patient with chest pain. Although a firstyear resident may be able to provide satisfactory care of an individual patient with chest pain, it would likely take more of these patient evaluations to develop proficiency

444

than compared to the pharyngitis EPA. This EPA is obviously of more diagnostic difficulty and risk. Accordingly, this EPA involves Milestone sub-competencies of greater proficiency levels. Each EPA may incorporate multiple Milestone sub-competencies. EPAs bridge the gap between the perception that competencies have lack of applicability to real-world practice and the application of Milestone sub-competencies to patient care (13). EPAs may be significantly more understandable to faculty as well as residents, and therefore easier to evaluate. This contrasts with the difficulty we have observed in faculty understanding how to grade the proficiency level of a resident within a given sub-competency based on limited clinical observations. HOW EPAS ARE RELATED TO MILESTONES Although the EM Milestones are based on many of the ABEM-developed KSAs, they are unique to a specific sub-competency (i.e., PC2: Performance of focused history and physical examination). EPAs are clinical scenarios that incorporate multiple sub-competencies and may require different KSAs from different subcompetencies to complete the patient care activity. For example, to care for a critically ill patient, the trainee must incorporate KSAs from multiple sub-competencies such as PC1: Emergency stabilization; PC2: Performance of focused history and physical examination; PC3: Diagnostic studies; PC4: Diagnosis; PC5: Pharmacotherapy, as well as others. This results in a significant difference when considerations of assessment are made. Currently, clinical encounters must be generalized to assess any subcompetency proficiency level. To assess an EPA, the relevant KSAs are evaluated in the context of whether a clinical scenario was satisfactorily completed. Completion of these KSAs can then be mapped back to specific sub-competencies and proficiency levels afterwards,

M. S. Beeson et al.

without placing that burden on the faculty evaluator. As an example, for the EPA of pharyngitis, one KSA may be whether or not Centor criteria were assessed (14). For a faculty evaluator, this requires little faculty development training other than knowledge of the Centor criteria, and a process to directly observe patient care and evaluate patient presentations. This particular KSA is directly relevant to the EM Milestone sub-competency of Performance of a Focused History and Physical Examination (PC2), and specifically, to the Milestone of ‘‘Performs and communicates a focused history and physical examination that effectively addresses the chief complaint and urgent patient issues.’’ If the faculty member needed to directly evaluate this milestone, they would likely need some training to know how to apply this Milestone to specific clinical cases. The faculty member likely already knows the Centor criteria, can ascertain whether or not the Centor criteria were assessed, and does not necessarily need to know that this would map to a proficiency level of 2 within the Focused History and Physical Examination sub-competency. Without the EPA framework, the faculty evaluator may observe the resident evaluate and manage a patient with pharyngitis, and based on the totality of the patient care observation, answer to what proficiency level within the Focused History and Physical Examination sub-competency a resident should be placed. This likely requires significantly more faculty development and much greater depth of understanding of each of the EM Milestone sub-competencies and their individual Milestones. In this context, KSAs bridge the gap between EPAs and Milestone sub-competencies (Figure 2). EPAs are made up of various KSAs related to Milestone sub-competencies. Once EPA KSAs are mapped to the various Milestone sub-competency proficiency levels, faculty as well as residents can focus on aspects of the clinical care with Milestone proficiency evaluation subsequently being generated and reported.

Figure 2. The relationship between EPAs, KSAs, and Milestones. EPA = Entrustable Professional Activities; KSA = the knowledge, skills, and attitudes; PC = patient care.

EPAs: Making Sense of the EM Milestones

To continue the example of pharyngitis, multiple KSAs could be evaluated. These include: 1. 2. 3. 4.

5. 6.

7.

8. 9. 10. 11. 12. 13. 14. 15.

Introduces self to patient and verifies patient Obtains focused history and physical examination Ascertains allergies Applies Centor criteria or other scoring system to determine need for diagnostic testing and treatment Presents the history and physical examination in an organized manner Can list a differential diagnosis for pharyngitis, including those with potential morbidity and mortality If indicated, is able to choose an appropriate antibiotic for the treatment of streptococcal pharyngitis Can choose an alternative antibiotic based on allergy history Designs an appropriate disposition and follow-up If other patients being evaluated, is able to manage their needs as well Demonstrates empathy towards the patient Ascertains patient’s expectations of ED visit and whether they were fulfilled Can identify areas of weakness in the evaluation and care of the patient with pharyngitis Based on the patient’s presentation of pharyngitis, provides cost-effective care Documents the patient’s visit completely

All of these KSAs are items that can be completed on a checklist sheet directly related to management of the patient with pharyngitis. They may be answered in a binary fashion, yes/no, or in a Likert scale as long as the scale is anchored. The Standardized Direct Observation Tool (SDOT) designed by EM educators uses a three-level scale that has been shown to have excellent reliability (15,16). A three-level scaled question may possibly be mapped to different proficiency levels of the same subcompetency. Once scored, the scores can be translated to specific proficiency levels within different subcompetencies (Table 2). For ease and efficiency of faculty assessments, the EPA assessment could have an initial question of ‘‘Would you entrust this resident to independently provide comprehensive care for a patient presenting with pharyngitis?’’ If the answer is ‘‘yes,’’ no further evaluation would need to be done, because each item on the above checklist could then automatically be reported as ‘‘yes.’’ If the answer is ‘‘no,’’ then that EPA could be ‘‘unpacked’’ to identify areas for improvement (17). The resulting Milestone sub-competency matrix (Table 3) demonstrates that 11 Milestone subcompetencies have been answered to proficiency level. The faculty who scored this pharyngitis case, however,

445

had to know little, if any, of the relationships of Milestone sub-competency proficiency levels with rating the encounter with a patient with pharyngitis. It is likely that a resident will be entrusted to evaluate a patient with pharyngitis at an earlier stage in training compared to a patient with chest pain. To evaluate a patient with chest pain requires greater proficiency levels among the sub-competencies. Due to the greater complexity and higher risk, it will take more resident encounters with patients having chest pain than patients with pharyngitis to be entrusted with unsupervised care. To evaluate and manage a patient with chest pain requires more advanced KSAs than a patient with pharyngitis. Potential KSAs in the evaluation and management of a patient with chest pain include: 1. Introduces self to patient and verifies patient 2. Obtains key information from history and physical examination: a. Pertinent history b. Past cardiac history c. Past medical history including risk factors d. Cardiac and pulmonary auscultation e. Evaluation of JVD [jugular venous distention], calf tenderness, pedal edema 3. Ascertains allergies 4. Orders appropriate diagnostic studies for chest pain 5. Obtains data from multiple sources to care for the patient 6. Applies TIMI [Thrombolysis in Myocardial Infarction] score criteria or other scoring system for risk assessment 7. Presents the history and physical examination in an organized manner 8. Can list a differential diagnosis for chest pain, including those with potential morbidity and mortality 9. Administers aspirin to the patient unless not indicated 10. Efficiently provides appropriate medications for the patient with chest pain 11. Re-evaluates patient and vital signs 12. Designs an appropriate disposition and follow-up 13. If other patients are concurrently being evaluated, is able to manage their needs as well 14. If other patients are concurrently being evaluated, is able to prioritize patient care and disposition 15. Demonstrates empathy toward the patient 16. Communicates the patient’s diagnosis in a way to alleviate stress 17. Manages the patient with chest pain as a team member or leader 18. Ascertains patient’s expectations of ED visit and whether they were fulfilled

446

M. S. Beeson et al.

Table 2. Mapping of Pharyngitis KSAs to Sub-competency Proficiency Levels Pharyngitis EPA KSA

Milestone Sub-competency

Introduces self to patient and verifies Patient-Centered Communication patient (ICS1) Obtains focused history and physical Performance of Focused History and examination Physical Examination (PC2)

Ascertains allergies

Pharmacotherapy (PC5)

Applies Centor criteria or other scoring system to determine need for diagnostic testing and treatment

Emergency Stabilization (PC1)

Diagnostic Studies (PC3) Presents the history and physical examination in an organized manner

Performance of Focused History and Physical Examination (PC2)

Can list a differential diagnosis for pharyngitis, including those with potential morbidity and mortality If indicated, is able to choose an appropriate antibiotic for the treatment of streptococcal pharyngitis Can choose an alternative antibiotic based on allergy history

Diagnosis (PC4)

Designs an appropriate disposition and follow-up

Disposition (PC7)

Pharmacotherapy (PC5)

Pharmacotherapy (PC5)

If other patients being evaluated, is Multi-tasking (Task-switching) (PC8) able to manage their needs as well Demonstrates empathy towards the Professional values (PROF1) patient

Ascertains patient’s expectations of Patient Centered Communication ED visit and whether they were (ICS1) fulfilled Can identify areas of weakness in the Practice-based Performance evaluation and care of the patient Improvement (PBLI) with pharyngitis Based on the patient’s presentation of pharyngitis provides cost- effective care Documents the patient’s visit completely

Systems-based Management (SBP2) Technology (SBP3)

Milestone(s) Affected

Proficiency Level

Establishes rapport with and demonstrates empathy toward patients and their families Performs and communicates a focused history and physical examination that effectively addresses the chief complaint and urgent patient issues Consistently asks patients for drug allergies Discerns relevant data to formulate a diagnostic impression and plan

1

Determines the necessity of diagnostic studies; or Orders appropriate diagnostic studies Performs and communicates a focused history and physical examination that effectively addresses the chief complaint and urgent patient issues Constructs a list of potential diagnoses with the greatest potential for morbidity or mortality Applies medical knowledge for selection of appropriate agent for therapeutic intervention

1 or 2

Considers array of drug therapy for treatment. Selects appropriate agent based on mechanism of action, intended effect, and anticipates adverse side effects Formulates a specific follow-up plan for common ED complaints with appropriate resource utilization Task switches between different patients Demonstrates behavior that conveys caring, honesty, genuine interest and tolerance when interacting with a diverse population of patients and families Elicits patients’ reasons for seeking health care and expectations from the ED visit Performs self-assessment to identify areas for continued selfimprovement and implements learning plans Practices cost-effective care

3

Ensures that medical records are complete, with attention to preventing confusion and error

2

1 2

2

2 2

2 2 1

2 3

3 2

ED = emergency department; EPA = Entrustable Professional Activities; KSA = knowledge, skills, and attitudes; ICS = interpersonal communications skills; PC = Patient Care; PROF = professionalism; PBLI = Practice-based Learning and Improvement; SBP = Systems-based Practice.

19. Can identify areas of weakness in the evaluation and care of the patient with chest pain 20. Based on the patient’s presentation of chest pain provides cost-effective care 21. Documents the patient’s visit completely

These KSAs for chest pain, although arbitrarily chosen, map to 14 different sub-competencies (Table 4), and to proficiency levels generally higher than that for pharyngitis (Table 3). KSAs can be written to a high degree of specificity, such as, ‘‘Were IIB/IIIA inhibitors

EPAs: Making Sense of the EM Milestones

447

Table 3. Matrix of Scoring Pharyngitis and Chest Pain EPAs Sub-competency

PC1

Results of pharyngitis EPA scoring (satisfactory care provided) Proficiency level 2 Sub-competency PROF1 Proficiency level 1 Results of chest pain EPA Scoring (satisfactory care provided) Proficiency level 2, 3 Sub-competency PROF1 Proficiency level 1 or 2

PC2

PC3

2, 2 PROF2

ICS1 1, 2

PC4

PC5

2 ICS2

1, 2, 3 PBLI 3

PC6

PC7

2 SBP1 SBP2 3

PC8

2 SBP3 2

2, 2 or 3, 4 2 or 3 2 or 3 1, 2, 2 or 3 3 2 or 3 2 or 3, 3 PROF2 ICS1 ICS2 PBLI SBP1 SBP2 SBP3 1, 3 1, 2, or 3 3 3 2

EPA = Entrustable Professional Activities; PC = Patient Care; PROF = professionalism; ICS = interpersonal communications skills; PBLI = Practice-based Learning and Improvement; SBP = Systems-based Practice.

appropriately and efficiently ordered in the patient with STEMI [ST-segment elevation myocardial infarction]?’’ However, this can result in a large number of items in a checklist sheet, resulting in potentially less reliable scoring from faculty as well as compliance. The important aspect is that the KSAs describe actions directly related to a patient’s care. This facilitates faculty evaluation of the resident. The mapping to Milestone sub-competencies and proficiency levels can then be transparent to the faculty, and occurs when the checklist sheet data are entered into a database that maps these questions to specific subcompetencies and proficiency levels. For several of the KSAs for chest pain, using the SDOT schema of scoring (needs improvement, meets expectations, and above expectations) the different levels can be anchored to specific proficiency levels. As an example, one of the questions for chest pain is ‘‘Orders appropriate diagnostic studies for chest pain.’’ ‘‘Meets expectations’’ could be found with simply ordering the appropriate diagnostic studies for a patient with chest pain, level 2 proficiency level. However, ‘‘above expectations’’ could be anchored to the resident who prioritizes seeing the electrocardiogram soon after patient arrival, or the level 3 Milestone of ‘‘Prioritizes essential testing.’’ For EPAs to be successful in describing Milestone progression, multiple EPAs need to be designed. Ten Cate suggests that no more than 20 to 30 be developed (1). EM is a specialty where the patient presents typically in an undifferentiated way, with signs or symptoms as their chief complaint. The Model of the Clinical Practice of Emergency Medicine has Signs, Symptoms, and Presentations as the first conditions and components listed. Additionally, the frequency and importance of each of these as determined by ABEM diplomates is known (9,10). EPAs can be designed with varying levels of complexity that span the levels of proficiency of the Milestone sub-competencies. As a resident progresses through residency, EPA checklist sheets are completed by faculty, contributing to the ultimate proficiency-level reporting by the residency for that resident to the

ACGME. Depending on the EPA, many more of one may need to be completed, compared to others of lesser complexity. Again, the contrast between pharyngitis with a limited number of causes and occasional morbidity/mortality with chest pain, which has many causes, and frequent morbidity/mortality highlights these differences. One EPA may need 20 presentations for the resident to be ‘‘entrusted’’ to care for a patient with that presentation. Another EPA may require 100 for the resident to be proficient and trusted to manage well. For a system of evaluation to be successful, multiple EPAs of varying complexity and related to different Milestone sub-competencies need to be developed. It is not yet clear how specific, granular, or detailed EPAs in EM should be, but an example EPA list might include: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23.

Altered mental status Back pain Headache Multiple trauma Abdominal pain Pelvic pain Vaginal bleeding Chest pain Shortness of breath Cough Eye disorders ENT [ear, nose, and throat] complaints Rash Ankle/wrist injury Pediatric respiratory distress Pediatric dehydration Depression The critical patient (by vital signs, respiratory failure, etc.) Delivering bad news Difficult patient/family/consultant interaction General procedures Airway management Wound care

448

M. S. Beeson et al.

Table 4. Mapping of Chest Pain KSAs to Sub-Competency Proficiency Levels Chest Pain EPA KSA

Milestone Sub-competency

Applies TIMI score criteria or other Emergency Stabilization (PC1) scoring system for risk assessment If other patients being evaluated, is Emergency Stabilization (PC1) able to prioritize patient care and disposition Obtains key information from history Performance of Focused History and and physical examination Physical Examination (PC2)

Presents the history and physical examination in an organized manner

Performance of Focused History and Physical Examination (PC2)

Obtains data from multiple sources to Performance of Focused History and care for the patient Physical Examination (PC2) Orders appropriate diagnostic studies for chest pain

Diagnostic Studies (PC3)

Can list a differential diagnosis for chest pain, including those with potential morbidity and mortality

Diagnosis (PC4)

Ascertains allergies

Pharmacotherapy (PC5)

Administers baby aspirin to the patient unless not indicated

Pharmacotherapy (PC5)

Efficiently provides appropriate medications for the patient with chest pain

Pharmacotherapy (PC5)

Re-evaluates patient and vital signs

Observation and Reassessment (PC6)

Designs an appropriate disposition and follow-up

Disposition (PC7)

If other patients being evaluated, is Multi-tasking (Task-switching) (PC8) able to manage their needs as well

Milestone(s) Affected

Proficiency Level

Discerns relevant data to formulate a diagnostic impression and plan Manages and prioritizes critically ill or injured patients

2

Performs and communicates a focused history and physical examination that effectively addresses the chief complaint and urgent patient issues; or Prioritizes essential components of a history/ physical examination given a limited or dynamic circumstance Performs and communicates a focused history and physical examination that effectively addresses the chief complaint and urgent patient issues Synthesizes essential data necessary for the correct management of patients using all potential sources of data Orders appropriate diagnostic studies; or Prioritizes essential testing Constructs a list of potential diagnoses with the greatest potential for morbidity or mortality; or Uses all available medical information to develop a list of ranked differential diagnoses including those with the greatest potential for morbidity or mortality Consistently asks patients for drug allergies Applies medical knowledge for selection of appropriate agent for therapeutic intervention Applies medical knowledge for selection of appropriate agent for therapeutic intervention; or Considers array of drug therapy for treatment. Selects appropriate agent based on mechanism of action, intended effect, and anticipates potential adverse side effects Monitors a patient’s clinical status at timely intervals during their stay in the ED Formulates a specific follow-up plan for common ED complaints with appropriate resource utilization; or Formulates and provides patient education regarding diagnosis, treatment plan, medication review and PCP/consultant appointments for complicated patients Task switches between different patients; or Employs task switching in an efficient and timely manner to manage multiple patients

2 or 3

3

2

4

2 or 3 2 or 3

1 2 2 or 3

3 2 or 3

2 or 3

(Continued )

EPAs: Making Sense of the EM Milestones

449

Table 4. Continued Chest Pain EPA KSA

Milestone Sub-competency

If other patients being evaluated, is able to prioritize patient care and disposition Demonstrates empathy towards the patient

Multi-tasking (Task-switching) (PC8)

Milestone(s) Affected

Employs task switching in an efficient and timely manner to manage multiple patients Professional values (PROF1) Demonstrates behavior that conveys caring, honesty, genuine interest and tolerance when interacting with a diverse population of patients and families; or Demonstrates an understanding of the importance of compassion, integrity, respect, sensitivity and responsiveness and exhibits these attitudes consistently in common/ uncomplicated situations and with diverse populations Introduces self to patient and verifies Patient Centered Communication Establishes rapport with and patient (ICS1) demonstrates empathy toward patients and their families Communicates the patient’s Patient Centered Communication Manages the expectations of those diagnosis in a way to alleviate (ICS1) who receive care in the ED and stress uses communication methods that minimize the potential for stress, conflict, and misunderstanding Manages the patient with chest pain Team Management (ICS2) Participates as a member of a patient as a team member or leader care team; or Communicates pertinent information to emergency physicians and other healthcare colleagues; or Ensures transitions of care are accurately and efficiently communicated Can identify areas of weakness in the Practice-based Performance Performs self-assessment to identify evaluation and care of the patient Improvement (PBLI) areas for continued selfwith chest pain improvement and implements learning plans; Demonstrates the ability to critically appraise scientific literature and apply evidence-based medicine to improve one’s individual performance Based on the patient’s presentation Systems-based Management (SBP2) Practices cost-effective care; of chest pain provides costDemonstrates the ability to call effective care effectively on other resources in the system to provide optimal health care Documents the patient’s visit Technology (SBP3) Ensures that medical records are completely complete, with attention to preventing confusion and error

Proficiency Level 3 1 or 2

1 3

1, 2, or 3

3

3

2

EPA = Entrustable Professional Activities; KSA = knowledge, skills, and attitudes; TIMI = Thrombolysis in Myocardial Infarction; PC = Patient Care; ED = emergency department; PROF = professionalism; ICS = interpersonal communications skills; PBLI = Practicebased Learning and Improvement; PCP = primary care provider; SBP = Systems-based Practice.

24. Ultrasound POC [point of care] 25. Vascular access 26. Anesthesia and pain management This particular list was arbitrarily chosen as an example list, based on authors’ experience and corresponding to high frequency or high importance presentations or procedures (10). These EPAs span a continuum of low complexity, low risk to high complexity and high risk. These also span different proficiency levels of different sub-competencies.

DISCUSSION Competency-based education and assessment is a relatively new term that has evolved since the Flexner report was published in 1910, in which university-based medical education was viewed as essential to quality medical education (18). At the time, and leading into the late 20th century, it was expected that faculty could recognize good clinicians. Formal assessment was primarily based on medical knowledge examinations. In 1990, Miller

450

published the landmark article describing the framework of medical knowledge (19). At the base is ‘‘knows’’ (knowledge), followed by ‘‘knows how’’ (competence), then ‘‘shows how’’ (performance), with ‘‘does’’ (action) at the top of the pyramid. This created the realization that demonstration of knowledge itself was insufficient for the practice of medicine. Assessment based primarily on knowledge acquisition did not address all of the steps required to ‘‘show how’’ or ‘‘does.’’ These skills have become known as competencies. Occurring at the same time was the desire for more humanistic qualities to be instilled into physicians. Medical schools and specialty residencies responded in various degrees to teach these higher-order competencies. For graduate medical education, the ACGME responded by mandating the education and assessment of the general competencies (2). It is now recognized that the general competencies were vague, and difficult to define in a clinical context as well as assess. To focus more on outcomes, each specialty was asked to develop Milestones of progression through various specialty-defined sub-competencies of the general competencies. For EM, this was done by consensus of an expert group representing all of the stakeholder organizations in EM (7). The EM Milestones are based on substantive data from ABEM, derived from their work toward identifying initial certification standards (20). Specialty Milestones are complex attributes of individuals caring for patients. They are descriptors of the physician. When residents care for patients, they are assessed in terms of the care provided to that patient. Although a subtle difference, it results in a difficult task for the assessing physician to cross-interpret patient care into the context of a descriptor of the physician. The Milestones have greater specificity than the general competencies. However, there is still difficulty in applying a specific proficiency level of a sub-competency to a specific action taken in the course of patient care. Lurie describes this difficulty as socially constructed ideas (competencies) mistaken for objective phenomena (21). What is needed is a bridge from clinical patient care assessment to assignment to proficiency levels of the various sub-competencies. EPAs are used by different specialties to help determine when a resident may practice that patient care activity unsupervised (entrustment). EPAs for use in EM postgraduate medical education have not yet been defined. However, other specialties have begun investigating and describing EPAs. Boyce et al. surveyed the Fellows of the Royal Australian and New Zealand College of Psychiatrists (RANZCP) to begin to develop EPAs for their specialty (17). They described how, if a supervisor was unable to entrust a trainee to do a certain EPA, they would then have the supervisor ‘‘unpack’’ the EPA, evaluating the specific competencies and KSAs

M. S. Beeson et al.

involved in that EPA to identify the areas for improvement or remediation (17). Additionally, it was found that the use of EPAs in psychiatry, specifically for electroconvulsive therapy, was well aligned with the current requirements for training and credentialing this skill (22). Within internal medicine (IM), EPAs for both general IM training and the medical home have been developed (23,24). Subsequently, a feasibility study with two of the developed IM EPAs was conducted; most residents and attendings felt that use of these EPAs improved education or feedback on the topic being assessed (25). Family medicine has developed a list of EPAs, however, this list has 73 EPAs included, which is well beyond the 20–30 EPAs per specialty recommended by ten Cate (1,26). Pediatrics has also begun to develop EPAs to evaluate their trainees in various aspects of pediatric training (27). Other health care professions, such as physician assistant training programs, have started to develop EPAs for their trainees as well (28). EM is different from other specialties in that there is 24-hour supervision of patients, and due to the reimbursement process, all patients are also evaluated by faculty. EPAs could be used for EM education in determining the degree of supervision provided to a resident. As an example, the intern who has seen 15 patients with pharyngitis will likely need less direct oversight than the incoming intern who has limited patient care experiences with pharyngitis. EPAs may be extremely valuable in EM resident education. They are clinically oriented, focused on the tasks required to complete care. Faculty can more easily evaluate whether different items related to patient care are accomplished. It has been shown that specific patient care actions can be mapped onto related general competencies (29,30). Both the resident and faculty can understand if the resident did or did not do something needed in patient care. Milestones, on the other hand, describe steps along the progression of resident maturation within the general competencies and subcompetencies. Faculty must extrapolate the care given to a particular patient to the proficiency level of a resident. Milestones, as descriptors of residents, are not necessarily descriptors of patient care. EPAs bridge this gap, allowing for the translation of patient care activities to Milestone proficiencies. This is much more understandable to faculty and residents, and requires less faculty training in the evaluation process. For a residency to narrow down the many potential signs and symptoms and conditions to between 20 and 30 EPAs, expert panels within EM or the individual residencies need to define the EPAs that will be developed. These expert panels can be aided by the Model, as well as the frequency and importance of the different conditions and components within the Model. EPAs can be

EPAs: Making Sense of the EM Milestones

designed that answer some of the difficult Milestone proficiency questions, such as interactions with difficult patients, delivering bad news, etc. There are six Milestone sub-competencies that are focused on procedures. Each of these can have EPAs developed specifically for them, using checklist sheets for the various components of that procedure that map to the specific proficiency levels of that procedure’s sub-competency. Just as a resident as well as faculty member can quickly understand the pertinence of individual items in a central line checklist, the same is true for checklists focused on the evaluation and management of patients with specific presentations. This is far different than a resident and faculty trying to understand the applicability of Milestone proficiency levels of a subcompetency to specific care. Once EPAs are designed with their checklists, they can be used with direct patient care, or with other simulation platforms, such as standardized patients, mannequins, and hybrid models (the ABEM Oral Examination process). EPAs then become an efficient tool for training as well as evaluation. Consequently, residents and faculty can concentrate on clinical education, rather than on artificially trying to describe a resident in terms of proficiency level of a sub-competency. EPAs will not solve the requirement of Milestone subcompetency proficiency-level reporting. However, the development and use of EPAs may bridge the gap of understanding how Milestones apply to resident education. Proficiency-level reporting is expected to be based on objective data (3). The use of EPAs is but one possible way of introducing objective Milestone evaluation of residents. It is not clear whether EPAs are more appropriate for lower proficiency levels, or whether they can equally assess higher-level proficiencies. Using EPAs to assess individual patient care management may not be valid for all presentations. For training and formative assessment purposes, the use of EPAs in simulated modality environments may not be able to assess higher-level proficiency levels due to the lack of cues that are usually present with real direct patient care. It may be that EPAs are most useful in defining the lower levels of proficiencies early in a resident’s overall development. Higher levels of proficiency may only be gained with many more patient encounters that challenge a resident’s comfort level within a subcompetency. As an example, the Emergency Stabilization (PC1) sub-competency has ‘‘Recognizes in a timely fashion when further clinical intervention is futile’’ as well as ‘‘Integrates hospital support services into a management strategy for a problematic stabilization situation’’ as Level 4 Milestones. Both of these can be achieved only with experience and many encounters

451

with routine patients to gain the experience with the few more difficult patients. This concept is similar to the ‘‘deliberate practice’’ model described by Ericsson To become proficient in a skill, deliberate practice with multiple encounters is necessary (31). Finally, it is not clear that using EPAs will adequately demonstrate resident progression through the subcompetencies. Investigation is needed to demonstrate that EPAs can demonstrate competency progression and the artificial mapping of clinical care checklist items to specific sub-competency proficiency levels as a viable assessment method. FUTURE DIRECTIONS EPAs represent a bridge between patient care and Milestone proficiency-level determination. To move in this direction, individual residencies will need to determine their EPAs and develop databases that support EPA data reporting. A far more efficient mechanism could be a specialty-wide approach. A specialty-wide approach could focus on development in these areas: 1. Identification of EPAs by expert group consensus 2. For each EPA, determine clinically relevant items to score 3. Map each EPA’s items to sub-competency and proficiency levels 4. Determine scoring mechanism for each item (binary (Yes/No), SDOT-style, Likert) that aligns with proficiency levels 5. Development of database structure for data collection that translate EPA checklist data into proficiency level 6. Pilot testing of individual EPA scenarios 7. Engagement of residency management software vendors to incorporate the designed database structure 8. Large scale deployment of the EPA project Much research needs to be done on validity and reliability of different assessment tools developed for Milestone proficiency determination. The use of EPAs as an assessment tool is no different. A specialty-wide approach would efficiently leverage the resources needed to make a significant project like this viable. CONCLUSION EPAs are units of professional practice aligned with clinical care. EPAs have the potential to translate elements of clinical practice into proficiency levels of the Milestone sub-competencies. As EPAs are clinically related, it may be far easier for faculty to assess clinical practice, compared to Milestones. Once scored, the EPAs flow

452

M. S. Beeson et al.

into Milestone proficiency scoring, transparent to the faculty and resident. Much work needs to be done to test the feasibility of EPA use in this way. The potential benefits include less time with faculty training, potentially greater reliability of assessment instruments, and a more understandable assessment to the resident. EPAs may be the vehicle that makes sense of the Milestones.

16.

17. 18.

REFERENCES 1. ten Cate O. Nuts and bolts of entrustable professional activities. J Grad Med Educ 2013;5:157–8. 2. Swing S. ACGME launches outcomes assessment project. JAMA 1998;279:1492. 3. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system- rationale and benefits. N Engl J Med 2012;366:1051–6. 4. Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S. General competencies and accreditation in graduate medical education. Health Aff (Millwood) 2002;21:103–11. 5. Nasca TJ. The CEO’s first column—the next step in the outcomesbased accreditation project. ACGME Bull 2008;2–4. 6. Dreyfus HL, Dreyfus SE. Five steps from novice to expert. In: Mind over machine. New York, NY: Free Press; 1988:16–51. 7. Swing SR, Beeson MS, Carraccio C, et al. Educational milestone development in the first 7 specialties to enter the next accreditation system. J Grad Med Educ 2013;5:98–106. 8. Beeson MS, Carter WA, Christopher TA, et al. Emergency medicine milestones. J Grad Med Educ 2013;5:5–13. 9. Perina DG, Beeson MS, Char DM, et al. The 2007 model of the clinical practice of emergency medicine: the 2009 update. Acad Emerg Med 2011;18:e8–26. 10. Beeson MS, Marco CA, Nelson RN, Korte RC. Components of the model of the clinical practice of emergency medicine: a survey of American Board of Emergency Medicine diplomates. J Emerg Med 2013;44:1153–66. 11. Accreditation Council of Graduate Medical Education. The emergency medicine milestone project. 2012. Available at: https:// www.abem.org/public/docs/default-source/migrated-documentsand-files/em-milestones.pdf?sfvrsn=4. Accessed January 31, 2014. 12. ten Cate O. Entrustability of professional activities and competency-based training. Med Educ 2005;39:1176–7. 13. Carraccio C, Burke AE. Beyond competencies and milestones: adding meaning through context. J Grad Med Educ 2010;2:419–22. 14. Centor RM, Witherspoon JM, Dalton HP, Brody CE, Link K. The diagnosis of strep throat in adults in the emergency room. Med Decis Making 1981;1:239–46. 15. Shayne P, Gallahue F, Rinnert S, Anderson CL, Hern G, Katz E. Reliability of a core competency checklist assessment in the emer-

19. 20. 21. 22. 23. 24.

25.

26. 27. 28.

29. 30.

31.

gency department: the standardized direct observation assessment tool. Acad Emerg Med 2006;13:727–32. LaMantia J, Kane B, Yarris L, et al. Real-time inter-rater reliability of the council of emergency medicine residency directors standardized direct observation assessment tool. Acad Emerg Med 2009;16: S51–7. Boyce P, Spratt C, Davies M, McEvoy P. Using entrustable professional activities to guide curriculum development in psychiatry training. BMC Med Educ 2011;11:96–104. Kirch DG. Commentary: the Flexnerian Legacy in the 21st century. Acad Med 2010;85:190–2. Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65:S63–7. Ling LJ, Beeson MS. Milestones in emergency medicine. J Acute Med 2012;2:65–9. Lurie SJ. History and practice of competency-based assessment. Med Educ 2012;46:49–57. Port N, Weiss A, Maudsley I. Electroconvulsive therapy training: can it be a model of an entrustable professional activity in a competency program? Australas Psych 2012;20:242–5. Hauer KE, Kohlwes J, Cornett P, et al. Identifying entrustable professional activities in internal medicine training. J Grad Med Educ 2013;5:54–9. Chang A, Bowen JL, Buranosky RA, et al. Transforming primary care training—patient-centered medical home entrustable professional activities for internal medicine residents. J Gen Intern Med 2012;28:801–9. Hauer KE, Soni K, Cornett P, et al. Developing entrustable professional activities as the basis for assessment of competence in an internal medicine residency: a feasibility study. J Gen Intern Med 2013;28:1110–4. Shaughnessy AF, Sparks J, Cohen-Osher M, Goodell KH, Sawin GL, Gravel J. Entrustable professional activities in family medicine. J Grad Med Educ 2013;5:112–8. O’Keeffe M. Clinical competency in developmentalbehavioural paediatrics: raising the bar. J Paediatr Child Health 2014;50:3–10. Mulder H, Ten Cate O, Rieneke D, Berkvens J. Building a competency based workplace curriculum around entrustable professional activities: the case of physician assistant training. Med Teach 2010;32:e453–9. Johnston KC. Responding to the ACGME’s competency requirements: an innovative instrument from the University of Virginia’s neurology residency. Acad Med 2003;78:1217–20. Chapman DM, Hayden S, Sanders AB, et al. Integrating the accreditation council for graduate medical education core competencies into the model of the clinical practice of emergency medicine. Ann Emerg Med 2004;43:756–69. Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med 2004;79:S70–81.