The Journal of Emergency Medicine, Vol. 46, No. 4, pp. 537–543, 2014 Copyright Ó 2014 Elsevier Inc. Printed in the USA. All rights reserved 0736-4679/$ - see front matter
http://dx.doi.org/10.1016/j.jemermed.2013.08.119
Education THE MULTIPLE MINI-INTERVIEW FOR EMERGENCY MEDICINE RESIDENT SELECTION Laura R. Hopson, MD,* John C. Burkhardt, MD, MA,*† R. Brent Stansfield, PHD,† Taher Vohra, MD,‡ Danielle Turner-Lawrence, MD,§ and Eve D. Losman, MD* *Department of Emergency Medicine, University of Michigan, Ann Arbor, Michigan, †Department of Medical Education, University of Michigan, Ann Arbor, Michigan, ‡Department of Emergency Medicine, Henry Ford Hospital, Detroit, Michigan, and §Department of Emergency Medicine, Oakland University, William Beaumont School of Medicine, Rochester, Michigan Reprint Address: Laura R. Hopson, MD, Department of Emergency Medicine, University of Michigan, 1500 E. Medical Center Drive, TC B1 382, Ann Arbor, MI 48109-5305
, Abstract—Background: The Multiple Mini-Interview (MMI) uses multiple, short-structured contacts to evaluate communication and professionalism. It predicts medical school success better than the traditional interview and application. Its acceptability and utility in emergency medicine (EM) residency selection are unknown. Objective: We theorized that participants would judge the MMI equal to a traditional unstructured interview and it would provide new information for candidate assessment. Methods: Seventyone interns from 3 programs in the first month of training completed an eight-station MMI focused on EM topics. Pre- and post-surveys assessed reactions. MMI scores were compared with application data. Results: EM grades correlated with MMI performance (F[1, 66] = 4.18; p < 0.05) with honors students having higher scores. Higher third-year clerkship grades were associated with higher MMI performance, although this was not statistically significant. MMI performance did not correlate with match desirability and did not predict most other components of an application. There was a correlation between lower MMI scores and lower global ranking on the Standardized Letter of Recommendation. Participants preferred a traditional interview (mean difference = 1.36; p < 0.01). A mixed format (traditional interview and MMI) was preferred over a MMI alone (mean difference = 1.1; p < 0.01). MMI performance did not significantly correlate with preference for the MMI. Conclusions: Although the MMI alone was viewed less favorably than a traditional interview, participants were receptive to
a mixed-methods interview. The MMI does correlate with performance on the EM clerkship and therefore can measure important abilities for EM success. Future work will determine whether MMI performance predicts residency performance. Ó 2014 Elsevier Inc. , Keywords—graduate medical education; residency selection; Multiple Mini-Interview; professionalism; communication
INTRODUCTION Selection of appropriate residency candidates is a highstakes process for both the training program and applicant. In medical school admissions, traditional measures of applicant quality, such as interviews, medical college admission tests, and grade-point average, have fared poorly in predicting medical school performance (1 4). Current literature in emergency medicine (EM) suggests that the best predictors of residency performance include clerkship grades and quality of medical school attended (5). However, research at a Canadian Family Medicine residency suggests that although interviews and application files can correctly predict outstanding residents, they are insensitive for identifying potential ‘‘problem’’ residents (4).
RECEIVED: 27 January 2013; FINAL SUBMISSION RECEIVED: 23 July 2013; ACCEPTED: 23 August 2013 537
538
L. R. Hopson et al.
To address the limitations of the traditional application process, Eva et al. developed the Multiple Mini-Interview (MMI), which uses an objective structured clinical examination format to have multiple short, structured contacts with an applicant and improves assessment of the applicant’s communication and problem-solving skills (6,7). Performance on the MMI correlates with preclinical grades and performance on clerkships (8,9). There is also some suggestion that the MMI might be able to pick out candidates with professionalism issues, who might not otherwise be detected until later in their training (8,9). The MMI format has been used successfully at medical schools throughout the world, and is becoming increasingly prevalent at United States (US) medical schools (7,10,11). The residency selection process is a different environment than medical school admissions. A more rigorous interview technique might adversely affect a program’s match outcomes. The few reports on the MMI in the postgraduate environment have not identified significant impediments to utilizing this method on the part of the participants (12 15). However, there are significant limitations to extrapolating to an EM residency within the US, as they either do not include participants from
the specialty or include substantial numbers of international medical graduates, whose motivations and competitiveness for a residency slot might differ from US graduates. In addition, it has not been demonstrated that the MMI provides additional useful information about the applicant in the post-graduate realm beyond that provided in a traditional interview and residency application package. The objective of this study was to determine the perspectives of participants on the MMI in a simulated EM interview situation. A secondary objective was to determine if the MMI would provide new information to assess residency candidates. METHODS Study Design An eight-station MMI was created by the authors using information published in the literature (6,16). The scenarios were drawn directly from published examples or created de novo based on characteristics believed to be desirable in EM trainees (Table 1). In their first month of training, EM interns were used as surrogates for residency applicants. As prior knowledge of the stations
Table 1. Examples of Multiple-Mini Interview Stations (6)* Station Description Give and receive instructions (two stations) for an origami paper-folding activity using only verbal means
Disclosure of damaging a borrower computer (role-playing)
Discussing a patient’s condition with a parent while respecting confidentiality (role-playing)
Giving verbal instruction to hang pictures
Discussing decision-making process whether or not to recommend testing for a genetic condition
Deciding which tasks to complete given a limited amount of time and explaining decision-making process
Desired Characteristic
ACGME Competency
Adaptability Hard working Problem solving Strong communication skills Teamwork Altruism Ethical Problem solving Strong communication skills Adaptability Altruism Aware of issues facing medicine Compassionate Ethical Problem solving Strong communication skills Adaptability Problem-solving ability Strong communication skills Teamwork Aware of issues facing medicine Compassionate Ethical Problem-solving ability Strong communication skills Drive to excel Strong communication skills Problem-solving ability Hard working
Communication skills Professionalism
Communication skills Professionalism Communication skills Medical knowledge Patient care Professionalism
Communication skills Professionalism Communication skills Medical knowledge Patient care Professionalism Communication skills Professionalism
* Stations used included some published in the literature as well as new stations developed de novo based on desired applicant characteristics for Emergency Medicine training and Accreditation Council for Graduate Medical Education (ACGME) core competencies (6,7).
MMI for EM Resident Selection
has not been shown to significantly affect performance, all participants experienced the same set of stations (17,18). The study was reviewed and approved by the Institutional Review Board at all three participating institutions and consent was obtained from participants. Study Setting and Population Participants were recruited, with participation voluntary, from the intern classes at three EM residencies during the first month of their residency, with a total of 71 of 72 eligible participating. The three EM residences included both 3-year and 4-year programs, as well as both urban and suburban training sites. During the MMI, participants were identified by codes and all study results were encoded by a data manager in order to blind participants’ identities. Measurements Participants completed pre- and post-experience surveys to judge their perceptions of the activity using a 5-point scale. Participants completed the MMI and were scored by preceptors recruited from residency faculty and staff recruited from all three participating institutions, most of whom had significant experience with the traditional interview process. Scoring was done on a 7-point global rating scale, with the preceptors being instructed as to general characteristics of a ‘‘good’’ answer (11). In many cases, the station was designed to focus on communication skills and the actual content of the answer was less important. The overall score assigned at each individual station was summed to determine the final MMI score. Data coded by a unique participant identifier was maintained in an Excel database (Microsoft, Redmond, WA). Background characteristics, including medical school performance markers, Standardized Letter of Recommendation (SLoR) scores, and US Medical Licensing Examination (USMLE) scores were obtained for the participants. In order to facilitate comparisons among participants spanning multiple institutions and classes, a five-tier match desirability rating was generated by each program. This allowed categorization of the relative strength of applicants by different programs and different years of application. A consensus opinion on quality of medical schools was also formulated to allow ranking on a five-tier scale. Data Analysis To test whether participants preferred MMI, traditional, or mixed interview styles, paired t-tests were performed on the 5-point scale ratings from the post-interview survey. To assess whether MMI performance was related to MMI preference, Pearson’s correlation coefficients
539
were computed between total MMI score (the sum of the overall performance rating on each station) and the post-interview preference rating for MMI, traditional, and mixed interview styles. To determine whether MMI performance was related to other measures of clinical performance (e.g., USMLE scores, clerkship grades), linear models were performed separately for each measure; ordinal variables were treated as continuous predictors. All comparisons used r = 0.05 to test for statistical significance. Statistics were performed using R software (version 2.11.1), with the exception of descriptive statistics for demographics, which were performed with Microsoft Excel 2010. RESULTS Seventy-one of 72 eligible interns in EM or a combined EM-internal medicine (IM) residency participated in the complete MMI during their first month of training. One resident did not participate due to a scheduling conflict. Forty-two of the participants were drawn from three consecutive classes at a single 4-year EM program, 15 from a single class of EM and EM-IM at a second program, and 14 from a single class at the third site (1,14). Baseline demographics and responses to the proposed experience were collected on a pre-MMI survey with responses available for analysis from 70 of 71 participants (98.6% response rate). Mean age of the participants ranged from 25 to 29 years and 31 (44.3%) were female. The participants represented 38 distinct medical schools. None reported any prior experience with an MMI. The MMI itself was scored using the sum of the global ranking score of each station. The maximum possible score was 56 (8 stations with a maximum score of 7 points each) and the minimum possible score was 0. All participants completed all stations. The 71 participants in the MMI had a mean score of 42.4, median score of 42.5, range from 33 to 52, and standard deviation (SD) of 3.4. Participants completed surveys that included the scenario of an offer to interview at an EM residency program, followed by a brief description of the MMI and its potential benefit in ‘‘identifying candidates with outstanding communication and problem-solving skills.’’ Participant responses showed a negative effect on a decision to interview at this program on both pre (mean 2.7, SD 0.7) and post (mean 2.8, SD 0.9) surveys (Figure 1). In addition, while there was not a clear negative effect of the MMI on participants’ decisions to rank the program (mean 3.0, SD 0.8), there was a slight negative effect toward the program’s position on their rank-order list (mean 2.8, SD 0.9) (Figure 2). Both pre- and postexperience surveys had a 98.6% (70 of 71) response rate. Participants responded more favorably to the prospect of a mixed interview involving both MMI stations and
540
Figure 1. Effect of the Multiple-Mini Interview (MMI) on participant’s decision to interview at the program. Although the majority of applicants indicate that the presence of the MMI would not affect their decision to interview at a program, the pre- and post-MMI survey suggests that a number of potential applicants would have a negative effect on their decision to accept an interview.
unstructured interactions when compared with MMI alone, with a mean difference of 1.1 rating points (t[68] = 10.24; p < 0.01). In addition, participants’ preferences for a mix of MMI and traditional interview did not differ from their preference for a traditional interview alone (t[68] = 1.1; p = NS). Performance on the MMI did not significantly correlate with preference for the MMI. However, the effect was in the expected direction, with higher MMI performance associated with higher MMI preference (r = 0.15, t[65] = 1.19; p = NS) and lower MMI performance associated with higher tradition interview preference (r = 0.22, t[65] = 1.86; p = NS). MMI performance did not correlate with preference for a mix of interview methods (r = 0.08, t[65] = 0.63; p = NS). Participants leaned toward agreement with the statements, ‘‘the activities in the MMI were an accurate assessment of my communication skills’’ (mean 3.3, SD
L. R. Hopson et al.
1.0; with 3 indicating neutrality and 97.1% [69 of 71] responding) and ‘‘problem-solving skills’’ (mean 3.3, SD 0.9 with 3 indicating neutrality and 97.1% [69 of 71] responding). Each participant was assigned a match desirability ranking by their program to reflect their program’s opinion of their strength as an applicant. This ordinal variable had five levels of desirability (least, less, neutral, more, most). This ranking did not correlate with MMI performance (F[1, 66] = 1.3; p = NS). Also, MMI performance did not correlate with either USMLE Step 1 (r = 0.04, t[65] = 0.31; p = NS) or USMLE Step 2 (r = 0.13, t[60] = 0.99, p = NS) scores. Finally, medical school quality (assessed on a 1 to 5 scale, with 1 being high) vs. MMI performance did not show a correlation (r = 0.07, t[65] = 0.59; p = NS). EM grades from both third and fourth-year rotations (n = 67) and third-year clerkship grades in medicine (n = 64), surgery (n = 62), and pediatrics (n = 62) were standardized to pass, high pass, or honors. Grades from institutions with pass/fail grading and no indication of honors performance on the Medical Student Performance Evaluation or letters of recommendation were excluded from the analysis. Performance on the EM clerkship did predict MMI performance (F[1, 66] = 4.18; p < 0.05). The effect was in the expected direction, with mean MMI performance of participants with a grade of ‘‘pass’’ scoring 41.19, those with ‘‘high pass’’ scoring 42.15, and those with ‘‘honors’’ scoring 43.13. Other core clerkship grades also showed MMI performance means in the expected direction, although none were significant (Table 2). EM utilizes an SLoR from EM faculty writing letters for students. The SLoR asks the recommender to use a global ranking of outstanding (top 10%), excellent (top one third), very good (middle one third), or good (lower one third) to stratify the applicant. Utilizing the lowest global ranking received by an applicant on any of their SLoRs, there is a correlation with MMI performance (R = 0.36; r < 0.05). DISCUSSION
Figure 2. Effect of the Multiple-Mini Interview (MMI) on participant rank-order list decision. During the postexperience survey, the majority of participants indicated that the MMI would not have an effect on their decision to rank the program using the MMI or its position on their rank list. However, slightly more participants indicated that the MMI would have a negative than positive effect on their rank list decisions about the program.
The MMI has been used successfully in undergraduate medical admissions. For this study, a MMI was successfully created for potential use in the EM postgraduate admissions process. In order to address concerns of a negative effect on a program’s ability to recruit highquality applicants, the MMI was run utilizing early interns as surrogates for applicants. In addition, the use of early interns allowed us to track performance outcomes on the participants in a manner that would not have been possible with a large group of applicants who would ultimately disperse to many training
MMI for EM Resident Selection
541
Table 2. Third-Year Clerkship Grades Were Not Significantly Correlated with Multiple-Mini Interview Performance and Emergency Medicine Grades Did Show A Significant Correlation MMI Score Medical School Rotation Emergency medicine Medicine Surgery Pediatrics
Correlation
Honors
High Pass
Pass
F(1, 66) = 4.18; p < 0.05 F(1, 63) = 0.7; p = NS F(1, 61) = 0.13; p = NS F(1, 61) = 2.59; p = NS
43.13 42.97 42.60 43.06
42.15 42.68 42.63 43.27
41.19 42.13 42.26 41.50
MMI = Multiple-Mini Interview; NS = not significant.
programs. Given that the participants are already securely matched into a residency training program and several months removed from the pressures of the interview process, there must be some concern that their performance and responses might not exactly approximate those of an applicant in the midst of the process. We believed, however, that early interns are appropriate surrogates, as they have had minimal training and experiences beyond medical school and would likely not have acclimated fully into their role as trainees. Support for this is also provided by the endorsement of perceptions of apprehension about the activity during their pre- and post-experience surveys. In addition, the logistical and practical hurdles involved in utilizing and tracking outcomes in residency applicants would likely have rendered collection of outcomes data impractical in that population. The results suggest that the MMI provides information that is not being reported by other means, such as USMLE scores, medical school quality, and third-year core clerkship scores. This MMI does correlate with EM clerkship grades, which suggest that it is reflective of an element valuable in actual EM clinical situations. It is, therefore, potentially useful for predicting EM resident performance. The MMI is not measuring clinical skills in general, however. Although mean MMI scores were in the expected direction for other clerkship grades (with participants with honors grades outperforming those with pass grades), none of these effects were as strong as that of EM clerkship scores. This MMI was developed with desirable characteristics for EM in mind, so it might be that an MMI developed specifically for another specialty area with different stations would be able to better predict resident performance in different domains. There is a correlation between lower MMI performance and the lowest assigned global ranking on the SLoR provided by EM faculty. One might speculate that this ranking, like the MMI, is sensitive to communication ability and other ‘‘soft’’ skills. Limitations Methodological limitations include a lack of multiple raters on stations, which would increase reliability; how-
ever, the process design is in keeping with previous research and admissions implementation and are within the minimum number of recommended contacts (1,2). Additionally the same stations were repeated at each site and with each new cohort, however, it has been shown that advance knowledge of the station and retesting do not appear to alter outcomes (17,19). Finally, the study was implemented at only three sites, all within a similar geographic area, which might have resulted in a bias in underlying subject characteristics. Preceptor bias is another potential confounder in this study, as the interactions are conducted face-to-face. Although it is possible to fully blind the preceptor to the identity of the participants, we attempted to minimize this by identifying individuals by codes rather than names and utilizing a variety of faculty and staff, including some from different participating institutions. CONCLUSIONS Although one interpretation of the available data is that the MMI is generating different information about an applicant, more definitive conclusions should wait until outcome data exist. Follow-up studies to determine whether the MMI correlates with future performance in EM are being conducted. The hypothesis is that an individual’s performance on the MMI will correlate more strongly with future rankings of global performance than the traditional candidate assessments (as compiled into the Match Desirability Rating) at the end of intern year and at the conclusion of residency training. The response of participants to the MMI is negative, although a mix of MMI stations and traditional unstructured interviews appears to be received much more positively and might potentially provide a mechanism to integrate this method into EM residency interviews. Future work will determine whether the extra information obtained by using an MMI offsets the potential loss of potential applicants turned off by the process. At this time, the response to the MMI must be considered a potential limiting factor to its adoption for EM candidate selection.
542
L. R. Hopson et al.
REFERENCES 1. Edwards JC, Johnson EK, Molidor JB. The interview in the admission process. Acad Med 1990;65:167–77. 2. Eva KW, Reiter HI. Where judgement fails: pitfalls in the selection process for medical personnel. Adv Health Sci Educ Theory Pract 2004;9:161–74. 3. Kreiter CD, Yin P, Solow C, Brennan R. Investigating the reliability of the medical school admissions interview. Adv Health Sci Educ Theory Pract 2004;9:147–59. 4. Salvatori P. Reliability and validity of admissions tools used to select students for the health professions. Adv Health Sci Educ Theory Pract 2001;6:159–75. 5. National Resident Matching Program and Association of American Medical Colleges. Charting Outcomes in the Match: Characteristics of Applicants Who Matched to Their Preferred Specialty in the 2011 Main Residency Match; August 2011. 6. Eva KW, Rosenfeld J, Reiter HI, Norman GR. An admissions OSCE: the multiple mini-interview. Med Educ 2004;38:314–26. 7. Lemay J-F, Lockyer JM, Collin VT, Brownell AKW. Assessment of non-cognitive traits through the admissions multiple mini-interview. Med Educ 2007;41:573–9. 8. Eva KW, Reiter HI, Rosenfeld J, Norman GR. The ability of the multiple mini-interview to predict preclerkship performance in medical school. Acad Med 2004;79(Suppl.):S40–2. 9. Reiter HI, Eva KW, Rosenfeld J, Norman GR. Multiple miniinterviews predict clerkship and licensing examination performance. Med Educ 2007;41:378–84. 10. Harris S, Owen C. Discerning quality: using the multiple miniinterview in student selection for the Australian National University Medical School. Med Educ 2007;41:234–41.
11. Roberts C, Walton M, Rothnie I, et al. Factors affecting the utility of the multiple mini-interview in selecting candidates for graduate-entry medical school. Med Educ 2008;42:396– 404. 12. Finlayson HC, Townson AF. Resident selection for a physical medicine and rehabilitation program: feasibility and reliability of the multiple mini-interview. Am J Phys Med Rehabil 2011; 90:330–5. 13. Dore KL, Kreuger S, Ladhani M, et al. The reliability and acceptability of the Multiple Mini-Interview as a selection instrument for postgraduate admissions. Acad Med 2010; 85(Suppl.):S60–3. 14. Hofmeister M, Lockyer J, Crutcher R. The acceptability of the multiple mini interview for resident selection. Fam Med 2008;40: 734–40. 15. Humphrey S, Dowson S, Wall D, Diwakar V, Goodyear HM. Multiple mini-interviews: opinions of candidates and interviewers. Med Educ 2008;42:207–13. 16. Rosenfeld JM, Reiter HI, Trinh K, Eva KW. A cost efficiency comparison between the multiple mini-interview and traditional admissions interviews. Adv Health Sci Educ Theory Pract 2008; 13:43–58. 17. Griffin B, Harding DW, Wilson IG, Yeomans ND. Does practice make perfect? The effect of coaching and retesting on selection tests used for admission to an Australian medical school. Med J Aust 2008;189:270–3. 18. Axelson RD, Kreiter CD. Rater and occasion impacts on the reliability of pre-admission assessments. Med Educ 2009;43: 1198–202. 19. Reiter H, Salvatori P, Rosenfeld J, Trinh K, Eva KW. The effect of defined violations of test security on admissions outcomes using multiple mini-interviews. Med Educ 2006;40:36–42.
MMI for EM Resident Selection
ARTICLE SUMMARY 1. Why is this topic important? The Multiple-Mini Interview (MMI) provides a potential mechanism for emergency medicine (EM) residency candidate selection that, based on medical school admissions research, may better identify high-performing candidates, particularly in the critical domains of communications and problem solving, than traditional methods. 2. What does this study attempt to show? This study evaluates the response of potential participants to utilizing the MMI for EM residency admissions and whether the method may have consequences for the program using it. In addition, this study also evaluates whether the MMI provides novel information about a candidate. 3. What are the key findings? The key findings of the study are that the traditional unstructured interview is preferred over the MMI by participants and that using the MMI can negatively impact a program’s recruitment. In addition, EM grades correlated with MMI performance; however, MMI performance did not correlate with match desirability and did not predict most other components of an application. 4. How is patient care affected? The impact of the MMI on patient care is indirect through the potential to identify individuals with strengths in interpersonal communications and problem solving, which are integral to the high-level practice of EM.
543
http://dx.doi.org/10.1016/j.jemermed.2013.08.119
Education THE MULTIPLE MINI-INTERVIEW FOR EMERGENCY MEDICINE RESIDENT SELECTION Laura R. Hopson, MD,* John C. Burkhardt, MD, MA,*† R. Brent Stansfield, PHD,† Taher Vohra, MD,‡ Danielle Turner-Lawrence, MD,§ and Eve D. Losman, MD* *Department of Emergency Medicine, University of Michigan, Ann Arbor, Michigan, †Department of Medical Education, University of Michigan, Ann Arbor, Michigan, ‡Department of Emergency Medicine, Henry Ford Hospital, Detroit, Michigan, and §Department of Emergency Medicine, Oakland University, William Beaumont School of Medicine, Rochester, Michigan Reprint Address: Laura R. Hopson, MD, Department of Emergency Medicine, University of Michigan, 1500 E. Medical Center Drive, TC B1 382, Ann Arbor, MI 48109-5305
, Abstract—Background: The Multiple Mini-Interview (MMI) uses multiple, short-structured contacts to evaluate communication and professionalism. It predicts medical school success better than the traditional interview and application. Its acceptability and utility in emergency medicine (EM) residency selection are unknown. Objective: We theorized that participants would judge the MMI equal to a traditional unstructured interview and it would provide new information for candidate assessment. Methods: Seventyone interns from 3 programs in the first month of training completed an eight-station MMI focused on EM topics. Pre- and post-surveys assessed reactions. MMI scores were compared with application data. Results: EM grades correlated with MMI performance (F[1, 66] = 4.18; p < 0.05) with honors students having higher scores. Higher third-year clerkship grades were associated with higher MMI performance, although this was not statistically significant. MMI performance did not correlate with match desirability and did not predict most other components of an application. There was a correlation between lower MMI scores and lower global ranking on the Standardized Letter of Recommendation. Participants preferred a traditional interview (mean difference = 1.36; p < 0.01). A mixed format (traditional interview and MMI) was preferred over a MMI alone (mean difference = 1.1; p < 0.01). MMI performance did not significantly correlate with preference for the MMI. Conclusions: Although the MMI alone was viewed less favorably than a traditional interview, participants were receptive to
a mixed-methods interview. The MMI does correlate with performance on the EM clerkship and therefore can measure important abilities for EM success. Future work will determine whether MMI performance predicts residency performance. Ó 2014 Elsevier Inc. , Keywords—graduate medical education; residency selection; Multiple Mini-Interview; professionalism; communication
INTRODUCTION Selection of appropriate residency candidates is a highstakes process for both the training program and applicant. In medical school admissions, traditional measures of applicant quality, such as interviews, medical college admission tests, and grade-point average, have fared poorly in predicting medical school performance (1 4). Current literature in emergency medicine (EM) suggests that the best predictors of residency performance include clerkship grades and quality of medical school attended (5). However, research at a Canadian Family Medicine residency suggests that although interviews and application files can correctly predict outstanding residents, they are insensitive for identifying potential ‘‘problem’’ residents (4).
RECEIVED: 27 January 2013; FINAL SUBMISSION RECEIVED: 23 July 2013; ACCEPTED: 23 August 2013 537
538
L. R. Hopson et al.
To address the limitations of the traditional application process, Eva et al. developed the Multiple Mini-Interview (MMI), which uses an objective structured clinical examination format to have multiple short, structured contacts with an applicant and improves assessment of the applicant’s communication and problem-solving skills (6,7). Performance on the MMI correlates with preclinical grades and performance on clerkships (8,9). There is also some suggestion that the MMI might be able to pick out candidates with professionalism issues, who might not otherwise be detected until later in their training (8,9). The MMI format has been used successfully at medical schools throughout the world, and is becoming increasingly prevalent at United States (US) medical schools (7,10,11). The residency selection process is a different environment than medical school admissions. A more rigorous interview technique might adversely affect a program’s match outcomes. The few reports on the MMI in the postgraduate environment have not identified significant impediments to utilizing this method on the part of the participants (12 15). However, there are significant limitations to extrapolating to an EM residency within the US, as they either do not include participants from
the specialty or include substantial numbers of international medical graduates, whose motivations and competitiveness for a residency slot might differ from US graduates. In addition, it has not been demonstrated that the MMI provides additional useful information about the applicant in the post-graduate realm beyond that provided in a traditional interview and residency application package. The objective of this study was to determine the perspectives of participants on the MMI in a simulated EM interview situation. A secondary objective was to determine if the MMI would provide new information to assess residency candidates. METHODS Study Design An eight-station MMI was created by the authors using information published in the literature (6,16). The scenarios were drawn directly from published examples or created de novo based on characteristics believed to be desirable in EM trainees (Table 1). In their first month of training, EM interns were used as surrogates for residency applicants. As prior knowledge of the stations
Table 1. Examples of Multiple-Mini Interview Stations (6)* Station Description Give and receive instructions (two stations) for an origami paper-folding activity using only verbal means
Disclosure of damaging a borrower computer (role-playing)
Discussing a patient’s condition with a parent while respecting confidentiality (role-playing)
Giving verbal instruction to hang pictures
Discussing decision-making process whether or not to recommend testing for a genetic condition
Deciding which tasks to complete given a limited amount of time and explaining decision-making process
Desired Characteristic
ACGME Competency
Adaptability Hard working Problem solving Strong communication skills Teamwork Altruism Ethical Problem solving Strong communication skills Adaptability Altruism Aware of issues facing medicine Compassionate Ethical Problem solving Strong communication skills Adaptability Problem-solving ability Strong communication skills Teamwork Aware of issues facing medicine Compassionate Ethical Problem-solving ability Strong communication skills Drive to excel Strong communication skills Problem-solving ability Hard working
Communication skills Professionalism
Communication skills Professionalism Communication skills Medical knowledge Patient care Professionalism
Communication skills Professionalism Communication skills Medical knowledge Patient care Professionalism Communication skills Professionalism
* Stations used included some published in the literature as well as new stations developed de novo based on desired applicant characteristics for Emergency Medicine training and Accreditation Council for Graduate Medical Education (ACGME) core competencies (6,7).
MMI for EM Resident Selection
has not been shown to significantly affect performance, all participants experienced the same set of stations (17,18). The study was reviewed and approved by the Institutional Review Board at all three participating institutions and consent was obtained from participants. Study Setting and Population Participants were recruited, with participation voluntary, from the intern classes at three EM residencies during the first month of their residency, with a total of 71 of 72 eligible participating. The three EM residences included both 3-year and 4-year programs, as well as both urban and suburban training sites. During the MMI, participants were identified by codes and all study results were encoded by a data manager in order to blind participants’ identities. Measurements Participants completed pre- and post-experience surveys to judge their perceptions of the activity using a 5-point scale. Participants completed the MMI and were scored by preceptors recruited from residency faculty and staff recruited from all three participating institutions, most of whom had significant experience with the traditional interview process. Scoring was done on a 7-point global rating scale, with the preceptors being instructed as to general characteristics of a ‘‘good’’ answer (11). In many cases, the station was designed to focus on communication skills and the actual content of the answer was less important. The overall score assigned at each individual station was summed to determine the final MMI score. Data coded by a unique participant identifier was maintained in an Excel database (Microsoft, Redmond, WA). Background characteristics, including medical school performance markers, Standardized Letter of Recommendation (SLoR) scores, and US Medical Licensing Examination (USMLE) scores were obtained for the participants. In order to facilitate comparisons among participants spanning multiple institutions and classes, a five-tier match desirability rating was generated by each program. This allowed categorization of the relative strength of applicants by different programs and different years of application. A consensus opinion on quality of medical schools was also formulated to allow ranking on a five-tier scale. Data Analysis To test whether participants preferred MMI, traditional, or mixed interview styles, paired t-tests were performed on the 5-point scale ratings from the post-interview survey. To assess whether MMI performance was related to MMI preference, Pearson’s correlation coefficients
539
were computed between total MMI score (the sum of the overall performance rating on each station) and the post-interview preference rating for MMI, traditional, and mixed interview styles. To determine whether MMI performance was related to other measures of clinical performance (e.g., USMLE scores, clerkship grades), linear models were performed separately for each measure; ordinal variables were treated as continuous predictors. All comparisons used r = 0.05 to test for statistical significance. Statistics were performed using R software (version 2.11.1), with the exception of descriptive statistics for demographics, which were performed with Microsoft Excel 2010. RESULTS Seventy-one of 72 eligible interns in EM or a combined EM-internal medicine (IM) residency participated in the complete MMI during their first month of training. One resident did not participate due to a scheduling conflict. Forty-two of the participants were drawn from three consecutive classes at a single 4-year EM program, 15 from a single class of EM and EM-IM at a second program, and 14 from a single class at the third site (1,14). Baseline demographics and responses to the proposed experience were collected on a pre-MMI survey with responses available for analysis from 70 of 71 participants (98.6% response rate). Mean age of the participants ranged from 25 to 29 years and 31 (44.3%) were female. The participants represented 38 distinct medical schools. None reported any prior experience with an MMI. The MMI itself was scored using the sum of the global ranking score of each station. The maximum possible score was 56 (8 stations with a maximum score of 7 points each) and the minimum possible score was 0. All participants completed all stations. The 71 participants in the MMI had a mean score of 42.4, median score of 42.5, range from 33 to 52, and standard deviation (SD) of 3.4. Participants completed surveys that included the scenario of an offer to interview at an EM residency program, followed by a brief description of the MMI and its potential benefit in ‘‘identifying candidates with outstanding communication and problem-solving skills.’’ Participant responses showed a negative effect on a decision to interview at this program on both pre (mean 2.7, SD 0.7) and post (mean 2.8, SD 0.9) surveys (Figure 1). In addition, while there was not a clear negative effect of the MMI on participants’ decisions to rank the program (mean 3.0, SD 0.8), there was a slight negative effect toward the program’s position on their rank-order list (mean 2.8, SD 0.9) (Figure 2). Both pre- and postexperience surveys had a 98.6% (70 of 71) response rate. Participants responded more favorably to the prospect of a mixed interview involving both MMI stations and
540
Figure 1. Effect of the Multiple-Mini Interview (MMI) on participant’s decision to interview at the program. Although the majority of applicants indicate that the presence of the MMI would not affect their decision to interview at a program, the pre- and post-MMI survey suggests that a number of potential applicants would have a negative effect on their decision to accept an interview.
unstructured interactions when compared with MMI alone, with a mean difference of 1.1 rating points (t[68] = 10.24; p < 0.01). In addition, participants’ preferences for a mix of MMI and traditional interview did not differ from their preference for a traditional interview alone (t[68] = 1.1; p = NS). Performance on the MMI did not significantly correlate with preference for the MMI. However, the effect was in the expected direction, with higher MMI performance associated with higher MMI preference (r = 0.15, t[65] = 1.19; p = NS) and lower MMI performance associated with higher tradition interview preference (r = 0.22, t[65] = 1.86; p = NS). MMI performance did not correlate with preference for a mix of interview methods (r = 0.08, t[65] = 0.63; p = NS). Participants leaned toward agreement with the statements, ‘‘the activities in the MMI were an accurate assessment of my communication skills’’ (mean 3.3, SD
L. R. Hopson et al.
1.0; with 3 indicating neutrality and 97.1% [69 of 71] responding) and ‘‘problem-solving skills’’ (mean 3.3, SD 0.9 with 3 indicating neutrality and 97.1% [69 of 71] responding). Each participant was assigned a match desirability ranking by their program to reflect their program’s opinion of their strength as an applicant. This ordinal variable had five levels of desirability (least, less, neutral, more, most). This ranking did not correlate with MMI performance (F[1, 66] = 1.3; p = NS). Also, MMI performance did not correlate with either USMLE Step 1 (r = 0.04, t[65] = 0.31; p = NS) or USMLE Step 2 (r = 0.13, t[60] = 0.99, p = NS) scores. Finally, medical school quality (assessed on a 1 to 5 scale, with 1 being high) vs. MMI performance did not show a correlation (r = 0.07, t[65] = 0.59; p = NS). EM grades from both third and fourth-year rotations (n = 67) and third-year clerkship grades in medicine (n = 64), surgery (n = 62), and pediatrics (n = 62) were standardized to pass, high pass, or honors. Grades from institutions with pass/fail grading and no indication of honors performance on the Medical Student Performance Evaluation or letters of recommendation were excluded from the analysis. Performance on the EM clerkship did predict MMI performance (F[1, 66] = 4.18; p < 0.05). The effect was in the expected direction, with mean MMI performance of participants with a grade of ‘‘pass’’ scoring 41.19, those with ‘‘high pass’’ scoring 42.15, and those with ‘‘honors’’ scoring 43.13. Other core clerkship grades also showed MMI performance means in the expected direction, although none were significant (Table 2). EM utilizes an SLoR from EM faculty writing letters for students. The SLoR asks the recommender to use a global ranking of outstanding (top 10%), excellent (top one third), very good (middle one third), or good (lower one third) to stratify the applicant. Utilizing the lowest global ranking received by an applicant on any of their SLoRs, there is a correlation with MMI performance (R = 0.36; r < 0.05). DISCUSSION
Figure 2. Effect of the Multiple-Mini Interview (MMI) on participant rank-order list decision. During the postexperience survey, the majority of participants indicated that the MMI would not have an effect on their decision to rank the program using the MMI or its position on their rank list. However, slightly more participants indicated that the MMI would have a negative than positive effect on their rank list decisions about the program.
The MMI has been used successfully in undergraduate medical admissions. For this study, a MMI was successfully created for potential use in the EM postgraduate admissions process. In order to address concerns of a negative effect on a program’s ability to recruit highquality applicants, the MMI was run utilizing early interns as surrogates for applicants. In addition, the use of early interns allowed us to track performance outcomes on the participants in a manner that would not have been possible with a large group of applicants who would ultimately disperse to many training
MMI for EM Resident Selection
541
Table 2. Third-Year Clerkship Grades Were Not Significantly Correlated with Multiple-Mini Interview Performance and Emergency Medicine Grades Did Show A Significant Correlation MMI Score Medical School Rotation Emergency medicine Medicine Surgery Pediatrics
Correlation
Honors
High Pass
Pass
F(1, 66) = 4.18; p < 0.05 F(1, 63) = 0.7; p = NS F(1, 61) = 0.13; p = NS F(1, 61) = 2.59; p = NS
43.13 42.97 42.60 43.06
42.15 42.68 42.63 43.27
41.19 42.13 42.26 41.50
MMI = Multiple-Mini Interview; NS = not significant.
programs. Given that the participants are already securely matched into a residency training program and several months removed from the pressures of the interview process, there must be some concern that their performance and responses might not exactly approximate those of an applicant in the midst of the process. We believed, however, that early interns are appropriate surrogates, as they have had minimal training and experiences beyond medical school and would likely not have acclimated fully into their role as trainees. Support for this is also provided by the endorsement of perceptions of apprehension about the activity during their pre- and post-experience surveys. In addition, the logistical and practical hurdles involved in utilizing and tracking outcomes in residency applicants would likely have rendered collection of outcomes data impractical in that population. The results suggest that the MMI provides information that is not being reported by other means, such as USMLE scores, medical school quality, and third-year core clerkship scores. This MMI does correlate with EM clerkship grades, which suggest that it is reflective of an element valuable in actual EM clinical situations. It is, therefore, potentially useful for predicting EM resident performance. The MMI is not measuring clinical skills in general, however. Although mean MMI scores were in the expected direction for other clerkship grades (with participants with honors grades outperforming those with pass grades), none of these effects were as strong as that of EM clerkship scores. This MMI was developed with desirable characteristics for EM in mind, so it might be that an MMI developed specifically for another specialty area with different stations would be able to better predict resident performance in different domains. There is a correlation between lower MMI performance and the lowest assigned global ranking on the SLoR provided by EM faculty. One might speculate that this ranking, like the MMI, is sensitive to communication ability and other ‘‘soft’’ skills. Limitations Methodological limitations include a lack of multiple raters on stations, which would increase reliability; how-
ever, the process design is in keeping with previous research and admissions implementation and are within the minimum number of recommended contacts (1,2). Additionally the same stations were repeated at each site and with each new cohort, however, it has been shown that advance knowledge of the station and retesting do not appear to alter outcomes (17,19). Finally, the study was implemented at only three sites, all within a similar geographic area, which might have resulted in a bias in underlying subject characteristics. Preceptor bias is another potential confounder in this study, as the interactions are conducted face-to-face. Although it is possible to fully blind the preceptor to the identity of the participants, we attempted to minimize this by identifying individuals by codes rather than names and utilizing a variety of faculty and staff, including some from different participating institutions. CONCLUSIONS Although one interpretation of the available data is that the MMI is generating different information about an applicant, more definitive conclusions should wait until outcome data exist. Follow-up studies to determine whether the MMI correlates with future performance in EM are being conducted. The hypothesis is that an individual’s performance on the MMI will correlate more strongly with future rankings of global performance than the traditional candidate assessments (as compiled into the Match Desirability Rating) at the end of intern year and at the conclusion of residency training. The response of participants to the MMI is negative, although a mix of MMI stations and traditional unstructured interviews appears to be received much more positively and might potentially provide a mechanism to integrate this method into EM residency interviews. Future work will determine whether the extra information obtained by using an MMI offsets the potential loss of potential applicants turned off by the process. At this time, the response to the MMI must be considered a potential limiting factor to its adoption for EM candidate selection.
542
L. R. Hopson et al.
REFERENCES 1. Edwards JC, Johnson EK, Molidor JB. The interview in the admission process. Acad Med 1990;65:167–77. 2. Eva KW, Reiter HI. Where judgement fails: pitfalls in the selection process for medical personnel. Adv Health Sci Educ Theory Pract 2004;9:161–74. 3. Kreiter CD, Yin P, Solow C, Brennan R. Investigating the reliability of the medical school admissions interview. Adv Health Sci Educ Theory Pract 2004;9:147–59. 4. Salvatori P. Reliability and validity of admissions tools used to select students for the health professions. Adv Health Sci Educ Theory Pract 2001;6:159–75. 5. National Resident Matching Program and Association of American Medical Colleges. Charting Outcomes in the Match: Characteristics of Applicants Who Matched to Their Preferred Specialty in the 2011 Main Residency Match; August 2011. 6. Eva KW, Rosenfeld J, Reiter HI, Norman GR. An admissions OSCE: the multiple mini-interview. Med Educ 2004;38:314–26. 7. Lemay J-F, Lockyer JM, Collin VT, Brownell AKW. Assessment of non-cognitive traits through the admissions multiple mini-interview. Med Educ 2007;41:573–9. 8. Eva KW, Reiter HI, Rosenfeld J, Norman GR. The ability of the multiple mini-interview to predict preclerkship performance in medical school. Acad Med 2004;79(Suppl.):S40–2. 9. Reiter HI, Eva KW, Rosenfeld J, Norman GR. Multiple miniinterviews predict clerkship and licensing examination performance. Med Educ 2007;41:378–84. 10. Harris S, Owen C. Discerning quality: using the multiple miniinterview in student selection for the Australian National University Medical School. Med Educ 2007;41:234–41.
11. Roberts C, Walton M, Rothnie I, et al. Factors affecting the utility of the multiple mini-interview in selecting candidates for graduate-entry medical school. Med Educ 2008;42:396– 404. 12. Finlayson HC, Townson AF. Resident selection for a physical medicine and rehabilitation program: feasibility and reliability of the multiple mini-interview. Am J Phys Med Rehabil 2011; 90:330–5. 13. Dore KL, Kreuger S, Ladhani M, et al. The reliability and acceptability of the Multiple Mini-Interview as a selection instrument for postgraduate admissions. Acad Med 2010; 85(Suppl.):S60–3. 14. Hofmeister M, Lockyer J, Crutcher R. The acceptability of the multiple mini interview for resident selection. Fam Med 2008;40: 734–40. 15. Humphrey S, Dowson S, Wall D, Diwakar V, Goodyear HM. Multiple mini-interviews: opinions of candidates and interviewers. Med Educ 2008;42:207–13. 16. Rosenfeld JM, Reiter HI, Trinh K, Eva KW. A cost efficiency comparison between the multiple mini-interview and traditional admissions interviews. Adv Health Sci Educ Theory Pract 2008; 13:43–58. 17. Griffin B, Harding DW, Wilson IG, Yeomans ND. Does practice make perfect? The effect of coaching and retesting on selection tests used for admission to an Australian medical school. Med J Aust 2008;189:270–3. 18. Axelson RD, Kreiter CD. Rater and occasion impacts on the reliability of pre-admission assessments. Med Educ 2009;43: 1198–202. 19. Reiter H, Salvatori P, Rosenfeld J, Trinh K, Eva KW. The effect of defined violations of test security on admissions outcomes using multiple mini-interviews. Med Educ 2006;40:36–42.
MMI for EM Resident Selection
ARTICLE SUMMARY 1. Why is this topic important? The Multiple-Mini Interview (MMI) provides a potential mechanism for emergency medicine (EM) residency candidate selection that, based on medical school admissions research, may better identify high-performing candidates, particularly in the critical domains of communications and problem solving, than traditional methods. 2. What does this study attempt to show? This study evaluates the response of potential participants to utilizing the MMI for EM residency admissions and whether the method may have consequences for the program using it. In addition, this study also evaluates whether the MMI provides novel information about a candidate. 3. What are the key findings? The key findings of the study are that the traditional unstructured interview is preferred over the MMI by participants and that using the MMI can negatively impact a program’s recruitment. In addition, EM grades correlated with MMI performance; however, MMI performance did not correlate with match desirability and did not predict most other components of an application. 4. How is patient care affected? The impact of the MMI on patient care is indirect through the potential to identify individuals with strengths in interpersonal communications and problem solving, which are integral to the high-level practice of EM.
543
