Advances in Health Sciences Education 1: 3-16, 1996. ( 1996 Kluwer Academic Publishers. Printedin the Netherlands.
3
Admission to Medical School: International Perspectives JANINE C. EDWARDS*, EUGENE K. JOHNSON and JOHN B. MOLIDOR *Medical College of Wisconsin, 8701 Watertown Plank Road, Milwaukee, Wisconsin 53226, U.S.A.
Abstract. Admission to medical school is the goal of many students in many countries. The admission process varies from country to country. In some countries, students compete in an open market to gain a position in medical school. In other countries, "intake" is a more routine, planned beaureaucratic process. Where competition reigns, the interview is an important part of the selection process. The interview has been defined by Bingham and Moore [1] as: A serious conversation directed to a definite purpose other than satisfaction in the conversation itself... We must recognize that not only spoken words, but other means of face-to-face communication also are used. Inflection, qualities of voice, facial expression, glint of the eye, posture, gestures, and general behavior supplement what is said. They all contribute to the purposeful exchange of meanings which is the interview. Faculty members in medical schools interview patients all the time. This type of interview, however, is different from the admission interview conducted for applicants to medical school. Patient interviews are highly patterned and structured to obtain specific information. Interviews of applicants, on the other hand, usually are more open-ended. The psychology of the two types of interviews differ also. Applicants to medical school, if accepted, will in time become colleagues with their interviewers and will have increasing levels of responsibility and respect. Patients, however, will always be dependent upon the physicians who treat them. In this article, we present basic facts, conclusions, and recommendations from a review of literature about the interview [2]. Results of a survey of admission interviews at Canadian and United Kingdom medical schools are presented for the first time, and comparisons with United States interview practices are drawn. Finally, descriptions of the selection process at several medical schools with problem based learning curricula are provided and comparisons are noted. Key words: admission, medical education, problem-based learning
Purposes of Interviewing Interviewing applicants to medical school can serve four purposes: 1) gathering information; 2) making the decision to accept or reject; 3) verifying information provided in the application; 4) recruiting particular applicants. Gathering information may be the most important purpose of interviews. Quantitative information, such as transcripts and background information, is obviously gathered most expeditiously by paper or by computer. The more elusive information, such as motivation, leadership, altruism, and interpersonal skills, referred to by McGaghie [3] as noncognitive information or nonacademic information, is best evaluated in interviews.
4
JANINE C. EDWARDS ET AL.
A serious, face-to-face conversation can probably reveal more non-cognitive information about an applicant than any other form of evaluation. Purposes of the interview may be weighted differently for various groups of applicants. There is no legal impediment to evaluating different subgroups by different criteria because these subgroups are self-selected; that is, their credentials are the basis for inclusion in the subgroup. However, all individuals within a subgroup must be treated consistently. The interview will be most effective if it is tailored to fulfill various purposes. Decision making, of course, is the end result of the application process. Most medical schools make decisions after gathering written and interview information. The written information is usually reviewed first and then invitations to interview are issued on the basis of the written information. Decisions to accept or reject are a combination of all the available information. Verifying information is another important function of interviewing. Checking out the veracity of information provided in personal statements, autobiographical sketches, secondary information materials, and the like can be done during the course of interviews. By linking the interview to the initial review of application materials, the admissions committee can doublecheck or verify the authenticity of what has been presented. Finally, the interview can serve a recruitment function. Many medical schools want to recruit particular types of applicants or to generate good will toward the school. The interview is usually regarded as the centerpiece of an applicant's visit to a medical school. If the applicant is treated well during his/her visit and is encountered with respect and dignity, the school is more likely to attract desirable candidates. For applicants with top academic qualifications, interviewers may wish to focus first on determining whether the person has the necessary personality and character traits to become a physician. After that determination is made positively, the interviewers may wish to focus on recruiting the applicant. For the large pool of average candidates, the interviewers may focus on obtaining non-cognitive information and verifying the written information. Making fine distinctions about character and personality among large numbers of applicants who have similar academic qualifications is a worthwhile endeavor. Ruling out candidates requires a somewhat different process. Interviewers skilled in psychological analysis, such as psychiatrists and psychologists, may be the best interviewers for candidates who give evidence of psychological problems. Any evidence of abnormality may trigger additional evaluation. The interview in the medical school application process will no doubt be used in different ways depending upon the size of the pool of applicants. Kassebaum and Szenas [4] described the decline and rise of the applicant pool in U.S. medical schools during the 1980's and early 1990's. They found that the decline of the applicant pool in the mid-1980's was related to changes in the number and pattern of undergraduate majors and changes in employment conditions for college graduates at that time. More precisely, employment of college graduates increased at
ADMISSION TO MEDICAL SCHOOL: INTERNATIONAL PERSPECTIVES
5
that time; thus, many fewer college graduates were interested in further training because they could obtain profitable jobs immediately after college. This also may explain the phenomenon that fewer applicants who were rejected applied to medical schools a second time. Furthermore, fewer college students were majoring in the biological and physical sciences, "the most common antecedents of medical school applicants." From 1988 until the early 1990's, the number of medical school applicants rose dramatically. During that period, there was deterioration in employment of college graduates and the number of degrees awarded in the biological sciences increased. Repeat applications also increased during this period. Therefore, trends in college majors and employment of college graduates seem to be related to the decline and rise of applications to U.S. medical schools during the past decade. At the current time, the applicant pool in the U.S. is at an all time high. The interview is particularly helpful in choosing among all the academically qualified applicants those who most closely match the desired characteristics or the ethos of the medical school. That requires, however, that the admissions committee have done some careful analysis of the criteria for selection and for interviewing. Structuring the interview is the most productive method of ensuring that the applicants who are most desirable are, in fact, chosen. To summarize then, the interview can serve various purposes. It can be weighted differently as a result of the size of the applicant pool, and different subgroups of the applicant pool may be evaluated in different ways within the interview. All individuals within a subgroup must be treated consistently. Gathering information, verifying information, recruiting individuals, and making the decision to accept or reject are all useful purposes of the interview. Interview Formats
Several formats of interviews are possible, including 1) one-on-one; 2) group; 3) panel; and 4) combination. The one-on-one interview is self-explanatory. The group interview consists of several interviewees and one interviewer. The panel interview is conducted by many interviewers with one applicant. The combination interview might consist, for instance, of a one-on-one interview followed by a panel interview. Types of Interviews Three types of interviews have been identified and researched: structured, semistructured and unstructured. To be considered structured, an interview must meet these criteria: 1) interview content is developed from a job analysis or what we call a Success Analysis of Medical Student (SAMS); 2) the questions are standardized; that is, the same questions are asked of every applicant; 3) sample answers to the questions are provided to the interviewers to help them give consistent ratings; 4) a panel interview is conducted. Interviews that have some, but not all, of these criteria
6
JANINE C. EDWARDS ET AL.
are termed semi-structured. Interviews that have none of these criteria are called unstructured. Research studies during the 1980's indicated that adding structure improves the validity and reliability of interviews. The key to improving the validity of interviews is structuring the content, which is accomplished by conducting a job analysis or Success Analysis of Medical Students (SAMS). A SAMS ensures that all interviewers focus their questions on the agreed upon content or objectives. Two methods of conducting a SAMS are described here - the Critical Incidents Technique and the Delphi Method. Other methods of forming consensus are possible, of course. The Critical Incidents Technique consist of "a set of procedures for collecting direct observations of human behavior in such a way as to facilitate their potential usefulness in solving practical problems and developing broad psychological principles" [5]. To conduct a SAMS using critical incidents, medical school faculty members would reflect on and write critical incidents of medical students. These incidents are typically recollections of actual events, those that reveal successful and unsuccessful behaviors, attitudes, and performance. Approximately 50 to 200 critical incidents are needed to adequately describe the objectives of medical students. After the incidents have been written as descriptions, these are edited for clarity. Then a second group of experts reads the critical incidents and groups them into a few major categories. Any critical incidents on which the experts disagree are discarded. Finally, within each category, the incidents are scaled from high (effective) to low (ineffective) values. The resulting set of categories and critical incidents define the objectives of medical students and provide the content upon which to develop questions in the interview. A study that developed critical incidents for interviewing applicants to medical school was completed by Johnson [6]. The Delphi Method is a method of structuring a group communication process so that a group of individuals, as a whole, deal with a complex problem. In the 1950's the Rand Corporation developed the Delphi Method to forecast technology futures [7, 8]. This method was widely used throughout the 1960's and 1970's as a consensus building method. The Delphi Method is useful when a group is too large to meet in person to carry out a nominal group process or when it would be too inconvenient to do so. More specifically, the Delphi Method is valuable when the problem can benefit from subjective judgment on a collective basis; individuals have diverse expertise, disagreements need refereeing, for individual heterogeneity must be preserved. Medical school faculty members with their time constraints certainly meet these conditions, making the Delphi Method a feasible method of forming consensus among them. The first step in the Delphi Method is to generate ideas through a paper and pencil questionnaire from the group as a whole or from a subgroup. Then the entire group assigns a priority score to each item through ranking or through the use of the Likert scale. Several such rounds of assigning priority are usually necessary before consensus develops about a core of items or objectives.
ADMISSION TO MEDICAL SCHOOL: INTERNATIONAL PERSPECTIVES
7
Another method of improving the validity of interviews is to standardize the questions asked of every applicant. This ensures that the same material is covered in every interview and that one applicant's chance of being accepted is not impaired or boosted by lack of key data or influx of extraneous information. Some faculty members have suggested that standardized questions would soon be memorized and passed on to applicants through "the grapevine." Standardized questions could have enough variations that this "copying" problem would not be likely to occur. Interrater reliability can be improved by providing each interviewer with sample answers for each level of performance for the categories. Sample answers are usually "behavioral anchors," that is, behaviors describing a level of performance. Thus, for the category of interpersonal skills, there might be six levels of performance, with anchors such as "establishes rapport with a variety of people" (high rating) and "frequently alienates people" (low rating). Research studies on the validity and reliability of interviews using classic measurement theory have found that panel interviews yield greater reliability than one-on-one interviews. Panel interviews eliminate much of the interrater variance. More recently, generalizability theory has pointed out that a researcher should attempt to identify all the likely sources of error in a measurement situation rather than trying to minimize sources of error as a panel interview does [9]. Therefore, generalizability theory would encourage us to conduct multiple one-on-one interviews rather than one panel interview in order to arrive at a true measurement of the applicant. If it is not feasible for an admissions committee to conduct multiple one-on-one interviews, then the recommendation to conduct one panel interview is useful. Recommendations from Research Relatively little research has been done on the interview in medical education. We do know, however, that focusing on nonacademic interview data influenced the decision of admission committees to select more applicants with high ratings on attributes such as leadership, motivation, range of interests, and interpersonal skills [10-13]. Some slight evidence exists that the selection interview that emphasizes nonacademic criteria predicts success in clinical training [14-17]. A large body of research in the psychology and business literature, however, has developed a number of sound recommendations about the interview. For many years, psychology researchers studied the various types of bias in what are now referred to as "indirect" or "microanalytic" studies [18]. Webster [19] initiated this line of research because he believed that many microbased social psychological factors moderate the validity of the interview. A number of useful findings resulted from this long line of research. Bias can arise from a number of sources, including, but not limited to, rater tendencies, stereotyping, and interviewer background. The following findings are
supported by research:
8
JANINE C. EDWARDS ET AL.
1. Unfavorable information carries more weight than favorable information [2022]. 2. Rating errors, including the halo effect and distribution errors (leniency, severity, central tendency) can be corrected. These tendencies may be reduced by manipulating scale formats, such as using an even number of points on the scale, by making raters aware of their tendencies or by using statistical corrections [23]. 3. Gender differences exist in interview ratings. Women are rated lower by both male and female interviewers [24-27]. 4. Nonverbal, as well as verbal interactions, influence decisions [28-30]. 5. Interviewers develop a stereotype of a good applicant and then try to match the candidates to the stereotype [30]. 6. An average candidate following several outstanding (or marginal) applicants will be rated worse (or better) in comparison. This is called the "contrast effect" [30]. Several research reports indicate that training interviewers can improve performance in interviewing [30]. Training, of course, does not directly affect the decision to accept or reject applicants into medical school. Schuh [31] describes three dimensions of a good training program: instruction, coaching, and supervised practice. Wexley and colleagues [32] gave workshops to college students, and Latham and colleagues [33] trained corporate managers using these dimensions. Keenan [34] found that interviewers with training were more confident and that candidates were more likely to accept jobs with firms whose interviewers were trained. Recently, psychology researchers have shifted emphasis from the microanalytic variables that had been thought to moderate interview validity to direct models of validity. Interview structure is a major variable that has been found to directly affect validity. A number of studies (both individual studies and meta-analyses) of structured interviews give evidence that validity and reliability are higher than that of semi-structured or unstructured interviews. However, validity and reliability figures for semi-structured interviews are higher than unstructured interviews and even approach those of structured interviews. Review of the data that support these conclusions are presented in a previous publication [2]. In practical terms, this means that admissions committees can achieve greater validity and reliability in their interviews by using only one or two of the criteria for structured interviews, such as performing a SAMS or using behavioral anchors for rating scales. The unstructured interview has considerably less validity and reliability than structured or semi-structured interviews. Several important findings emerged from the review of literature published in 1990, and these continue to be important. 1. Various purposes of the interview may be weighted differently for different groups of applicants. 2. Bias in the interviewing process can be reduced by training interviewers.
ADMISSION TO MEDICAL SCHOOL: INTERNATIONAL PERSPECTIVES
9
3. Adding structure to the interview improves its validity and reliability. 4. Four methods of adding structure have been described: performing a job analysis or SAMS; asking the same questions of all applicants; developing sample answers or behavioral anchors for rating scales; having multiple one-on-one interviews or using panel interviews. Survey of Admission Interviews at Canadian and United Kingdom Medical Schools A survey of admission interviews was conducted in United States, Canadian, and United Kingdom medical schools in late 1989. Information about practices in U.S. medical schools was published in 1991 [35]; the data from Canadian and United Kingdom medical schools are presented here for the first time. METHOD
A questionnaire was developed that asked detailed questions about interview practices based on variables and practices extracted from research on both medical school admission interviews and general selection interviews. A previous survey by Puryear and Lewis also was a source for the questions on this survey. Staff members of the Association of American Medical Colleges (AAMC) and the Student Affairs National Committee on Admissions of the AAMC reviewed the questionnaire and made suggestions for its development. Pilot tests were made of the first draft questionnaire with an admission committee member, an admission committee chair, and a dean of admissions. RESULTS AND DISCUSSION
Data were received from 12 of 16 (75%) Canadian medical schools and 10 of 28 (36%) U.K. medical schools. The overall response rate for U.S., Canadian, and U.K. medical schools was 63%. Nine of the 12 responding Canadian schools and eight of the 10 responding U.K. schools used interviews. This was somewhat lower than the 98% of U.S. schools that used interviews. Still, a majority of medical schools use interviews in the selection process. We asked the question, "Do interviews differ for different types of applicants (i.e., academically strong vs. weak applicants)?" All of the Canadian and U.K. schools that responded to this question answered "No." Interviews in the majority of U.S. medical schools did not differ for different types of applicants either. This finding may indicate that admission committees are afraid of legal repercussions if they evaluate subgroups of applicants differently. There is no legal impediment to doing so, as we stated earlier in this article, as long as individuals within subgroups are treated consistently. For example, residents within a defined geographical area may be evaluated differently than residents of other areas. It is likely that medical
10
JANINE C. EDWARDS ET AL.
schools do evaluate subgroups differently, but administrators are aware that the "socially correct" answer is consistency, and, therefore, give that answer on questionnaires. Medical schools may be more constrained, because of unfounded legal fears, in tailoring the interview than they need to be. The structure of the interview is of particular interest as the preceeding sections of this article explain. Of special interest was the question regarding analysis of the characteristics leading to success as a medical student. Of the 12 responding Canadian schools, six (50%) answered that they had analyzed success characteristics, a figure somewhat higher than the U.S. percentage (42%) that had conducted success analyses of medical students (SAMS). Four of the six that had conducted analyses used the educational mission of the institution and course objectives as a basis for their analyses; three had used characteristics identified by other institutions or reported in the literature for their bases. Only one had examined the performance and records of prior successful students to suggest successful characteristics. Of the 10 responding U.K. schools, only one answered that it had analyzed success characteristics; nine (90%) had not done such analyses. Prior to 1990, there were seven reports from medical schools of semistructured interviews in the literature; several of these reports were from United Kingdom medical schools. Nearly all of these studies involved some sort of job analysis or SAMS; however, the process each school used was not described. None of the medical schools reporting semistructured interviews in the literature responded to this questionnaire. Therefore, we know little about how the content for interviews in U.K. schools is determined. It would be interesting and valuable to learn more about the content of the interview in U.K. medical schools. Assessment of language skills can be done quite well in interviews. The vast majority of U.S. medical schools stated that they did assess language skills in the interview. The majority of both Canadian (66%) and U.K. (80%) schools also stated that they assessed language skills. Three specific methods of evaluating language skills were given on the questionnaire and "Other (please explain)" was a fourth option. The respondents were encouraged to circle as many of the four options as applied; therefore, the number of responses totaled more than 100%. Four Canadian schools indicated they used direct rating of language skills and four did not formally assess language although extremely good or poor language skills affected the overall interview rating. Three Canadian schools indicated that their interviewers rated language indirectly. Only one U.K. school assessed language directly; five schools did not formally assess language, but extreme skills in either direction of the continuum would affect interview rating. Three of the ten U.K. responding schools used language as an indication ofprofessionalism or communication skills. The number of interviews granted each applicant is of considerable interest. Interviewing applicants is a time consuming enterprise; faculty members' time is particularly valuable and the logistics of arranging interview sessions is complicated. The majority of U.K. and Canadian schools (70% in each nation) hold only one interview for applicants. This contrasts with U.S. schools, the majority of which
ADMISSION TO MEDICAL SCHOOL: INTERNATIONAL PERSPECTIVES
11
give two interviews to each applicant. This difference may be accounted for by the fact that the majority of Canadian and U.K. schools hold panel interviews instead of one-on-one interviews. Both physician faculty members and Ph.D. faculty members interview applicants in Canada and the U.K. In the schools from the U.K. responding, students, residents, alumni members, staff members, and community representatives did not conduct interviews. In some Canadian schools, all of these types of persons did interview applicants, as they do in many U.S. schools. Structuring the questions asked in interviews has been found to increase the validity and reliability of interviews. Therefore, the extent to which interview questions were standardized was queried. In Canadian schools, questions seem to be standardized more than in U.S. or U.K. schools. Three of the ten Canadian schools indicated that interviewers must ask standard questions; however, additional questions were permitted. Among the 96 U.S. schools responding, only four schools had interviewers ask standardized questions. None of the U.K. schools asked standardized questions; six of the ten schools indicated that interviewers were not regulated about the questions they asked. A series of questions were included on the survey to determine whether or to what extent medical schools had studied their own interview process. Only two Canadian schools had conducted follow-up studies to evaluate the effectiveness of the interview in predicting success in medical training; none of the U.K. schools claimed to have done so. On the other hand, four of 12 (33%) Canadian schools had assessed the interrater reliability of their interviewers. In contrast, only one U.K. school out of ten and 14 of 96 U.S. schools had done so. Because training interviewers holds the promise of increasing validity and reliability, training interested us. Only two of the ten U.K. schools provided training for interviewers. Eight of the twelve (66%) Canadian schools gave training, and 60 of the 96 (almost 66%) U.S. schools gave training to interviewers. In response to the final summary question on the survey about the degree of structure in interviews, the majority of U.S. schools indicated that they conducted loosely or moderately structured interviews. One-third of Canadian schools conducted highly structured interviews; more than half claimed to conduct moderately structured interviews. Seven of the ten U.K. schools described their interviews as loosely structured. Schools could circle more than one response to this question; therefore, the percentages total more than 100%. CONCLUSIONS FROM SURVEY
We can draw a few conclusions about the process of interviewing in Canadian medical schools, but due to the low response rate from U.K. schools, we cannot draw valid conclusions for medical schools in the United Kingdom. The majority of Canadian medical schools use interviews in the selection process. About half of the Canadian schools had analyzed the success characteristics of their med-
12
JANINE C. EDWARDS ET AL.
ical students, a somewhat larger percent than U.S. medical schools. About half of Canadian schools assessed applicants' language skills either directly or indirectly in interviews. Seventy percent of Canadian medical schools give one interview to applicants, but that tends to be a panel interview with several interviewers assessing one applicant. The interviewers in Canada, as in the U.S., include physician faculty members, Ph.D.'s, students, residents, alumni members, staff members, and community representatives. Interview questions were standardized in three of the twelve Canadian schools, a higher incidence of structure than in U.S. or U.K. schools. More Canadian schools had studied interrater reliability than either U.K. or U.S. schools although all numbers were far less than majority. Far more Canadian medical schools provide training to interviewers than do schools in the United States or the United Kingdom. In general, Canadian medical schools seem to be more aware of and also seem to have implemented more of the practices that research indicates will improve the validity and reliability of interviews. Meaningful comparisons between medical schools in the United Kingdom and other medical schools await the gathering of more information. The Admission Process in Problem-Based Learning Programs During the past twenty years, problem-based learning has become a major innovation in United States medical schools. The University of New Mexico, Southern Illinois University, and Harvard University Medical School have all experimented with problem-based learning curricula. A number of other U.S. medical schools have also developed problem-based learning programs, either as an alternate curriculum or as the sole curriculum. McMaster University in Canada is one of the few medical schools in the world to have developed a problem-based learning curriculum as its sole curriculum which it did from its inception. McMaster, therefore, represents problem-based learning in a "pure" state. The 1993-94 academic term was the occasion of the twentyfifth anniversary of the founding of the McMaster University Faculty of Health Sciences. As part of its academic celebration, faculty members and administrators at McMaster "took a look" in a formal way at many of their educational processes. For the first time since its creation, the admission process for the problem-based learning programs in medicine, nursing, and physiotherapy was examined critically. This critical examination of the McMaster admission process provided an occasion to examine the admission processes of several U.S. medical schools having problembased learning curricula as well. A description of the original McMaster admission process and its current changes, as well as descriptions of the admission processes of the New Mexico University School of Medicine, Southern Illinois University School of Medicine, and Harvard Medical School are presented here. Finally, a few comparisons are noted. At McMaster University three health science programs developed parallel admission processes in the 1970's: the undergraduate medical, nursing, and occu-
ADMISSION TO MEDICAL SCHOOL: INTERNATIONAL PERSPECTIVES
13
pational therapy and physical therapy programs. Recently, a midwifery program has been developed with a similar admission process. The first step in the process is screening of academic qualifications. For the undergraduate medical program, students must have a minimum grade point average of 3.0 on a 4.0 scale or a letter grade of "B" in three undergraduate years. A simple (unweighted) and a weighted grade point average are computed; the higher of the two averages is then used in the remainder of the process. The undergraduate medical program uses the grade point average continually in its process; that is, the medical admission committee seeks to select those students who have the highest academic qualifications and the best fit of human qualities as well. The nursing program, however, in recent years has used a "cut score" in its academic screening; once an applicant had satisfied the minimum grade point average, academic qualifications were not considered further in the admission process. The second step in the admission process for all programs is the assessment of the Autobiographical Submission (Auto Sub), a detailed written document unique to McMaster University. A fine-grained assessment, requiring several hours of time for each Auto Sub, is done by several faculty members. Letters of reference are then screened. Applicants are invited to interview based upon the preceding process. In the undergraduate medical program, applicants who are interviewed also are assessed as they participate in a problem-based group simulation. The final step in the admission process is termed "collation," which involves a review of the entire file by faculty members and assignment of a final score to each file. Positions are then offered to those applicants having the highest collation score. During the 1980's the Harvard Medical School developed a "hybrid curriculum" using problem-based learning and other methods as well. The admission process is the same for all applicants to the medical school. The total application is evaluated; the admission committee looks for evidence of integrity, maturity, concern for others, leadership potential, and an aptitude for working with people. Academic excellence is expected. Academic records and the applicant's essay are evaluated. Letters of reference are reviewed, and MCAT scores are screened. On the basis of this entire application, applicants are invited to interview. The admission committee makes the final selection based on a total and comparative appraisal of the applicant's suitability for medicine. The University of New Mexico School of Medicine has two curriculum tracks: the conventional track and the Primary Care Curriculum (PCC), problem-based learning program. Candidates apply for admission to the medical school. If they are admitted and if they desire to enter the PCC, they go through an additional admission process, which includes another interview. The faculty who select students for the problem-based learning program look for those who have a background giving evidence of self-guided experiences; frequently these are older, non-traditional students. Twenty students are selected for the PCC from the total pool of 73 students admitted to the medical school.
14
JANINE C. EDWARDS ET AL.
Admission to the problem-based learning (PBL) curriculum at the Southern Illinois University School of Medicine (SIU) also involves a second process. All candidates use the same application initially. Those candidates who wish to enter the PBL curriculum submit a supplemental application to the PBL curriculum at the same time. The PBL applicants are expected to investigate the PBL curriculum and request a PBL interview. Twenty students are selected each year for PBL based on their knowledge of the PBL curriculum, interpersonal skills and interest in PBL. These twenty students engage in the PBL curriculum during the first two years; the third and fourth years of clinical training are the same for all students. A few comparisons are immediately evident. McMaster and Harvard have one curriculum for all students. McMaster's curriculum is "pure" problem-based learning; Harvard's curriculum contains PBL and traditional curriculum elements as well. Both of these schools have one admission process. Harvard's essay may be compared to McMaster's Autobiographical Submission except that the Auto Sub is more highly structured. New Mexico and SIU both have supplemental admission processes because these schools have two curriculum tracks. The SIU supplemental application bears some resemblance to McMaster's Auto Sub. Southern Illinois University, however, requires applicants to answer only these two questions: describe academic, work, or volunteer experience relevant to small group tutorial process and self-directed learning process whereas McMaster's Auto Sub is much more detailed. Problem based learning curricula occur worldwide, notably at the University of Limburg in Maastricht, The Netherlands, the University of Newcastle, Australia, and Ben Gurion University in Reer-Sheva, Israel. Descriptions of the selection processes for these schools would be an interesting and useful contribution to the literature in medical education.
Conclusions In this article, we have restated some basic facts, conclusions, and recommendations about the interview in the admission process to medical school. Some interesting new information from a survey of admission officers in Canada and the United Kingdom has been presented and comparisons made with United States practices. Unfortunately, the lack of a majority response from the United Kingdom schools leaves us at the present time without a complete picture of admission practices in the U.K. Hopefully, more information from those schools can be gathered and disseminated. Problem based learning curricula have been developing during recent years, and we have taken a partial look at the selection processes for those programs. These international perspectives will whet the appetite, we hope, of medical school faculty members around the world to describe, examine, question, and study selection processes for physicians in training.
ADMISSION TO MEDICAL SCHOOL: INTERNATIONAL PERSPECTIVES
15
References 1. Bingham WVD, Moore BV. How To Interview. New York: Harper & Brothers, Publ, 1941. 2. Edwards JC, Johnson EK, Molidor JB. The interview in the admission process. Acad Med 1990; 65:167-177. 3. McGaghie WC. Qualitative variables in medical school admission. AcadMed 1990; 65:145-149. 4. Kassebaum DG, Szenas PL. The decline and rise of the medical school applicant pool. Acad Med 1995; 70:333-340. 5. Flanagan JC. The critical incident technique. Psych Bul 1954; 51:327-359. 6. Johnson EK. The structured interview: manipulating structuring criteria and the effects on validity, reliability, and practicality. Diss Ab 1990. 7. Linstone HA, Turoff M. The Delphi Method: Techniques and Application. Reading, Mass: Addison-Wesley Publ Co, 1975. 8. Worham JP. Applicationfor the DelphiMethod; A Selected Bibliography, 1980. 9. Norman G, Streiner DL. P DQ Statistics. Toronto: BC Decker Inc, 1986. 10. Burgess MM, Calkins V, Richards JM. The structured interview: a selection device. Psychol Rep 1972; 31:867-877. 11. Calkins EV, Richards JM, McCanse JM, Burgess MM, Willoughby TL. Impact on admission to a school of medicine of an innovation in selection procedures. Psychol Rep 1974; 35:1135-1142. 12. McManus IC, Richards P. Audit of admission to medical school: II - shortlisting and interviews. BrMed J 1984; 289:1288-1290. 13. Powis DA, Neame RI, Bristow T, Murphy LB. The objective structured interview for medical student selection. Br Med J 1988; 296:765-768. 14. Hobfoll SE, Benor DE. Prediction of student clinical performance. Med Educ 1981; 15:231-236. 15. Meredith KE, Dunlap MR, Baker HH. Subjective and objective admissions factors as predictors of clinical clerkship. J Med Educ 1982; 57:743-751. 16. Tarico VS, Altmaier EM, Smith WL, Franken EA, Berbaum KS. Development and validation of an accomplishment interview for radiology residents. J Med Educ 1986; 61:845-847. 17. Ross MW. Predictive value of selection interviews in success in first-year medicine. Med J Australia 1983:423-424. 18. Cronshaw SF, Wiesner WH. The validity of the employment interview: models for research and practice. In The Employment Interview: Theory, Research, and Practice.Ferris GR, Eder RW, eds. Beverly Hills, CA: Sage, 1991. 19. Webster EC. DecisionMaking in the Employment Interview. Montreal: Eagle, 1964. 20. Constantin SW. An investigation of information favorability in the employment interview. J Appl Psycho 1976; 61:743-749. 21. London M, Hakel MD. Effects of applicant stereotypes, order, and information on interview impressions. JAppl Psychol 1974; 59:157-162. 22. London M, Poplawski JR. Effects of performance on stereotype development in performance appraisal and interview contexts. J Appl Psychol 176; 61:199-205. 23. Nunnally JC. Psychometric Theory. New York: McGraw-Hill, 1967. 24. Arvey RD. Unfair discrimination in the employment interview: legal and psychological aspects. Psychol Bull 1979; 86:736-765. 25. Dipboye RI, Arvey RD, Terpstra DE. Sex and physical attractiveness of raters and applicants as determinants of resume evaluation. J Appl Psychol 1977; 62:288-294. 26. Dipboye RL, Fromkin HL, Wilback K. Relative importance of applicant sex, attractiveness, and scholastic standing in evaluation of job applicant resumes. J Appl Psychol 1975; 60:39-43. 27. Cash TF, Gillen B, Bums DS. Sexism and "beautyism" in personal consultant decision making. JApplPsychol 1977; 62:301-310. 28. Gifford R, NG CF. Wilkinson M. Nonverbal cues in the employment interview: links between applicant qualities and interviewer judgments. J Appl Psychol 1985; 70:729-736. 29. McComb KB, Jablin FM. Verbal correlates of interviewer emphathic listening and employment interview outcomes. Comm Monographs 1984; 51:353-371. 30. Webster EC. The Employment Interview: A Social Judgment. Schomberg, Ontario, Canada. SIP Publ, 1982.
16
JANINE C.EDWARDS ET AL.
31. Schuh AJ. Establishing, maintaining and evaluating an interviewer training program. Bull Psychonomic Soc 1981; 17:143-146. 32. Wexley KN, Sanders RE, Yukl GA. Training interviewers to eliminate contrast effects in employment interviews. J Appl Psychol 1973; 57:233-236. 33. Latham GP, Wesley KN, Pursell ED. Training managers to minimize rating errors in the observation of behavior. J Appl Psychol 1975; 60:550-555. 34. Keenan A. The selection interview: candidates' reactions and interviewers' judgments. BrJSoc Clin Psychol 1978; 17:201-209. 35. Johnson EK, Edwards JC. Current practices in admissions interviews at U.S. medical schools. Acad Med 1991; 66:408-412.
3
Admission to Medical School: International Perspectives JANINE C. EDWARDS*, EUGENE K. JOHNSON and JOHN B. MOLIDOR *Medical College of Wisconsin, 8701 Watertown Plank Road, Milwaukee, Wisconsin 53226, U.S.A.
Abstract. Admission to medical school is the goal of many students in many countries. The admission process varies from country to country. In some countries, students compete in an open market to gain a position in medical school. In other countries, "intake" is a more routine, planned beaureaucratic process. Where competition reigns, the interview is an important part of the selection process. The interview has been defined by Bingham and Moore [1] as: A serious conversation directed to a definite purpose other than satisfaction in the conversation itself... We must recognize that not only spoken words, but other means of face-to-face communication also are used. Inflection, qualities of voice, facial expression, glint of the eye, posture, gestures, and general behavior supplement what is said. They all contribute to the purposeful exchange of meanings which is the interview. Faculty members in medical schools interview patients all the time. This type of interview, however, is different from the admission interview conducted for applicants to medical school. Patient interviews are highly patterned and structured to obtain specific information. Interviews of applicants, on the other hand, usually are more open-ended. The psychology of the two types of interviews differ also. Applicants to medical school, if accepted, will in time become colleagues with their interviewers and will have increasing levels of responsibility and respect. Patients, however, will always be dependent upon the physicians who treat them. In this article, we present basic facts, conclusions, and recommendations from a review of literature about the interview [2]. Results of a survey of admission interviews at Canadian and United Kingdom medical schools are presented for the first time, and comparisons with United States interview practices are drawn. Finally, descriptions of the selection process at several medical schools with problem based learning curricula are provided and comparisons are noted. Key words: admission, medical education, problem-based learning
Purposes of Interviewing Interviewing applicants to medical school can serve four purposes: 1) gathering information; 2) making the decision to accept or reject; 3) verifying information provided in the application; 4) recruiting particular applicants. Gathering information may be the most important purpose of interviews. Quantitative information, such as transcripts and background information, is obviously gathered most expeditiously by paper or by computer. The more elusive information, such as motivation, leadership, altruism, and interpersonal skills, referred to by McGaghie [3] as noncognitive information or nonacademic information, is best evaluated in interviews.
4
JANINE C. EDWARDS ET AL.
A serious, face-to-face conversation can probably reveal more non-cognitive information about an applicant than any other form of evaluation. Purposes of the interview may be weighted differently for various groups of applicants. There is no legal impediment to evaluating different subgroups by different criteria because these subgroups are self-selected; that is, their credentials are the basis for inclusion in the subgroup. However, all individuals within a subgroup must be treated consistently. The interview will be most effective if it is tailored to fulfill various purposes. Decision making, of course, is the end result of the application process. Most medical schools make decisions after gathering written and interview information. The written information is usually reviewed first and then invitations to interview are issued on the basis of the written information. Decisions to accept or reject are a combination of all the available information. Verifying information is another important function of interviewing. Checking out the veracity of information provided in personal statements, autobiographical sketches, secondary information materials, and the like can be done during the course of interviews. By linking the interview to the initial review of application materials, the admissions committee can doublecheck or verify the authenticity of what has been presented. Finally, the interview can serve a recruitment function. Many medical schools want to recruit particular types of applicants or to generate good will toward the school. The interview is usually regarded as the centerpiece of an applicant's visit to a medical school. If the applicant is treated well during his/her visit and is encountered with respect and dignity, the school is more likely to attract desirable candidates. For applicants with top academic qualifications, interviewers may wish to focus first on determining whether the person has the necessary personality and character traits to become a physician. After that determination is made positively, the interviewers may wish to focus on recruiting the applicant. For the large pool of average candidates, the interviewers may focus on obtaining non-cognitive information and verifying the written information. Making fine distinctions about character and personality among large numbers of applicants who have similar academic qualifications is a worthwhile endeavor. Ruling out candidates requires a somewhat different process. Interviewers skilled in psychological analysis, such as psychiatrists and psychologists, may be the best interviewers for candidates who give evidence of psychological problems. Any evidence of abnormality may trigger additional evaluation. The interview in the medical school application process will no doubt be used in different ways depending upon the size of the pool of applicants. Kassebaum and Szenas [4] described the decline and rise of the applicant pool in U.S. medical schools during the 1980's and early 1990's. They found that the decline of the applicant pool in the mid-1980's was related to changes in the number and pattern of undergraduate majors and changes in employment conditions for college graduates at that time. More precisely, employment of college graduates increased at
ADMISSION TO MEDICAL SCHOOL: INTERNATIONAL PERSPECTIVES
5
that time; thus, many fewer college graduates were interested in further training because they could obtain profitable jobs immediately after college. This also may explain the phenomenon that fewer applicants who were rejected applied to medical schools a second time. Furthermore, fewer college students were majoring in the biological and physical sciences, "the most common antecedents of medical school applicants." From 1988 until the early 1990's, the number of medical school applicants rose dramatically. During that period, there was deterioration in employment of college graduates and the number of degrees awarded in the biological sciences increased. Repeat applications also increased during this period. Therefore, trends in college majors and employment of college graduates seem to be related to the decline and rise of applications to U.S. medical schools during the past decade. At the current time, the applicant pool in the U.S. is at an all time high. The interview is particularly helpful in choosing among all the academically qualified applicants those who most closely match the desired characteristics or the ethos of the medical school. That requires, however, that the admissions committee have done some careful analysis of the criteria for selection and for interviewing. Structuring the interview is the most productive method of ensuring that the applicants who are most desirable are, in fact, chosen. To summarize then, the interview can serve various purposes. It can be weighted differently as a result of the size of the applicant pool, and different subgroups of the applicant pool may be evaluated in different ways within the interview. All individuals within a subgroup must be treated consistently. Gathering information, verifying information, recruiting individuals, and making the decision to accept or reject are all useful purposes of the interview. Interview Formats
Several formats of interviews are possible, including 1) one-on-one; 2) group; 3) panel; and 4) combination. The one-on-one interview is self-explanatory. The group interview consists of several interviewees and one interviewer. The panel interview is conducted by many interviewers with one applicant. The combination interview might consist, for instance, of a one-on-one interview followed by a panel interview. Types of Interviews Three types of interviews have been identified and researched: structured, semistructured and unstructured. To be considered structured, an interview must meet these criteria: 1) interview content is developed from a job analysis or what we call a Success Analysis of Medical Student (SAMS); 2) the questions are standardized; that is, the same questions are asked of every applicant; 3) sample answers to the questions are provided to the interviewers to help them give consistent ratings; 4) a panel interview is conducted. Interviews that have some, but not all, of these criteria
6
JANINE C. EDWARDS ET AL.
are termed semi-structured. Interviews that have none of these criteria are called unstructured. Research studies during the 1980's indicated that adding structure improves the validity and reliability of interviews. The key to improving the validity of interviews is structuring the content, which is accomplished by conducting a job analysis or Success Analysis of Medical Students (SAMS). A SAMS ensures that all interviewers focus their questions on the agreed upon content or objectives. Two methods of conducting a SAMS are described here - the Critical Incidents Technique and the Delphi Method. Other methods of forming consensus are possible, of course. The Critical Incidents Technique consist of "a set of procedures for collecting direct observations of human behavior in such a way as to facilitate their potential usefulness in solving practical problems and developing broad psychological principles" [5]. To conduct a SAMS using critical incidents, medical school faculty members would reflect on and write critical incidents of medical students. These incidents are typically recollections of actual events, those that reveal successful and unsuccessful behaviors, attitudes, and performance. Approximately 50 to 200 critical incidents are needed to adequately describe the objectives of medical students. After the incidents have been written as descriptions, these are edited for clarity. Then a second group of experts reads the critical incidents and groups them into a few major categories. Any critical incidents on which the experts disagree are discarded. Finally, within each category, the incidents are scaled from high (effective) to low (ineffective) values. The resulting set of categories and critical incidents define the objectives of medical students and provide the content upon which to develop questions in the interview. A study that developed critical incidents for interviewing applicants to medical school was completed by Johnson [6]. The Delphi Method is a method of structuring a group communication process so that a group of individuals, as a whole, deal with a complex problem. In the 1950's the Rand Corporation developed the Delphi Method to forecast technology futures [7, 8]. This method was widely used throughout the 1960's and 1970's as a consensus building method. The Delphi Method is useful when a group is too large to meet in person to carry out a nominal group process or when it would be too inconvenient to do so. More specifically, the Delphi Method is valuable when the problem can benefit from subjective judgment on a collective basis; individuals have diverse expertise, disagreements need refereeing, for individual heterogeneity must be preserved. Medical school faculty members with their time constraints certainly meet these conditions, making the Delphi Method a feasible method of forming consensus among them. The first step in the Delphi Method is to generate ideas through a paper and pencil questionnaire from the group as a whole or from a subgroup. Then the entire group assigns a priority score to each item through ranking or through the use of the Likert scale. Several such rounds of assigning priority are usually necessary before consensus develops about a core of items or objectives.
ADMISSION TO MEDICAL SCHOOL: INTERNATIONAL PERSPECTIVES
7
Another method of improving the validity of interviews is to standardize the questions asked of every applicant. This ensures that the same material is covered in every interview and that one applicant's chance of being accepted is not impaired or boosted by lack of key data or influx of extraneous information. Some faculty members have suggested that standardized questions would soon be memorized and passed on to applicants through "the grapevine." Standardized questions could have enough variations that this "copying" problem would not be likely to occur. Interrater reliability can be improved by providing each interviewer with sample answers for each level of performance for the categories. Sample answers are usually "behavioral anchors," that is, behaviors describing a level of performance. Thus, for the category of interpersonal skills, there might be six levels of performance, with anchors such as "establishes rapport with a variety of people" (high rating) and "frequently alienates people" (low rating). Research studies on the validity and reliability of interviews using classic measurement theory have found that panel interviews yield greater reliability than one-on-one interviews. Panel interviews eliminate much of the interrater variance. More recently, generalizability theory has pointed out that a researcher should attempt to identify all the likely sources of error in a measurement situation rather than trying to minimize sources of error as a panel interview does [9]. Therefore, generalizability theory would encourage us to conduct multiple one-on-one interviews rather than one panel interview in order to arrive at a true measurement of the applicant. If it is not feasible for an admissions committee to conduct multiple one-on-one interviews, then the recommendation to conduct one panel interview is useful. Recommendations from Research Relatively little research has been done on the interview in medical education. We do know, however, that focusing on nonacademic interview data influenced the decision of admission committees to select more applicants with high ratings on attributes such as leadership, motivation, range of interests, and interpersonal skills [10-13]. Some slight evidence exists that the selection interview that emphasizes nonacademic criteria predicts success in clinical training [14-17]. A large body of research in the psychology and business literature, however, has developed a number of sound recommendations about the interview. For many years, psychology researchers studied the various types of bias in what are now referred to as "indirect" or "microanalytic" studies [18]. Webster [19] initiated this line of research because he believed that many microbased social psychological factors moderate the validity of the interview. A number of useful findings resulted from this long line of research. Bias can arise from a number of sources, including, but not limited to, rater tendencies, stereotyping, and interviewer background. The following findings are
supported by research:
8
JANINE C. EDWARDS ET AL.
1. Unfavorable information carries more weight than favorable information [2022]. 2. Rating errors, including the halo effect and distribution errors (leniency, severity, central tendency) can be corrected. These tendencies may be reduced by manipulating scale formats, such as using an even number of points on the scale, by making raters aware of their tendencies or by using statistical corrections [23]. 3. Gender differences exist in interview ratings. Women are rated lower by both male and female interviewers [24-27]. 4. Nonverbal, as well as verbal interactions, influence decisions [28-30]. 5. Interviewers develop a stereotype of a good applicant and then try to match the candidates to the stereotype [30]. 6. An average candidate following several outstanding (or marginal) applicants will be rated worse (or better) in comparison. This is called the "contrast effect" [30]. Several research reports indicate that training interviewers can improve performance in interviewing [30]. Training, of course, does not directly affect the decision to accept or reject applicants into medical school. Schuh [31] describes three dimensions of a good training program: instruction, coaching, and supervised practice. Wexley and colleagues [32] gave workshops to college students, and Latham and colleagues [33] trained corporate managers using these dimensions. Keenan [34] found that interviewers with training were more confident and that candidates were more likely to accept jobs with firms whose interviewers were trained. Recently, psychology researchers have shifted emphasis from the microanalytic variables that had been thought to moderate interview validity to direct models of validity. Interview structure is a major variable that has been found to directly affect validity. A number of studies (both individual studies and meta-analyses) of structured interviews give evidence that validity and reliability are higher than that of semi-structured or unstructured interviews. However, validity and reliability figures for semi-structured interviews are higher than unstructured interviews and even approach those of structured interviews. Review of the data that support these conclusions are presented in a previous publication [2]. In practical terms, this means that admissions committees can achieve greater validity and reliability in their interviews by using only one or two of the criteria for structured interviews, such as performing a SAMS or using behavioral anchors for rating scales. The unstructured interview has considerably less validity and reliability than structured or semi-structured interviews. Several important findings emerged from the review of literature published in 1990, and these continue to be important. 1. Various purposes of the interview may be weighted differently for different groups of applicants. 2. Bias in the interviewing process can be reduced by training interviewers.
ADMISSION TO MEDICAL SCHOOL: INTERNATIONAL PERSPECTIVES
9
3. Adding structure to the interview improves its validity and reliability. 4. Four methods of adding structure have been described: performing a job analysis or SAMS; asking the same questions of all applicants; developing sample answers or behavioral anchors for rating scales; having multiple one-on-one interviews or using panel interviews. Survey of Admission Interviews at Canadian and United Kingdom Medical Schools A survey of admission interviews was conducted in United States, Canadian, and United Kingdom medical schools in late 1989. Information about practices in U.S. medical schools was published in 1991 [35]; the data from Canadian and United Kingdom medical schools are presented here for the first time. METHOD
A questionnaire was developed that asked detailed questions about interview practices based on variables and practices extracted from research on both medical school admission interviews and general selection interviews. A previous survey by Puryear and Lewis also was a source for the questions on this survey. Staff members of the Association of American Medical Colleges (AAMC) and the Student Affairs National Committee on Admissions of the AAMC reviewed the questionnaire and made suggestions for its development. Pilot tests were made of the first draft questionnaire with an admission committee member, an admission committee chair, and a dean of admissions. RESULTS AND DISCUSSION
Data were received from 12 of 16 (75%) Canadian medical schools and 10 of 28 (36%) U.K. medical schools. The overall response rate for U.S., Canadian, and U.K. medical schools was 63%. Nine of the 12 responding Canadian schools and eight of the 10 responding U.K. schools used interviews. This was somewhat lower than the 98% of U.S. schools that used interviews. Still, a majority of medical schools use interviews in the selection process. We asked the question, "Do interviews differ for different types of applicants (i.e., academically strong vs. weak applicants)?" All of the Canadian and U.K. schools that responded to this question answered "No." Interviews in the majority of U.S. medical schools did not differ for different types of applicants either. This finding may indicate that admission committees are afraid of legal repercussions if they evaluate subgroups of applicants differently. There is no legal impediment to doing so, as we stated earlier in this article, as long as individuals within subgroups are treated consistently. For example, residents within a defined geographical area may be evaluated differently than residents of other areas. It is likely that medical
10
JANINE C. EDWARDS ET AL.
schools do evaluate subgroups differently, but administrators are aware that the "socially correct" answer is consistency, and, therefore, give that answer on questionnaires. Medical schools may be more constrained, because of unfounded legal fears, in tailoring the interview than they need to be. The structure of the interview is of particular interest as the preceeding sections of this article explain. Of special interest was the question regarding analysis of the characteristics leading to success as a medical student. Of the 12 responding Canadian schools, six (50%) answered that they had analyzed success characteristics, a figure somewhat higher than the U.S. percentage (42%) that had conducted success analyses of medical students (SAMS). Four of the six that had conducted analyses used the educational mission of the institution and course objectives as a basis for their analyses; three had used characteristics identified by other institutions or reported in the literature for their bases. Only one had examined the performance and records of prior successful students to suggest successful characteristics. Of the 10 responding U.K. schools, only one answered that it had analyzed success characteristics; nine (90%) had not done such analyses. Prior to 1990, there were seven reports from medical schools of semistructured interviews in the literature; several of these reports were from United Kingdom medical schools. Nearly all of these studies involved some sort of job analysis or SAMS; however, the process each school used was not described. None of the medical schools reporting semistructured interviews in the literature responded to this questionnaire. Therefore, we know little about how the content for interviews in U.K. schools is determined. It would be interesting and valuable to learn more about the content of the interview in U.K. medical schools. Assessment of language skills can be done quite well in interviews. The vast majority of U.S. medical schools stated that they did assess language skills in the interview. The majority of both Canadian (66%) and U.K. (80%) schools also stated that they assessed language skills. Three specific methods of evaluating language skills were given on the questionnaire and "Other (please explain)" was a fourth option. The respondents were encouraged to circle as many of the four options as applied; therefore, the number of responses totaled more than 100%. Four Canadian schools indicated they used direct rating of language skills and four did not formally assess language although extremely good or poor language skills affected the overall interview rating. Three Canadian schools indicated that their interviewers rated language indirectly. Only one U.K. school assessed language directly; five schools did not formally assess language, but extreme skills in either direction of the continuum would affect interview rating. Three of the ten U.K. responding schools used language as an indication ofprofessionalism or communication skills. The number of interviews granted each applicant is of considerable interest. Interviewing applicants is a time consuming enterprise; faculty members' time is particularly valuable and the logistics of arranging interview sessions is complicated. The majority of U.K. and Canadian schools (70% in each nation) hold only one interview for applicants. This contrasts with U.S. schools, the majority of which
ADMISSION TO MEDICAL SCHOOL: INTERNATIONAL PERSPECTIVES
11
give two interviews to each applicant. This difference may be accounted for by the fact that the majority of Canadian and U.K. schools hold panel interviews instead of one-on-one interviews. Both physician faculty members and Ph.D. faculty members interview applicants in Canada and the U.K. In the schools from the U.K. responding, students, residents, alumni members, staff members, and community representatives did not conduct interviews. In some Canadian schools, all of these types of persons did interview applicants, as they do in many U.S. schools. Structuring the questions asked in interviews has been found to increase the validity and reliability of interviews. Therefore, the extent to which interview questions were standardized was queried. In Canadian schools, questions seem to be standardized more than in U.S. or U.K. schools. Three of the ten Canadian schools indicated that interviewers must ask standard questions; however, additional questions were permitted. Among the 96 U.S. schools responding, only four schools had interviewers ask standardized questions. None of the U.K. schools asked standardized questions; six of the ten schools indicated that interviewers were not regulated about the questions they asked. A series of questions were included on the survey to determine whether or to what extent medical schools had studied their own interview process. Only two Canadian schools had conducted follow-up studies to evaluate the effectiveness of the interview in predicting success in medical training; none of the U.K. schools claimed to have done so. On the other hand, four of 12 (33%) Canadian schools had assessed the interrater reliability of their interviewers. In contrast, only one U.K. school out of ten and 14 of 96 U.S. schools had done so. Because training interviewers holds the promise of increasing validity and reliability, training interested us. Only two of the ten U.K. schools provided training for interviewers. Eight of the twelve (66%) Canadian schools gave training, and 60 of the 96 (almost 66%) U.S. schools gave training to interviewers. In response to the final summary question on the survey about the degree of structure in interviews, the majority of U.S. schools indicated that they conducted loosely or moderately structured interviews. One-third of Canadian schools conducted highly structured interviews; more than half claimed to conduct moderately structured interviews. Seven of the ten U.K. schools described their interviews as loosely structured. Schools could circle more than one response to this question; therefore, the percentages total more than 100%. CONCLUSIONS FROM SURVEY
We can draw a few conclusions about the process of interviewing in Canadian medical schools, but due to the low response rate from U.K. schools, we cannot draw valid conclusions for medical schools in the United Kingdom. The majority of Canadian medical schools use interviews in the selection process. About half of the Canadian schools had analyzed the success characteristics of their med-
12
JANINE C. EDWARDS ET AL.
ical students, a somewhat larger percent than U.S. medical schools. About half of Canadian schools assessed applicants' language skills either directly or indirectly in interviews. Seventy percent of Canadian medical schools give one interview to applicants, but that tends to be a panel interview with several interviewers assessing one applicant. The interviewers in Canada, as in the U.S., include physician faculty members, Ph.D.'s, students, residents, alumni members, staff members, and community representatives. Interview questions were standardized in three of the twelve Canadian schools, a higher incidence of structure than in U.S. or U.K. schools. More Canadian schools had studied interrater reliability than either U.K. or U.S. schools although all numbers were far less than majority. Far more Canadian medical schools provide training to interviewers than do schools in the United States or the United Kingdom. In general, Canadian medical schools seem to be more aware of and also seem to have implemented more of the practices that research indicates will improve the validity and reliability of interviews. Meaningful comparisons between medical schools in the United Kingdom and other medical schools await the gathering of more information. The Admission Process in Problem-Based Learning Programs During the past twenty years, problem-based learning has become a major innovation in United States medical schools. The University of New Mexico, Southern Illinois University, and Harvard University Medical School have all experimented with problem-based learning curricula. A number of other U.S. medical schools have also developed problem-based learning programs, either as an alternate curriculum or as the sole curriculum. McMaster University in Canada is one of the few medical schools in the world to have developed a problem-based learning curriculum as its sole curriculum which it did from its inception. McMaster, therefore, represents problem-based learning in a "pure" state. The 1993-94 academic term was the occasion of the twentyfifth anniversary of the founding of the McMaster University Faculty of Health Sciences. As part of its academic celebration, faculty members and administrators at McMaster "took a look" in a formal way at many of their educational processes. For the first time since its creation, the admission process for the problem-based learning programs in medicine, nursing, and physiotherapy was examined critically. This critical examination of the McMaster admission process provided an occasion to examine the admission processes of several U.S. medical schools having problembased learning curricula as well. A description of the original McMaster admission process and its current changes, as well as descriptions of the admission processes of the New Mexico University School of Medicine, Southern Illinois University School of Medicine, and Harvard Medical School are presented here. Finally, a few comparisons are noted. At McMaster University three health science programs developed parallel admission processes in the 1970's: the undergraduate medical, nursing, and occu-
ADMISSION TO MEDICAL SCHOOL: INTERNATIONAL PERSPECTIVES
13
pational therapy and physical therapy programs. Recently, a midwifery program has been developed with a similar admission process. The first step in the process is screening of academic qualifications. For the undergraduate medical program, students must have a minimum grade point average of 3.0 on a 4.0 scale or a letter grade of "B" in three undergraduate years. A simple (unweighted) and a weighted grade point average are computed; the higher of the two averages is then used in the remainder of the process. The undergraduate medical program uses the grade point average continually in its process; that is, the medical admission committee seeks to select those students who have the highest academic qualifications and the best fit of human qualities as well. The nursing program, however, in recent years has used a "cut score" in its academic screening; once an applicant had satisfied the minimum grade point average, academic qualifications were not considered further in the admission process. The second step in the admission process for all programs is the assessment of the Autobiographical Submission (Auto Sub), a detailed written document unique to McMaster University. A fine-grained assessment, requiring several hours of time for each Auto Sub, is done by several faculty members. Letters of reference are then screened. Applicants are invited to interview based upon the preceding process. In the undergraduate medical program, applicants who are interviewed also are assessed as they participate in a problem-based group simulation. The final step in the admission process is termed "collation," which involves a review of the entire file by faculty members and assignment of a final score to each file. Positions are then offered to those applicants having the highest collation score. During the 1980's the Harvard Medical School developed a "hybrid curriculum" using problem-based learning and other methods as well. The admission process is the same for all applicants to the medical school. The total application is evaluated; the admission committee looks for evidence of integrity, maturity, concern for others, leadership potential, and an aptitude for working with people. Academic excellence is expected. Academic records and the applicant's essay are evaluated. Letters of reference are reviewed, and MCAT scores are screened. On the basis of this entire application, applicants are invited to interview. The admission committee makes the final selection based on a total and comparative appraisal of the applicant's suitability for medicine. The University of New Mexico School of Medicine has two curriculum tracks: the conventional track and the Primary Care Curriculum (PCC), problem-based learning program. Candidates apply for admission to the medical school. If they are admitted and if they desire to enter the PCC, they go through an additional admission process, which includes another interview. The faculty who select students for the problem-based learning program look for those who have a background giving evidence of self-guided experiences; frequently these are older, non-traditional students. Twenty students are selected for the PCC from the total pool of 73 students admitted to the medical school.
14
JANINE C. EDWARDS ET AL.
Admission to the problem-based learning (PBL) curriculum at the Southern Illinois University School of Medicine (SIU) also involves a second process. All candidates use the same application initially. Those candidates who wish to enter the PBL curriculum submit a supplemental application to the PBL curriculum at the same time. The PBL applicants are expected to investigate the PBL curriculum and request a PBL interview. Twenty students are selected each year for PBL based on their knowledge of the PBL curriculum, interpersonal skills and interest in PBL. These twenty students engage in the PBL curriculum during the first two years; the third and fourth years of clinical training are the same for all students. A few comparisons are immediately evident. McMaster and Harvard have one curriculum for all students. McMaster's curriculum is "pure" problem-based learning; Harvard's curriculum contains PBL and traditional curriculum elements as well. Both of these schools have one admission process. Harvard's essay may be compared to McMaster's Autobiographical Submission except that the Auto Sub is more highly structured. New Mexico and SIU both have supplemental admission processes because these schools have two curriculum tracks. The SIU supplemental application bears some resemblance to McMaster's Auto Sub. Southern Illinois University, however, requires applicants to answer only these two questions: describe academic, work, or volunteer experience relevant to small group tutorial process and self-directed learning process whereas McMaster's Auto Sub is much more detailed. Problem based learning curricula occur worldwide, notably at the University of Limburg in Maastricht, The Netherlands, the University of Newcastle, Australia, and Ben Gurion University in Reer-Sheva, Israel. Descriptions of the selection processes for these schools would be an interesting and useful contribution to the literature in medical education.
Conclusions In this article, we have restated some basic facts, conclusions, and recommendations about the interview in the admission process to medical school. Some interesting new information from a survey of admission officers in Canada and the United Kingdom has been presented and comparisons made with United States practices. Unfortunately, the lack of a majority response from the United Kingdom schools leaves us at the present time without a complete picture of admission practices in the U.K. Hopefully, more information from those schools can be gathered and disseminated. Problem based learning curricula have been developing during recent years, and we have taken a partial look at the selection processes for those programs. These international perspectives will whet the appetite, we hope, of medical school faculty members around the world to describe, examine, question, and study selection processes for physicians in training.
ADMISSION TO MEDICAL SCHOOL: INTERNATIONAL PERSPECTIVES
15
References 1. Bingham WVD, Moore BV. How To Interview. New York: Harper & Brothers, Publ, 1941. 2. Edwards JC, Johnson EK, Molidor JB. The interview in the admission process. Acad Med 1990; 65:167-177. 3. McGaghie WC. Qualitative variables in medical school admission. AcadMed 1990; 65:145-149. 4. Kassebaum DG, Szenas PL. The decline and rise of the medical school applicant pool. Acad Med 1995; 70:333-340. 5. Flanagan JC. The critical incident technique. Psych Bul 1954; 51:327-359. 6. Johnson EK. The structured interview: manipulating structuring criteria and the effects on validity, reliability, and practicality. Diss Ab 1990. 7. Linstone HA, Turoff M. The Delphi Method: Techniques and Application. Reading, Mass: Addison-Wesley Publ Co, 1975. 8. Worham JP. Applicationfor the DelphiMethod; A Selected Bibliography, 1980. 9. Norman G, Streiner DL. P DQ Statistics. Toronto: BC Decker Inc, 1986. 10. Burgess MM, Calkins V, Richards JM. The structured interview: a selection device. Psychol Rep 1972; 31:867-877. 11. Calkins EV, Richards JM, McCanse JM, Burgess MM, Willoughby TL. Impact on admission to a school of medicine of an innovation in selection procedures. Psychol Rep 1974; 35:1135-1142. 12. McManus IC, Richards P. Audit of admission to medical school: II - shortlisting and interviews. BrMed J 1984; 289:1288-1290. 13. Powis DA, Neame RI, Bristow T, Murphy LB. The objective structured interview for medical student selection. Br Med J 1988; 296:765-768. 14. Hobfoll SE, Benor DE. Prediction of student clinical performance. Med Educ 1981; 15:231-236. 15. Meredith KE, Dunlap MR, Baker HH. Subjective and objective admissions factors as predictors of clinical clerkship. J Med Educ 1982; 57:743-751. 16. Tarico VS, Altmaier EM, Smith WL, Franken EA, Berbaum KS. Development and validation of an accomplishment interview for radiology residents. J Med Educ 1986; 61:845-847. 17. Ross MW. Predictive value of selection interviews in success in first-year medicine. Med J Australia 1983:423-424. 18. Cronshaw SF, Wiesner WH. The validity of the employment interview: models for research and practice. In The Employment Interview: Theory, Research, and Practice.Ferris GR, Eder RW, eds. Beverly Hills, CA: Sage, 1991. 19. Webster EC. DecisionMaking in the Employment Interview. Montreal: Eagle, 1964. 20. Constantin SW. An investigation of information favorability in the employment interview. J Appl Psycho 1976; 61:743-749. 21. London M, Hakel MD. Effects of applicant stereotypes, order, and information on interview impressions. JAppl Psychol 1974; 59:157-162. 22. London M, Poplawski JR. Effects of performance on stereotype development in performance appraisal and interview contexts. J Appl Psychol 176; 61:199-205. 23. Nunnally JC. Psychometric Theory. New York: McGraw-Hill, 1967. 24. Arvey RD. Unfair discrimination in the employment interview: legal and psychological aspects. Psychol Bull 1979; 86:736-765. 25. Dipboye RI, Arvey RD, Terpstra DE. Sex and physical attractiveness of raters and applicants as determinants of resume evaluation. J Appl Psychol 1977; 62:288-294. 26. Dipboye RL, Fromkin HL, Wilback K. Relative importance of applicant sex, attractiveness, and scholastic standing in evaluation of job applicant resumes. J Appl Psychol 1975; 60:39-43. 27. Cash TF, Gillen B, Bums DS. Sexism and "beautyism" in personal consultant decision making. JApplPsychol 1977; 62:301-310. 28. Gifford R, NG CF. Wilkinson M. Nonverbal cues in the employment interview: links between applicant qualities and interviewer judgments. J Appl Psychol 1985; 70:729-736. 29. McComb KB, Jablin FM. Verbal correlates of interviewer emphathic listening and employment interview outcomes. Comm Monographs 1984; 51:353-371. 30. Webster EC. The Employment Interview: A Social Judgment. Schomberg, Ontario, Canada. SIP Publ, 1982.
16
JANINE C.EDWARDS ET AL.
31. Schuh AJ. Establishing, maintaining and evaluating an interviewer training program. Bull Psychonomic Soc 1981; 17:143-146. 32. Wexley KN, Sanders RE, Yukl GA. Training interviewers to eliminate contrast effects in employment interviews. J Appl Psychol 1973; 57:233-236. 33. Latham GP, Wesley KN, Pursell ED. Training managers to minimize rating errors in the observation of behavior. J Appl Psychol 1975; 60:550-555. 34. Keenan A. The selection interview: candidates' reactions and interviewers' judgments. BrJSoc Clin Psychol 1978; 17:201-209. 35. Johnson EK, Edwards JC. Current practices in admissions interviews at U.S. medical schools. Acad Med 1991; 66:408-412.
