Teaching and Learning in Medicine, 27(2), 155–162 Copyright Ó 2015, Taylor & Francis Group, LLC ISSN: 1040-1334 print / 1532-8015 online DOI: 10.1080/10401334.2015.1011649
VALIDATION Increasing the Relative Weight of Noncognitive Admission Criteria Improves Underrepresented Minority Admission Rates to Medical School Marlene P. Ballejos and Robert L. Rhyne Department of Family and Community Medicine, University of New Mexico, Albuquerque, New Mexico, USA
Jay Parkes Department of Individual, Family, and Community Education, University of New Mexico, Albuquerque, New Mexico, USA
Construct: The objective of this study was to evaluate the impact of varying the relative weights of cognitive versus noncognitive admission criteria on the proportion of underrepresented minorities admitted to medical school. It answers the question, “Can medical schools increase the admission rates of underrepresented minority (URM) students by balancing cognitive criteria with the experiences, attributes, and metrics of noncognitive data in the admission process?” Background: U.S. demographics are shifting, and by 2042 ethnic minority groups will make up approximately 50% of the population. Increasing diversity of the U.S. population foreshadows the need to increase the number of physicians from underrepresented minorities to help address healthcare disparities that are on the rise. Approach: A cohort of three medical school applicant classes (2007–2009) was used to model the impact on URM admission rates as the relative weights of cognitive and noncognitive admission criteria were varied. This study used the minimum admission standards established for the actual incoming classes. The URM rate of admission to medical school was the outcome. Cognitive criteria included Medical College Admission Test scores and grade point averages. Noncognitive criteria included four categories: background and diversity, interest and suitability for a career in medicine, problem-solving and communication skills, and letters of recommendation. Results: A cohort of 480 applicants from the three applicant classes were enrolled in the study. As the weighting scheme was varied from 50% cognitive/50% noncognitive weights to 35%/65%, the proportion of URM students accepted to medical school increased from 24% (42/177) to 30% (57/193; p < .001). Hispanic and Native American acceptance rates increased by 5.1% and 0.7%, respectively. Conclusions: Admission rates of URM students can be increased by weighting noncognitive higher relative to cognitive criteria without compromising admission standards. Challenging conventional practice in the admissions process may improve health disparities and diversify the physician workforce. Keywords
medical school admissions, selection criteria, underrepresented minorities
Correspondence may be sent to Marlene P. Ballejos, Department of Family and Community Medicine, University of New Mexico, MSC 09 5085 HSLIC Rm 125, Albuquerque, NM 87131-0001, USA. E-mail: [email protected]
INTRODUCTION Healthcare disparities in minority populations are adversely affecting the overall health of the United States.1 Although these health disparities are on the rise, the number of physicians from underrepresented minority (URM) populations is not on a similar trajectory to meet those needs.1,2 According to physician workforce data, only 12% of medical students graduating between 1980 and 2004 were URMs.3 Medical schools do a poor job in admitting and graduating adequate numbers of physicians from URM groups.3 Recent Association of American Medical Colleges (AAMC) data have shown that 94% of students who matriculate into medical school graduate within 5 years. Thus admitting more URMs to medical school would lead to a more diverse physician workforce.4 Research has shown that healthcare professionals from the same racial and ethnic background as their patients have a positive impact on the healthcare of underserved, multiethnic populations due to patient–doctor language concordance, interpersonal care, and increased access to care (see the online supplementary file).5–9 The African American and Hispanic U.S. populations are among the fastest growing and are the most severely underrepresented in the U.S. physician workforce.3 This may be a particularly acute issue in the five U.S. states and regions that are now majority–minority states— Hawaii, California, New Mexico, Texas, and the District of Columbia—where URMs now constitute more than half of the population.10 According to a recent study, 70% of Hispanic and 75% of African American medical students declare their intention to practice in medically underserved communities compared with 58% of non-Hispanic White students and 53% of Asian students.11 Therefore, admitting more URM students into medical school could potentially have a significant impact on minority health disparities and the diversity of the healthcare workforce.12 The challenge for medical schools is to increase the number of qualified URM matriculants without lowering admission
155
156
M. P. BALLEJOS, R. L. RHYNE, J. PARKES
standards. Selecting a diverse class of students to medical school must balance considerations of academic preparation, life experiences, community involvement, abilities, and other achievements.13 The question is whether changing the balance of admission selection criteria to include more emphasis on these noncognitive criteria will increase URM admissions. Many studies related to cognitive psychometric assessment currently exist, and there are an increasing number of studies that address noncognitive psychometric properties with respect to admissions of URMs and future academic performance.14–17 There is a difference of opinion in the literature of holistic review, and questions still remain regarding the validity and reliability of the interview data. One study showed no correlation between an emotional intelligence test and future academic performance.18 However, a recent article compared admission committee performance using holistic methods with independent ratings using component parts of an admission file and found a high association between academic and nonacademic components of the file review (Cronbach’s a D 0.69).19 Another study confirmed that an increased weighting of applicant interview scores increased the acceptance rate of URM applicants.20 The AAMC recently initiated the “holistic review project” to assist medical schools in implementing new admission policies that take into consideration multiple factors, which define an applicant’s whole identity, as opposed to reducing an individual to any single set of factors, like Medical College Admission Test (MCAT) scores and grade point averages (GPAs).13,21 The project supports a holistic review of each applicant, whereby a student’s academic preparation is considered along with other social and environmental obstacles in their path to applying. The holistic approach emphasizes a more balanced consideration of life experiences, personality attributes, and the conventional academic metrics. A recent U.S. Supreme Court case, Fisher v. University of Texas (2013),22 challenged the use of race in the undergraduate admissions process. The justices cautioned schools to use strict scrutiny as they consider race and ethnicity in the admissions process. Under the guidance of the AAMC, medical schools continue to have very individualized admissions processes that consider many different factors in addition to race and ethnicity. This study investigated the following research question: Does changing the relative weight of cognitive and noncognitive criteria in the admissions decision process improve the proportion of URM’s admitted to medical school in a 3-year cohort of medical student applicants? In this study we changed the relative weights of the cognitive and noncognitive criteria and analyzed how the resulting admission rates of URMs changed.
METHODS This study was conducted at the University of New Mexico School of Medicine, an accredited publicly funded medical
school in the southwestern United States, which enrolled 75 students annually at the time of the study. The annual class size has recently been expanded to 100. A historical study cohort was constructed using existing data for applicants from three consecutive admissions cycles (2007–2009). After Human Research Review Committee approval, demographic and academic data were collected from the AAMC American Medical College Application Service application materials, and interview and reviewer data were collected from the school’s admissions database. All identified data were removed from the files before analysis.
Selection of Participants The study period covered three consecutive admission cycles, 2007 through 2009, during which time 2,286 candidates applied to the UNM School of Medicine (1,093 in 2007, 589 in 2008, and 604 in 2009). Our admissions process restricts admission consideration to applicants from New Mexico. In 2007 we began publishing our strict in-state residency requirements in the Medical School Admissions Requirements publication, our printed brochures, website, and elsewhere, in an attempt to decrease total number of out-of-state applicants. This accounted for the decreased number of applicants after 2007. After prescreening, 578 applicants completed a secondary application and were interviewed for the 2007–2009 entering classes (195 in 2007, 198 in 2008, and 185 in 2009). Ninety-eight applicants were excluded from the study (44 in 2007, 29 in 2008, and 25 in 2009) for one or more of the following reasons: (a) they did not respond to race/ethnicity question (URM status was unknown), (b) the applicants were missing a noncognitive subcategory score, and (c) the applicants had fewer or more than two interviews. URM status was identified in accordance with the AAMC Guidelines, which state that being underrepresented in medicine is defined as those racial and ethnic populations that are underrepresented in the medical profession relative to their numbers in the general population; in our case Hispanic, Native American, and African American.8
Admissions Ranking Process This section describes the admissions committee process that occurred during the application years. Initial applications to American Medical College Application Service were prescreened based on state residency, a minimum MCAT score of 22 or higher, a minimum undergraduate GPA (UGPA) of 3.0 or higher, and fulfillment of the prerequisite coursework. Those meeting the screening criteria were invited to complete a secondary application to our medical school and to participate in two personal interviews by admissions committee members. Admission decisions were made based on cognitive criteria, noncognitive scores from the interviews, and review scores from the entire admissions committee membership.
157
MEDICAL SCHOOL ADMISSION CRITERIA FOR MINORITIES
TABLE 1 Description of cognitive and noncognitive medical school admission criteria Cognitive Criteria
Score
Ranking Points
MCAT Score Matrix
30 29 28 27 26 25 24 23 22* 4.00 3.75 3.50 3.25 3.00*
4.00 3.75 3.50 3.25 3.00 2.75 2.50 2.25 2.00 4.00 3.50 3.00 2.50 2.00
GPA Score Matrix
Noncognitive Criteria
Description
Background/Diversity
Factors that have influenced an applicant’s goals and preparation for medicine Quality of early educational environment Socioeconomic status Culture Race and ethnicity Life and work experiences Insight into the depth and understanding of an applicant’s role in the health professional field Clinical and community volunteer experiences Knowledge of unique healthcare needs and issues in New Mexico Critical thinking Decision-making skills Oral and written Nonverbal Listening Emotional awareness Maturity Integrity Ethics Social responsibility Flexible holistic score based on the first four noncognitive categories without a standardized weighting scheme
Interest and Suitability
Problem-Solving and Communication Skills
Letters of Recommendation
Summary Noncognitive Score
*Minimum Medical College Admission Test (MCAT)/grade point average (GPA) scores considered.
The cognitive score was calculated using an applicant’s UGPA and MCAT scores, each of which was converted using a computer program to a point scale between 2.0 and 4.0 on a continuous scale (Table 1). The average was calculated by equally weighting the converted scores. The noncognitive score was based on the personal interviews and written secondary application materials. Each interviewer rated applicants subjectively on a scale from 2.0 to 4.0
in quarter-point increments on each of four noncognitive subcategories: background/diversity, interest and suitability for a career in medicine, problem-solving and communication skills, and letters of recommendation (Table 1). The interrater reliability agreement index was calculated based on the two interviewer scores for each noncognitive subcategory item to check the proportion of time that Interviewer 1 and Interviewer 2 were in agreement. At a 0.50 score difference in
158
M. P. BALLEJOS, R. L. RHYNE, J. PARKES
agreement, the interrater reliability rates were as follows: Interest and Suitability Difference D 92.0, Problem-Solving Difference D 88.2, Background/Diversity Difference D 93.3, Letters of Recommendation Difference D 95.0. Based on the subcategory scores the interviewers then assigned a summary noncognitive score for each applicant, which had to be within the range of the four subcategory scores. Noncognitive summary scores of the two interviewers agreed within a 0.50 range with an interrater reliability of more than 0.9. A composite average score was then calculated by each of the two interviewers assuming 50/50, or equal weighting of the cognitive criteria and noncognitive criteria. The entire admissions committee then reviewed all materials from each applicant, including the written summaries and composite average scores given by the interviewers. Each committee member assigned a noncognitive review score based on all the information available using the two interviewer composite average scores as a guide. All individual committee member review scores were averaged for each applicant and then converted to a percentile final Rank Score, which ultimately determined who was accepted or rejected. Applicants with the top 75 Rank Scores were offered admission to medical school. Because not all admitted students chose to matriculate to our medical school, alternates were identified by going down the list to subsequent Rank Scores. Modeling Study For this study we varied the relative weighting scheme for the noncognitive scores compared to the cognitive scores for the applicant cohort (2007–2009). This allowed us to determine how different weightings would change the main outcome variable, the composition of each class with respect to the proportion of URMs admitted to medical school. The percentile Rank Score of the last applicant admitted to an actual
matriculating class determined the admission cutoff percentile for the simulated study class, or the Admission Index Score (AIS). The AIS for each cohort class was 85.0%, 88.0%, and 85.3%, respectively. Admission decisions for each of the simulation cohort classes were determined using the AIS for the actual class that was admitted, which established the minimum admission standard for the simulated admission cut scores. The relative weights of the cognitive versus the noncognitive scores from the actual admissions process were varied with each simulation. To assure that cognitive standards were not lowered to conduct this study, we retained our school’s minimum thresholds for MCAT/UGPA (MCAT score of 22, UGPA of 3.0) for determining the thresholds of cognitive scores to assure success in medical school and maintain standards for admission. The weights for the noncognitive scores were increased by 5% increments, whereas the weights of the cognitive scores were decreased by 5% for each simulation run, starting with the actual admission standard that was used (50/50). We calculated the proportion of URMs accepted with each weighting scheme until the difference in URM proportion admitted became statistically significantly different from the original admission decisions, using the chi-square test of independence. RESULTS The applicant cohort consisted of 480 medical school applicants; each class had 30% URMs (Hispanic, Native American, or African American) and at least 50% women (Table 2). As expected, each class was approximately one third of the total cohort. The final score, final noncognitive, and final cognitive scores all were statistically significantly different between URMs and non-URMs in the applicant cohort. Specifically, URMs had higher final noncognitive mean scores (M D 3.64, SD D .23) than non-URMs (M D 3.44, SD D .27; p < .05), whereas non-URMs had higher final cognitive mean scores
TABLE 2 Demographics of medical school applicant cohort: 2007–2009 Applicant Cohort by Admission Year Characteristics Gender Female Male Total Race/Ethnicity Hispanic Native American African American White Asian Total
2007 N (%)
2008 N (%)
2009 N (%)
Total N (%)
82 (54) 69 (46) 151 (31)
84 (50) 85 (50) 169 (35)
87 (54) 73 (46) 160 (33)
253 (53) 227 (47) 480 (100)
39 (26) 3 (2) 3 (2) 87 (58) 19 (13) 151
36 (21) 11 (7) 4 (2) 103 (61) 15 (9) 169
37 (23) 9 (6) 2 (1) 98 (61) 14 (9) 160
112 (23) 23 (5) 9 (2) 288 (60) 48 (10) 480
159
MEDICAL SCHOOL ADMISSION CRITERIA FOR MINORITIES
TABLE 3 Medical school admission results by weighting scheme for URMs and non-URMs Admissions Decision Weighting Scheme (% / %) 50 cognitive/50 noncognitive Non-URM URM Total 45 cognitive/55 noncognitive Non-URM URM Total 40 cognitive/60 noncognitive Non-URM URM Total 35 cognitive/65 noncognitive Non-URM URM Total
No N (%)
Yes N (%)
Total N
Chi-Squarea df D 1
201 (66%) 102 (34%) 303 (100)
135 (76%) 42 (24%) 177 (100)
336 144 480
Reference
207 (68%) 96 (32%) 303 (100)
129 (73%) 48 (27%) 177 (100)
336 144 480
1.70, p D .20
201 (69%) 92 (31%) 293 (100)
135 (72%) 52 (28%) 187 (100)
336 144 480
3.40, p D .10
200 (70%) 87 (30%) 287 (100)
136 (70%) 57 (30%) 193 (100)
336 144 480
7.58, p < .001y
a Results from the chi-square test of independence used to test the proportion of admitted students who were underrepresented minorities (URMs; N D 144) in each weighting scheme against expected values from the 50 cognitive/50 noncognitive weighting scheme. y Statistically significant at Bonferroni corrected a D .01667.
(M D 3.26, SD D .47) than URMs (M D 2.93, SD D .52; p < .05), which has been substantiated in the literature.23 After assembling the admission cohort we analyzed the resultant cohort classes for differences in the demographic composition between classes. There were no statistically significant differences in the demographic variables between the classes (Table 2); therefore we pooled the data from all classes for the remainder of the modeling analyses. Table 3 shows the total number of admissions for the three simulated classes by weighting scheme for URMs and non-
URMs. Using the 50/50 weighing scheme as a reference, the next two levels of Cognitive %/Noncognitive % criteria (45/55 and 40/60) were not statistically significantly different in the proportion of URM students accepted using the original AIS as the standard for admission. The 35/65 level did show a statistically significant difference (p < .001) compared to the 50/50 level. The admission rates for URMs increased slightly with each 5% increase in noncognitive weighting and became statistically significantly different at the 35/65 level (Table 3). The proportion of URMs increased from 24% to 30%, at the 35/65
TABLE 4 Demographics of 2007–2009 medical school admission cohorts by weighting scheme
% Cognitive/% Noncognitive Admission Criteria Total Admitted of 480 Applicants Race/Ethnicity Hispanic Native American White Asian Gender Male Female
N (%) 50/50
N (%) 45/55
N (%) 40/60
N (%) 35/65
177 (36.9)
177 (36.9)
187 (39.0)
193 (40.2)
36 (20.3) 6 (3.4) 118 (66.7) 17 (9.6)
42 (23.7) 6 (3.4) 112 (63.3) 17 (9.6)
45 (24.1) 7 (3.7) 117 (62.6) 18 (9.6)
49 (25.4) 8 (4.1) 120 (62.2) 16 (8.3)
87 (49.2) 90 (50.8)
89 (50.3) 88 (49.7)
92 (49.2) 95 (50.8)
96 (49.7) 97 (49.7)
160
M. P. BALLEJOS, R. L. RHYNE, J. PARKES
TABLE 5 Medical school admission cohorts by cognitive and noncognitive admissions criteria
MCAT Score Lowest Score GPA Lowest Score Final Cognitive Final Noncognitive Final Average Score
M (SD) 50/50
M (SD) 45/55
M (SD) 40/60
M (SD) 35/65
30.7 (3.3) 23 3.75 (0.2) 3.04 3.7 (0.2) 3.6 (0.2) 3.6 (0.1)
30.5 (3.4) 22 3.73 (.02) 3.04 3.6 (0.3) 3.6 (0.2) 3.6 (0.1)
30.0 (3.6) 22 3.73 (.02) 3.04 3.6 (0.3) 3.6 (0.2) 3.6 (.01)
29.8 (3.7) 22 3.71 (0.2) 3.03 3.5 (0.3) 3.7 (0.2) 3.6 (0.1)
level. The number of URMs admitted went from 42 for the 50/ 50 scheme to 57 for the 35/65 scheme—an increase of 15, or 6%. Table 4 describes the combined resultant admission cohort for the three simulated classes that would be admitted using each weighting scheme as the admission scoring criteria. The total numbers admitted for the three classes do not reach the class size of 75, because of the 98, or 17%, who were excluded from the analysis. The proportion of the applicant cohort (n D 480) admitted using the 40/60 and 35/65 schemes increased from 37% to 39% and 40%, which increased the class size by 10 and 16 students, respectively. Hispanic acceptance rates increased steadily to 5.1% over all weighting schemes, admitting 13 more students when the 35/65 weighting scheme was used. The Native American rates increased only by 0.7%, admitting only two more students; which may be explained by the small Native American applicant pool (Table 2). Conversely, rates for White students admitted decreased by 4.5%. The proportions of female students essentially remained the same. Although Table 2 indicates that there were nine African American applicants over the 2007–2009 cohorts, they are not represented in the race/ethnicity section of Table 4. They were excluded from this study due to one of the following exclusion factors: (a) missing noncognitive subcategory score, (b) had fewer or greater than two interviews, or (c) failed to meet the minimum MCAT and/or GPA thresholds. Overall the mean MCAT scores, GPAs, final cognitive scores, and final noncognitive scores did not change as the cognitive weighting decreased (Table 5). The AAMC considers an MCAT total score of plus or minus 2 points as falling within the 68% confidence interval and may not be meaningfully different from each other.4 The differences observed in Table 5 for the minimum MCAT and GPA scores were not statistically different and the minimum admissions criteria maintained for each class admitted.
DISCUSSION Although Hispanics, Native Americans, and African Americans comprised more than 25% of the nation’s population in
2007, these minorities were less than 9% of the physician workforce.9 Research has shown that healthcare providers with the same racial and ethnic background as their patients have a positive impact on underserved, multiethnic patients.5–9 Educating and graduating physicians from URMs will likely increase the physician workforce serving those populations. Medical schools can contribute to building the minority workforce by admitting more URM students,3 but minimum scholarly standards must be maintained in the admissions process. This study tests an alternative method of weighting cognitive versus noncognitive criteria in the medical student admissions process in an attempt to diversify incoming medical student classes and eventually, the U.S. healthcare workforce. URMs are not admitted to medical schools in sufficient numbers to address health disparities in minority populations, in part because they are underrepresented in the applicant pool and in part because, on average, academic qualifications of URM applicants lag behind those of their majority peers. However, it is not simply a lack of the number of minority applicants but also a problem of academic quality of URMs. In addition, the current admission process arguably focuses disproportionately on cognitive scores and academic achievement versus noncognitive criteria.24 Our study shows that by changing the relative weighting criteria more toward noncognitive versus cognitive criteria, medical schools can modestly increase the proportion of URMs without lowering admission standards. Changing admissions processes toward admitting more well-rounded, diverse students will require schools to examine their admissions practices and processes and challenge the status quo.25 More medical schools are now seeking applicants who not only have strong academic backgrounds but also have a solid foundation in the social determinants of health and are able to relate to a variety of different patients from diverse backgrounds. There have been recent calls for balancing the medical school admission criteria toward a holistic admissions process using “other academic, personal and experiential credentials.”12 In fact, the newly redesigned MCAT will now include psychological and social foundations of behavior as well as the critical analysis and reasoning skills that support other criteria that are important in the holistic review process.26 Using three years of applicant and medical school admissions data, our simulation study found that placing more emphasis on noncognitive criteria increased the URM acceptance rates. All students met the minimum criteria for admission (Table 5). When the relative weight of noncognitive criteria was increased from 50% to 65%, URM admission rates increased from 24% to 30% (Table 3). Although an increase of 6% in URM admissions may seem small, it would have resulted in an increase of 15 more URM students being admitted over the 3-year admissions period. The impact of having an average of five more URM students per class could potentially increase the minority physician workforce practicing in medically underserved, rural areas such as New Mexico.
MEDICAL SCHOOL ADMISSION CRITERIA FOR MINORITIES
The challenge in selecting a more diverse class of students is to ensure that all students have a strong academic foundation and a demonstrated set of noncognitive skills that would lead to successful completion of the medical school curriculum. Our modeling study retained the minimum criteria for admissions that our admissions committee used to admit students over the three years of the study, an MCAT score of 22 or higher and a UGPA of 3.0 or higher (Table 5). This allowed us to vary the relative balance between cognitive and noncognitive criteria without sacrificing our school’s standards for admission. Traditionally, non-URMs have performed higher in the cognitive criteria (MCATs and UGPAs) due to environmental, socioeconomic, and educational factors that relate to academic achievement.27 URMs tend to have a greater likelihood of coming from a rural or underserved area, being a firstgeneration college student, coming from a single-parent household, having English as a second language, and experiencing educational and financial disadvantages.27 Indeed, in our study URMs had higher overall noncognitive scores (M D 3.6 for URMs vs. 3.44 for non-URMs; p < .05). Admission committee members knew the URM status when they scored the noncognitive criteria, which may have contributed to the higher scores for URMs (Table 1), but URM status was not a specifically scored category. The Hispanic population, which is the fastest growing and among the most severely underrepresented in medicine, was most impacted by the changed admissions criteria. The proportion of Hispanic students admitted to medical school increased from 20.3% to 25.4% as the relative weight of noncognitive criteria was increased from 50% to 65%. An important limitation of these data is that the African American population is not represented because of their low prevalence in New Mexico. The literature suggests that caution is needed in making complex decisions about how to appropriately weight composite scores, such as MCAT/GPA with noncognitive criteria on group differences.28 In addition, some have questioned the validity and reliability of composite scores representing a holistic review process.29 The data from this medical school may not be generalizable to other medical schools, although these results may be most applicable to the other majority– minority states. The context, mission, and goals of each medical school need to be considered. In addition, this and previous studies have used the categories of cognitive and noncognitive criteria in the admissions process, but the new holistic review process supports the use of “experiences, attributes and academic metrics” as the recommended categories for medical school admission processes.12 The authors recognize that there are other limitations to this study. The raters’ knowledge of URM status when assessing noncognitive variables could possibly confound the study. However, knowledge of URM status is part of the noncognitive holistic review, which the AAMC considers to be a plus factor for selection if it supports the medical school mission. Our ranking
161
system assigned a maximum score of 4.0 for all MCAT scores of 30 and above (Table 1) and a minimum score based on a preset minimum standard for MCAT scores. The predominance of 4.0 assigned MCAT scores in the applicant pool (due to the high proportion of 30-plus MCAT scores) constrained the average MCAT score so that it was unlikely to change with the addition of a small number of lower MCAT scores as the weighting criteria changed (Table 5). This study does not include outcomes of the URM students, because this was a modeling study of the admissions process. Future studies should include URM graduation rates, performance in medical school, and the practice location of graduates. It is possible that the study results could have been biased if the students who were excluded because they didn’t respond to the race/ethnicity question were systematically different from those who were included in the study. We did not study the excluded students because their race/ethnicity was unknown; therefore, the results assume that those excluded had the same distribution as those included. As the trend toward diversifying the physician workforce continues to grow, all U.S. medical schools are now mandated to consider noncognitive characteristics beyond academic or cognitive ability in selecting applicants for medical school.30 The AAMC has charged medical schools to think differently about admissions and to incorporate an applicant’s “experiences, attributes and metrics” into the screening, application and selection processes.12 These admissions issues have become paramount for most medical schools that are struggling to increase diversity related goals in order to address racial and ethnic health disparities. Questions about filtering out applicants based solely on their academic performance have recently been posed, because this practice may have future ramifications on the healthcare workforce.31 As the AAMC suggests, perhaps we should begin to screen, interview, and select applicants based on the mission of the individual school and the social obligation to serve the surrounding communities and populations and not just cognitive criteria.2,32 This study was conducted at a state-funded medical school in a majority–minority state, which has a particular responsibility to the many diverse communities across the state. Finding balance between diversity and objectivity is a key component of the complexities of reweighting the admissions formula for this and other medical schools as demographic changes in the U.S. magnify the importance of caring for rapidly growing minority populations. Recent research confirms that minority status remains a strong predictor for practicing in underserved areas.33 The present study shows that diversityrelated initiatives could work and supports the notion that qualified URM applicants can be admitted to medical school in larger numbers.
SUPPLEMENTAL DATA Supplemental data for this article can be accessed on the publisher’s website at http://dx.doi.org/10.1080/ 10401334.2015.1011649.
162
M. P. BALLEJOS, R. L. RHYNE, J. PARKES
REFERENCES 1. 2011 National healthcare quality and disparities reports. Rockville, MD: Agency for Healthcare Research and Quality, 2012. Available at: http:// www.ahrq.gov/research/findings/nhqrdr/nhqrdr11/qrdr11.html. Accessed June 26, 2013. 2. Mullan F, Chen C, Petterson S, Kolsky G, Spagnola M. The social mission of medical education: Ranking the schools. Annals of Internal Medicine 2010;152:804–11. 3. America needs a more diverse physician workforce. Washington, DC: Association of American Medical Colleges, 2006. Available at: https:// www.aamc.org/download/87306/data/physiciandiversityfacts.pdf. Accessed January 28, 2010. 4. Using MCAT data in medical student selection. Washington, DC: Association of American Medical Colleges, 2012. Available at: https://www. aamc.org/students/download/267622/data/mcatstudentselectionguide.pdf. Accessed June 21, 2012. 5. Saha S, Shipman S. The rationale for diversity in the health professions: A review of the evidence. Washington, DC: U.S. Department of Health and Human Services, Health Resources Administration, Bureau of Health Professions, 2006. 6. Saha S, Shipman S. Race-neutral versus race-conscious workforce policy to improve access to care. Health Affairs 2008;27:234–45. 7. Smedley BD, Stith AY, Nelson AR (Eds.). Unequal treatment: Confronting racial and ethnic disparities in healthcare. Washington, DC: Institute of Medicine of the National Academies Press, 2003. 8. Coleman AL, Palmer SR, Winnick SY. Roadmap to diversity: Key legal and educational policy foundations for medical schools. Washington, DC: Association of American Medical Colleges, 2008. 9. American College of Physicians. Racial and ethnic disparities in health care. Philadelphia, PA: American College of Physicians, 2010. Available at: http://www.acponline.org/advocacy/current_policy_papers/ assets/racial_disparities.pdf. Accessed June 17, 2013. 10. Yen, H. Census shows whites lose US Majority among babies.Associated Press, 2011. Available at: http://news.yahoo.com/census-shows-whites-loseus-majority-among-babies-193257646.html. Accessed January 28, 2011. 11. Grbic D, Slapar F. Changes in medical students’ intentions to serve the underserved: Matriculation to graduation. Analysis in Brief. Washington, DC: Association of American Medical Colleges, 2010. 12. White CB, Day EL, Fantone JC. Analysis of factors that predict clinical performance in medical school. Advances in Health Sciences Education 2009;14:455–64. 13. Addams AN, Bletzinger RB, Sondheimer HM, White SE, Johnson LM. Roadmap to diversity: Integrating holistic review practices into medical school admission processes. Washington, DC: Association of American Medical Colleges, 2010. 14. Kulatunga-Moruzi C, Norman GR. Validity of admissions measures in predicting performance outcomes: The contribution of cognitive and noncognitive dimensions. Teaching and Learning in Medicine 2002;14:34– 42. 15. Kreiter CD, Yin P, Solow C, Brennan RL. Investigating the reliability of the medical school admissions interview. Advances in Health Sciences Education 2004;9:147–59.
16. Albanese MA, Snow MH, Skochelak SE, Huggett KN, Farrell PN. Assessing personal qualities in medical school admissions. Academic Medicine 2003;78:313–21. 17. Kreiter C. A proposal for evaluating the validity of holistic-based admission processes. Teaching and Learning in Medicine 2013;25:103–7. 18. Humprhrey-Murto S. Does emotional intelligence at medical school admission predict future academic performance? Academic Medicine 2014;89:1–6. 19. Hanson MD, Kulasegaram KM, Coombs DL , Herold J. Admissions file review: Applying the multiple independent sampling (MIS) methodology. Academic Medicine 2012;87:1335–40. 20. Edwards JC, Maldonado FG, Calvin JA. The effects of differently weighting interview scores on the admission of underrepresented minority medical students. Academic Medicine 1999;74:59–61. 21. Steinecke A, Beaudreau J, Bletzinger RB, Terrell C. Race-neutral admission approaches: Challenges and opportunities for medical schools. Academic Medicine 2007;82:117–26. 22. Fisher v. University of Texas at Austin. 132 US 1536(2012). 570 U.S. 23. Cohen GL, Garcia J, Apfel N, Master A. Reducing the racial achievement gap: A social-psychological intervention. Science 2006;313:1307–10. 24. Dunleavy D, Sondheimer H, Castillo-Page L, Bletzinger RB. Medical school admissions: More than grades and test scores. Analysis in Brief. Washington, DC: Association of American Medical Colleges, 2011. 25. Geller G. Tolerance for ambiguity: An ethics-based criterion for medical school selection. Academic Medicine 2013;88:581–4. 26. Schwartzstein RM, Rosenfeld GC, Hilborn R, Herndon Oyewhole S, Mitchell, K. Redesigning the MCAT Exam: Balancing multiple perspectives. Academic Medicine 2013;88:560–7. 27. Davis D, Dorsey K, Franks RD, Sackett PR, Searcy CA, Zhao X. Do racial and ethnic group differences in performance on the MCAT exam reflect test bias? Academic Medicine 2013;88:593–602. 28. Sackett PR, Ellingson JE. The effects of forming multi-predictor composites on group differences and adverse impact. Personnel Psychology 1997;50:707–22. 29. Kreiter CD. A measurement perspective on affirmative action in U.S. medical education, 3rd ed. Med Educ Online: April 2013. Available at: http:// www.ncbi.nlm.nih.gov/pmc/articles/PMC3623946/. Accessed December 3, 2013. 30. Liaison Committee on Medical Education. Accreditation standards. Washington, DC: Association of American Medical Colleges and the American Medical Association. Available at: http://www.lcme.org/stan dard.htm. 31. Association of American Medical Colleges. Recent studies and reports on physician shortages in the US. Washington, DC: Association of American Medical Colleges. Available at: https://www.aamc.org/download/100598/ data/. Accessed October 25, 2012. 32. Kirch DG, Prescott JE. From rankings to mission. Academic Medicine 2013;88:1064–1066. 33. Wayne SJ, Kalishman S, Jerabek RN, Timm C, Cosgrove E. Early predictors of physicians’ practice in medically underserved communities: A 12year follow-up study of University of New Mexico School of medicine graduates. Academic Medicine 2010;85:513–6.
Copyright of Teaching & Learning in Medicine is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
VALIDATION Increasing the Relative Weight of Noncognitive Admission Criteria Improves Underrepresented Minority Admission Rates to Medical School Marlene P. Ballejos and Robert L. Rhyne Department of Family and Community Medicine, University of New Mexico, Albuquerque, New Mexico, USA
Jay Parkes Department of Individual, Family, and Community Education, University of New Mexico, Albuquerque, New Mexico, USA
Construct: The objective of this study was to evaluate the impact of varying the relative weights of cognitive versus noncognitive admission criteria on the proportion of underrepresented minorities admitted to medical school. It answers the question, “Can medical schools increase the admission rates of underrepresented minority (URM) students by balancing cognitive criteria with the experiences, attributes, and metrics of noncognitive data in the admission process?” Background: U.S. demographics are shifting, and by 2042 ethnic minority groups will make up approximately 50% of the population. Increasing diversity of the U.S. population foreshadows the need to increase the number of physicians from underrepresented minorities to help address healthcare disparities that are on the rise. Approach: A cohort of three medical school applicant classes (2007–2009) was used to model the impact on URM admission rates as the relative weights of cognitive and noncognitive admission criteria were varied. This study used the minimum admission standards established for the actual incoming classes. The URM rate of admission to medical school was the outcome. Cognitive criteria included Medical College Admission Test scores and grade point averages. Noncognitive criteria included four categories: background and diversity, interest and suitability for a career in medicine, problem-solving and communication skills, and letters of recommendation. Results: A cohort of 480 applicants from the three applicant classes were enrolled in the study. As the weighting scheme was varied from 50% cognitive/50% noncognitive weights to 35%/65%, the proportion of URM students accepted to medical school increased from 24% (42/177) to 30% (57/193; p < .001). Hispanic and Native American acceptance rates increased by 5.1% and 0.7%, respectively. Conclusions: Admission rates of URM students can be increased by weighting noncognitive higher relative to cognitive criteria without compromising admission standards. Challenging conventional practice in the admissions process may improve health disparities and diversify the physician workforce. Keywords
medical school admissions, selection criteria, underrepresented minorities
Correspondence may be sent to Marlene P. Ballejos, Department of Family and Community Medicine, University of New Mexico, MSC 09 5085 HSLIC Rm 125, Albuquerque, NM 87131-0001, USA. E-mail: [email protected]
INTRODUCTION Healthcare disparities in minority populations are adversely affecting the overall health of the United States.1 Although these health disparities are on the rise, the number of physicians from underrepresented minority (URM) populations is not on a similar trajectory to meet those needs.1,2 According to physician workforce data, only 12% of medical students graduating between 1980 and 2004 were URMs.3 Medical schools do a poor job in admitting and graduating adequate numbers of physicians from URM groups.3 Recent Association of American Medical Colleges (AAMC) data have shown that 94% of students who matriculate into medical school graduate within 5 years. Thus admitting more URMs to medical school would lead to a more diverse physician workforce.4 Research has shown that healthcare professionals from the same racial and ethnic background as their patients have a positive impact on the healthcare of underserved, multiethnic populations due to patient–doctor language concordance, interpersonal care, and increased access to care (see the online supplementary file).5–9 The African American and Hispanic U.S. populations are among the fastest growing and are the most severely underrepresented in the U.S. physician workforce.3 This may be a particularly acute issue in the five U.S. states and regions that are now majority–minority states— Hawaii, California, New Mexico, Texas, and the District of Columbia—where URMs now constitute more than half of the population.10 According to a recent study, 70% of Hispanic and 75% of African American medical students declare their intention to practice in medically underserved communities compared with 58% of non-Hispanic White students and 53% of Asian students.11 Therefore, admitting more URM students into medical school could potentially have a significant impact on minority health disparities and the diversity of the healthcare workforce.12 The challenge for medical schools is to increase the number of qualified URM matriculants without lowering admission
155
156
M. P. BALLEJOS, R. L. RHYNE, J. PARKES
standards. Selecting a diverse class of students to medical school must balance considerations of academic preparation, life experiences, community involvement, abilities, and other achievements.13 The question is whether changing the balance of admission selection criteria to include more emphasis on these noncognitive criteria will increase URM admissions. Many studies related to cognitive psychometric assessment currently exist, and there are an increasing number of studies that address noncognitive psychometric properties with respect to admissions of URMs and future academic performance.14–17 There is a difference of opinion in the literature of holistic review, and questions still remain regarding the validity and reliability of the interview data. One study showed no correlation between an emotional intelligence test and future academic performance.18 However, a recent article compared admission committee performance using holistic methods with independent ratings using component parts of an admission file and found a high association between academic and nonacademic components of the file review (Cronbach’s a D 0.69).19 Another study confirmed that an increased weighting of applicant interview scores increased the acceptance rate of URM applicants.20 The AAMC recently initiated the “holistic review project” to assist medical schools in implementing new admission policies that take into consideration multiple factors, which define an applicant’s whole identity, as opposed to reducing an individual to any single set of factors, like Medical College Admission Test (MCAT) scores and grade point averages (GPAs).13,21 The project supports a holistic review of each applicant, whereby a student’s academic preparation is considered along with other social and environmental obstacles in their path to applying. The holistic approach emphasizes a more balanced consideration of life experiences, personality attributes, and the conventional academic metrics. A recent U.S. Supreme Court case, Fisher v. University of Texas (2013),22 challenged the use of race in the undergraduate admissions process. The justices cautioned schools to use strict scrutiny as they consider race and ethnicity in the admissions process. Under the guidance of the AAMC, medical schools continue to have very individualized admissions processes that consider many different factors in addition to race and ethnicity. This study investigated the following research question: Does changing the relative weight of cognitive and noncognitive criteria in the admissions decision process improve the proportion of URM’s admitted to medical school in a 3-year cohort of medical student applicants? In this study we changed the relative weights of the cognitive and noncognitive criteria and analyzed how the resulting admission rates of URMs changed.
METHODS This study was conducted at the University of New Mexico School of Medicine, an accredited publicly funded medical
school in the southwestern United States, which enrolled 75 students annually at the time of the study. The annual class size has recently been expanded to 100. A historical study cohort was constructed using existing data for applicants from three consecutive admissions cycles (2007–2009). After Human Research Review Committee approval, demographic and academic data were collected from the AAMC American Medical College Application Service application materials, and interview and reviewer data were collected from the school’s admissions database. All identified data were removed from the files before analysis.
Selection of Participants The study period covered three consecutive admission cycles, 2007 through 2009, during which time 2,286 candidates applied to the UNM School of Medicine (1,093 in 2007, 589 in 2008, and 604 in 2009). Our admissions process restricts admission consideration to applicants from New Mexico. In 2007 we began publishing our strict in-state residency requirements in the Medical School Admissions Requirements publication, our printed brochures, website, and elsewhere, in an attempt to decrease total number of out-of-state applicants. This accounted for the decreased number of applicants after 2007. After prescreening, 578 applicants completed a secondary application and were interviewed for the 2007–2009 entering classes (195 in 2007, 198 in 2008, and 185 in 2009). Ninety-eight applicants were excluded from the study (44 in 2007, 29 in 2008, and 25 in 2009) for one or more of the following reasons: (a) they did not respond to race/ethnicity question (URM status was unknown), (b) the applicants were missing a noncognitive subcategory score, and (c) the applicants had fewer or more than two interviews. URM status was identified in accordance with the AAMC Guidelines, which state that being underrepresented in medicine is defined as those racial and ethnic populations that are underrepresented in the medical profession relative to their numbers in the general population; in our case Hispanic, Native American, and African American.8
Admissions Ranking Process This section describes the admissions committee process that occurred during the application years. Initial applications to American Medical College Application Service were prescreened based on state residency, a minimum MCAT score of 22 or higher, a minimum undergraduate GPA (UGPA) of 3.0 or higher, and fulfillment of the prerequisite coursework. Those meeting the screening criteria were invited to complete a secondary application to our medical school and to participate in two personal interviews by admissions committee members. Admission decisions were made based on cognitive criteria, noncognitive scores from the interviews, and review scores from the entire admissions committee membership.
157
MEDICAL SCHOOL ADMISSION CRITERIA FOR MINORITIES
TABLE 1 Description of cognitive and noncognitive medical school admission criteria Cognitive Criteria
Score
Ranking Points
MCAT Score Matrix
30 29 28 27 26 25 24 23 22* 4.00 3.75 3.50 3.25 3.00*
4.00 3.75 3.50 3.25 3.00 2.75 2.50 2.25 2.00 4.00 3.50 3.00 2.50 2.00
GPA Score Matrix
Noncognitive Criteria
Description
Background/Diversity
Factors that have influenced an applicant’s goals and preparation for medicine Quality of early educational environment Socioeconomic status Culture Race and ethnicity Life and work experiences Insight into the depth and understanding of an applicant’s role in the health professional field Clinical and community volunteer experiences Knowledge of unique healthcare needs and issues in New Mexico Critical thinking Decision-making skills Oral and written Nonverbal Listening Emotional awareness Maturity Integrity Ethics Social responsibility Flexible holistic score based on the first four noncognitive categories without a standardized weighting scheme
Interest and Suitability
Problem-Solving and Communication Skills
Letters of Recommendation
Summary Noncognitive Score
*Minimum Medical College Admission Test (MCAT)/grade point average (GPA) scores considered.
The cognitive score was calculated using an applicant’s UGPA and MCAT scores, each of which was converted using a computer program to a point scale between 2.0 and 4.0 on a continuous scale (Table 1). The average was calculated by equally weighting the converted scores. The noncognitive score was based on the personal interviews and written secondary application materials. Each interviewer rated applicants subjectively on a scale from 2.0 to 4.0
in quarter-point increments on each of four noncognitive subcategories: background/diversity, interest and suitability for a career in medicine, problem-solving and communication skills, and letters of recommendation (Table 1). The interrater reliability agreement index was calculated based on the two interviewer scores for each noncognitive subcategory item to check the proportion of time that Interviewer 1 and Interviewer 2 were in agreement. At a 0.50 score difference in
158
M. P. BALLEJOS, R. L. RHYNE, J. PARKES
agreement, the interrater reliability rates were as follows: Interest and Suitability Difference D 92.0, Problem-Solving Difference D 88.2, Background/Diversity Difference D 93.3, Letters of Recommendation Difference D 95.0. Based on the subcategory scores the interviewers then assigned a summary noncognitive score for each applicant, which had to be within the range of the four subcategory scores. Noncognitive summary scores of the two interviewers agreed within a 0.50 range with an interrater reliability of more than 0.9. A composite average score was then calculated by each of the two interviewers assuming 50/50, or equal weighting of the cognitive criteria and noncognitive criteria. The entire admissions committee then reviewed all materials from each applicant, including the written summaries and composite average scores given by the interviewers. Each committee member assigned a noncognitive review score based on all the information available using the two interviewer composite average scores as a guide. All individual committee member review scores were averaged for each applicant and then converted to a percentile final Rank Score, which ultimately determined who was accepted or rejected. Applicants with the top 75 Rank Scores were offered admission to medical school. Because not all admitted students chose to matriculate to our medical school, alternates were identified by going down the list to subsequent Rank Scores. Modeling Study For this study we varied the relative weighting scheme for the noncognitive scores compared to the cognitive scores for the applicant cohort (2007–2009). This allowed us to determine how different weightings would change the main outcome variable, the composition of each class with respect to the proportion of URMs admitted to medical school. The percentile Rank Score of the last applicant admitted to an actual
matriculating class determined the admission cutoff percentile for the simulated study class, or the Admission Index Score (AIS). The AIS for each cohort class was 85.0%, 88.0%, and 85.3%, respectively. Admission decisions for each of the simulation cohort classes were determined using the AIS for the actual class that was admitted, which established the minimum admission standard for the simulated admission cut scores. The relative weights of the cognitive versus the noncognitive scores from the actual admissions process were varied with each simulation. To assure that cognitive standards were not lowered to conduct this study, we retained our school’s minimum thresholds for MCAT/UGPA (MCAT score of 22, UGPA of 3.0) for determining the thresholds of cognitive scores to assure success in medical school and maintain standards for admission. The weights for the noncognitive scores were increased by 5% increments, whereas the weights of the cognitive scores were decreased by 5% for each simulation run, starting with the actual admission standard that was used (50/50). We calculated the proportion of URMs accepted with each weighting scheme until the difference in URM proportion admitted became statistically significantly different from the original admission decisions, using the chi-square test of independence. RESULTS The applicant cohort consisted of 480 medical school applicants; each class had 30% URMs (Hispanic, Native American, or African American) and at least 50% women (Table 2). As expected, each class was approximately one third of the total cohort. The final score, final noncognitive, and final cognitive scores all were statistically significantly different between URMs and non-URMs in the applicant cohort. Specifically, URMs had higher final noncognitive mean scores (M D 3.64, SD D .23) than non-URMs (M D 3.44, SD D .27; p < .05), whereas non-URMs had higher final cognitive mean scores
TABLE 2 Demographics of medical school applicant cohort: 2007–2009 Applicant Cohort by Admission Year Characteristics Gender Female Male Total Race/Ethnicity Hispanic Native American African American White Asian Total
2007 N (%)
2008 N (%)
2009 N (%)
Total N (%)
82 (54) 69 (46) 151 (31)
84 (50) 85 (50) 169 (35)
87 (54) 73 (46) 160 (33)
253 (53) 227 (47) 480 (100)
39 (26) 3 (2) 3 (2) 87 (58) 19 (13) 151
36 (21) 11 (7) 4 (2) 103 (61) 15 (9) 169
37 (23) 9 (6) 2 (1) 98 (61) 14 (9) 160
112 (23) 23 (5) 9 (2) 288 (60) 48 (10) 480
159
MEDICAL SCHOOL ADMISSION CRITERIA FOR MINORITIES
TABLE 3 Medical school admission results by weighting scheme for URMs and non-URMs Admissions Decision Weighting Scheme (% / %) 50 cognitive/50 noncognitive Non-URM URM Total 45 cognitive/55 noncognitive Non-URM URM Total 40 cognitive/60 noncognitive Non-URM URM Total 35 cognitive/65 noncognitive Non-URM URM Total
No N (%)
Yes N (%)
Total N
Chi-Squarea df D 1
201 (66%) 102 (34%) 303 (100)
135 (76%) 42 (24%) 177 (100)
336 144 480
Reference
207 (68%) 96 (32%) 303 (100)
129 (73%) 48 (27%) 177 (100)
336 144 480
1.70, p D .20
201 (69%) 92 (31%) 293 (100)
135 (72%) 52 (28%) 187 (100)
336 144 480
3.40, p D .10
200 (70%) 87 (30%) 287 (100)
136 (70%) 57 (30%) 193 (100)
336 144 480
7.58, p < .001y
a Results from the chi-square test of independence used to test the proportion of admitted students who were underrepresented minorities (URMs; N D 144) in each weighting scheme against expected values from the 50 cognitive/50 noncognitive weighting scheme. y Statistically significant at Bonferroni corrected a D .01667.
(M D 3.26, SD D .47) than URMs (M D 2.93, SD D .52; p < .05), which has been substantiated in the literature.23 After assembling the admission cohort we analyzed the resultant cohort classes for differences in the demographic composition between classes. There were no statistically significant differences in the demographic variables between the classes (Table 2); therefore we pooled the data from all classes for the remainder of the modeling analyses. Table 3 shows the total number of admissions for the three simulated classes by weighting scheme for URMs and non-
URMs. Using the 50/50 weighing scheme as a reference, the next two levels of Cognitive %/Noncognitive % criteria (45/55 and 40/60) were not statistically significantly different in the proportion of URM students accepted using the original AIS as the standard for admission. The 35/65 level did show a statistically significant difference (p < .001) compared to the 50/50 level. The admission rates for URMs increased slightly with each 5% increase in noncognitive weighting and became statistically significantly different at the 35/65 level (Table 3). The proportion of URMs increased from 24% to 30%, at the 35/65
TABLE 4 Demographics of 2007–2009 medical school admission cohorts by weighting scheme
% Cognitive/% Noncognitive Admission Criteria Total Admitted of 480 Applicants Race/Ethnicity Hispanic Native American White Asian Gender Male Female
N (%) 50/50
N (%) 45/55
N (%) 40/60
N (%) 35/65
177 (36.9)
177 (36.9)
187 (39.0)
193 (40.2)
36 (20.3) 6 (3.4) 118 (66.7) 17 (9.6)
42 (23.7) 6 (3.4) 112 (63.3) 17 (9.6)
45 (24.1) 7 (3.7) 117 (62.6) 18 (9.6)
49 (25.4) 8 (4.1) 120 (62.2) 16 (8.3)
87 (49.2) 90 (50.8)
89 (50.3) 88 (49.7)
92 (49.2) 95 (50.8)
96 (49.7) 97 (49.7)
160
M. P. BALLEJOS, R. L. RHYNE, J. PARKES
TABLE 5 Medical school admission cohorts by cognitive and noncognitive admissions criteria
MCAT Score Lowest Score GPA Lowest Score Final Cognitive Final Noncognitive Final Average Score
M (SD) 50/50
M (SD) 45/55
M (SD) 40/60
M (SD) 35/65
30.7 (3.3) 23 3.75 (0.2) 3.04 3.7 (0.2) 3.6 (0.2) 3.6 (0.1)
30.5 (3.4) 22 3.73 (.02) 3.04 3.6 (0.3) 3.6 (0.2) 3.6 (0.1)
30.0 (3.6) 22 3.73 (.02) 3.04 3.6 (0.3) 3.6 (0.2) 3.6 (.01)
29.8 (3.7) 22 3.71 (0.2) 3.03 3.5 (0.3) 3.7 (0.2) 3.6 (0.1)
level. The number of URMs admitted went from 42 for the 50/ 50 scheme to 57 for the 35/65 scheme—an increase of 15, or 6%. Table 4 describes the combined resultant admission cohort for the three simulated classes that would be admitted using each weighting scheme as the admission scoring criteria. The total numbers admitted for the three classes do not reach the class size of 75, because of the 98, or 17%, who were excluded from the analysis. The proportion of the applicant cohort (n D 480) admitted using the 40/60 and 35/65 schemes increased from 37% to 39% and 40%, which increased the class size by 10 and 16 students, respectively. Hispanic acceptance rates increased steadily to 5.1% over all weighting schemes, admitting 13 more students when the 35/65 weighting scheme was used. The Native American rates increased only by 0.7%, admitting only two more students; which may be explained by the small Native American applicant pool (Table 2). Conversely, rates for White students admitted decreased by 4.5%. The proportions of female students essentially remained the same. Although Table 2 indicates that there were nine African American applicants over the 2007–2009 cohorts, they are not represented in the race/ethnicity section of Table 4. They were excluded from this study due to one of the following exclusion factors: (a) missing noncognitive subcategory score, (b) had fewer or greater than two interviews, or (c) failed to meet the minimum MCAT and/or GPA thresholds. Overall the mean MCAT scores, GPAs, final cognitive scores, and final noncognitive scores did not change as the cognitive weighting decreased (Table 5). The AAMC considers an MCAT total score of plus or minus 2 points as falling within the 68% confidence interval and may not be meaningfully different from each other.4 The differences observed in Table 5 for the minimum MCAT and GPA scores were not statistically different and the minimum admissions criteria maintained for each class admitted.
DISCUSSION Although Hispanics, Native Americans, and African Americans comprised more than 25% of the nation’s population in
2007, these minorities were less than 9% of the physician workforce.9 Research has shown that healthcare providers with the same racial and ethnic background as their patients have a positive impact on underserved, multiethnic patients.5–9 Educating and graduating physicians from URMs will likely increase the physician workforce serving those populations. Medical schools can contribute to building the minority workforce by admitting more URM students,3 but minimum scholarly standards must be maintained in the admissions process. This study tests an alternative method of weighting cognitive versus noncognitive criteria in the medical student admissions process in an attempt to diversify incoming medical student classes and eventually, the U.S. healthcare workforce. URMs are not admitted to medical schools in sufficient numbers to address health disparities in minority populations, in part because they are underrepresented in the applicant pool and in part because, on average, academic qualifications of URM applicants lag behind those of their majority peers. However, it is not simply a lack of the number of minority applicants but also a problem of academic quality of URMs. In addition, the current admission process arguably focuses disproportionately on cognitive scores and academic achievement versus noncognitive criteria.24 Our study shows that by changing the relative weighting criteria more toward noncognitive versus cognitive criteria, medical schools can modestly increase the proportion of URMs without lowering admission standards. Changing admissions processes toward admitting more well-rounded, diverse students will require schools to examine their admissions practices and processes and challenge the status quo.25 More medical schools are now seeking applicants who not only have strong academic backgrounds but also have a solid foundation in the social determinants of health and are able to relate to a variety of different patients from diverse backgrounds. There have been recent calls for balancing the medical school admission criteria toward a holistic admissions process using “other academic, personal and experiential credentials.”12 In fact, the newly redesigned MCAT will now include psychological and social foundations of behavior as well as the critical analysis and reasoning skills that support other criteria that are important in the holistic review process.26 Using three years of applicant and medical school admissions data, our simulation study found that placing more emphasis on noncognitive criteria increased the URM acceptance rates. All students met the minimum criteria for admission (Table 5). When the relative weight of noncognitive criteria was increased from 50% to 65%, URM admission rates increased from 24% to 30% (Table 3). Although an increase of 6% in URM admissions may seem small, it would have resulted in an increase of 15 more URM students being admitted over the 3-year admissions period. The impact of having an average of five more URM students per class could potentially increase the minority physician workforce practicing in medically underserved, rural areas such as New Mexico.
MEDICAL SCHOOL ADMISSION CRITERIA FOR MINORITIES
The challenge in selecting a more diverse class of students is to ensure that all students have a strong academic foundation and a demonstrated set of noncognitive skills that would lead to successful completion of the medical school curriculum. Our modeling study retained the minimum criteria for admissions that our admissions committee used to admit students over the three years of the study, an MCAT score of 22 or higher and a UGPA of 3.0 or higher (Table 5). This allowed us to vary the relative balance between cognitive and noncognitive criteria without sacrificing our school’s standards for admission. Traditionally, non-URMs have performed higher in the cognitive criteria (MCATs and UGPAs) due to environmental, socioeconomic, and educational factors that relate to academic achievement.27 URMs tend to have a greater likelihood of coming from a rural or underserved area, being a firstgeneration college student, coming from a single-parent household, having English as a second language, and experiencing educational and financial disadvantages.27 Indeed, in our study URMs had higher overall noncognitive scores (M D 3.6 for URMs vs. 3.44 for non-URMs; p < .05). Admission committee members knew the URM status when they scored the noncognitive criteria, which may have contributed to the higher scores for URMs (Table 1), but URM status was not a specifically scored category. The Hispanic population, which is the fastest growing and among the most severely underrepresented in medicine, was most impacted by the changed admissions criteria. The proportion of Hispanic students admitted to medical school increased from 20.3% to 25.4% as the relative weight of noncognitive criteria was increased from 50% to 65%. An important limitation of these data is that the African American population is not represented because of their low prevalence in New Mexico. The literature suggests that caution is needed in making complex decisions about how to appropriately weight composite scores, such as MCAT/GPA with noncognitive criteria on group differences.28 In addition, some have questioned the validity and reliability of composite scores representing a holistic review process.29 The data from this medical school may not be generalizable to other medical schools, although these results may be most applicable to the other majority– minority states. The context, mission, and goals of each medical school need to be considered. In addition, this and previous studies have used the categories of cognitive and noncognitive criteria in the admissions process, but the new holistic review process supports the use of “experiences, attributes and academic metrics” as the recommended categories for medical school admission processes.12 The authors recognize that there are other limitations to this study. The raters’ knowledge of URM status when assessing noncognitive variables could possibly confound the study. However, knowledge of URM status is part of the noncognitive holistic review, which the AAMC considers to be a plus factor for selection if it supports the medical school mission. Our ranking
161
system assigned a maximum score of 4.0 for all MCAT scores of 30 and above (Table 1) and a minimum score based on a preset minimum standard for MCAT scores. The predominance of 4.0 assigned MCAT scores in the applicant pool (due to the high proportion of 30-plus MCAT scores) constrained the average MCAT score so that it was unlikely to change with the addition of a small number of lower MCAT scores as the weighting criteria changed (Table 5). This study does not include outcomes of the URM students, because this was a modeling study of the admissions process. Future studies should include URM graduation rates, performance in medical school, and the practice location of graduates. It is possible that the study results could have been biased if the students who were excluded because they didn’t respond to the race/ethnicity question were systematically different from those who were included in the study. We did not study the excluded students because their race/ethnicity was unknown; therefore, the results assume that those excluded had the same distribution as those included. As the trend toward diversifying the physician workforce continues to grow, all U.S. medical schools are now mandated to consider noncognitive characteristics beyond academic or cognitive ability in selecting applicants for medical school.30 The AAMC has charged medical schools to think differently about admissions and to incorporate an applicant’s “experiences, attributes and metrics” into the screening, application and selection processes.12 These admissions issues have become paramount for most medical schools that are struggling to increase diversity related goals in order to address racial and ethnic health disparities. Questions about filtering out applicants based solely on their academic performance have recently been posed, because this practice may have future ramifications on the healthcare workforce.31 As the AAMC suggests, perhaps we should begin to screen, interview, and select applicants based on the mission of the individual school and the social obligation to serve the surrounding communities and populations and not just cognitive criteria.2,32 This study was conducted at a state-funded medical school in a majority–minority state, which has a particular responsibility to the many diverse communities across the state. Finding balance between diversity and objectivity is a key component of the complexities of reweighting the admissions formula for this and other medical schools as demographic changes in the U.S. magnify the importance of caring for rapidly growing minority populations. Recent research confirms that minority status remains a strong predictor for practicing in underserved areas.33 The present study shows that diversityrelated initiatives could work and supports the notion that qualified URM applicants can be admitted to medical school in larger numbers.
SUPPLEMENTAL DATA Supplemental data for this article can be accessed on the publisher’s website at http://dx.doi.org/10.1080/ 10401334.2015.1011649.
162
M. P. BALLEJOS, R. L. RHYNE, J. PARKES
REFERENCES 1. 2011 National healthcare quality and disparities reports. Rockville, MD: Agency for Healthcare Research and Quality, 2012. Available at: http:// www.ahrq.gov/research/findings/nhqrdr/nhqrdr11/qrdr11.html. Accessed June 26, 2013. 2. Mullan F, Chen C, Petterson S, Kolsky G, Spagnola M. The social mission of medical education: Ranking the schools. Annals of Internal Medicine 2010;152:804–11. 3. America needs a more diverse physician workforce. Washington, DC: Association of American Medical Colleges, 2006. Available at: https:// www.aamc.org/download/87306/data/physiciandiversityfacts.pdf. Accessed January 28, 2010. 4. Using MCAT data in medical student selection. Washington, DC: Association of American Medical Colleges, 2012. Available at: https://www. aamc.org/students/download/267622/data/mcatstudentselectionguide.pdf. Accessed June 21, 2012. 5. Saha S, Shipman S. The rationale for diversity in the health professions: A review of the evidence. Washington, DC: U.S. Department of Health and Human Services, Health Resources Administration, Bureau of Health Professions, 2006. 6. Saha S, Shipman S. Race-neutral versus race-conscious workforce policy to improve access to care. Health Affairs 2008;27:234–45. 7. Smedley BD, Stith AY, Nelson AR (Eds.). Unequal treatment: Confronting racial and ethnic disparities in healthcare. Washington, DC: Institute of Medicine of the National Academies Press, 2003. 8. Coleman AL, Palmer SR, Winnick SY. Roadmap to diversity: Key legal and educational policy foundations for medical schools. Washington, DC: Association of American Medical Colleges, 2008. 9. American College of Physicians. Racial and ethnic disparities in health care. Philadelphia, PA: American College of Physicians, 2010. Available at: http://www.acponline.org/advocacy/current_policy_papers/ assets/racial_disparities.pdf. Accessed June 17, 2013. 10. Yen, H. Census shows whites lose US Majority among babies.Associated Press, 2011. Available at: http://news.yahoo.com/census-shows-whites-loseus-majority-among-babies-193257646.html. Accessed January 28, 2011. 11. Grbic D, Slapar F. Changes in medical students’ intentions to serve the underserved: Matriculation to graduation. Analysis in Brief. Washington, DC: Association of American Medical Colleges, 2010. 12. White CB, Day EL, Fantone JC. Analysis of factors that predict clinical performance in medical school. Advances in Health Sciences Education 2009;14:455–64. 13. Addams AN, Bletzinger RB, Sondheimer HM, White SE, Johnson LM. Roadmap to diversity: Integrating holistic review practices into medical school admission processes. Washington, DC: Association of American Medical Colleges, 2010. 14. Kulatunga-Moruzi C, Norman GR. Validity of admissions measures in predicting performance outcomes: The contribution of cognitive and noncognitive dimensions. Teaching and Learning in Medicine 2002;14:34– 42. 15. Kreiter CD, Yin P, Solow C, Brennan RL. Investigating the reliability of the medical school admissions interview. Advances in Health Sciences Education 2004;9:147–59.
16. Albanese MA, Snow MH, Skochelak SE, Huggett KN, Farrell PN. Assessing personal qualities in medical school admissions. Academic Medicine 2003;78:313–21. 17. Kreiter C. A proposal for evaluating the validity of holistic-based admission processes. Teaching and Learning in Medicine 2013;25:103–7. 18. Humprhrey-Murto S. Does emotional intelligence at medical school admission predict future academic performance? Academic Medicine 2014;89:1–6. 19. Hanson MD, Kulasegaram KM, Coombs DL , Herold J. Admissions file review: Applying the multiple independent sampling (MIS) methodology. Academic Medicine 2012;87:1335–40. 20. Edwards JC, Maldonado FG, Calvin JA. The effects of differently weighting interview scores on the admission of underrepresented minority medical students. Academic Medicine 1999;74:59–61. 21. Steinecke A, Beaudreau J, Bletzinger RB, Terrell C. Race-neutral admission approaches: Challenges and opportunities for medical schools. Academic Medicine 2007;82:117–26. 22. Fisher v. University of Texas at Austin. 132 US 1536(2012). 570 U.S. 23. Cohen GL, Garcia J, Apfel N, Master A. Reducing the racial achievement gap: A social-psychological intervention. Science 2006;313:1307–10. 24. Dunleavy D, Sondheimer H, Castillo-Page L, Bletzinger RB. Medical school admissions: More than grades and test scores. Analysis in Brief. Washington, DC: Association of American Medical Colleges, 2011. 25. Geller G. Tolerance for ambiguity: An ethics-based criterion for medical school selection. Academic Medicine 2013;88:581–4. 26. Schwartzstein RM, Rosenfeld GC, Hilborn R, Herndon Oyewhole S, Mitchell, K. Redesigning the MCAT Exam: Balancing multiple perspectives. Academic Medicine 2013;88:560–7. 27. Davis D, Dorsey K, Franks RD, Sackett PR, Searcy CA, Zhao X. Do racial and ethnic group differences in performance on the MCAT exam reflect test bias? Academic Medicine 2013;88:593–602. 28. Sackett PR, Ellingson JE. The effects of forming multi-predictor composites on group differences and adverse impact. Personnel Psychology 1997;50:707–22. 29. Kreiter CD. A measurement perspective on affirmative action in U.S. medical education, 3rd ed. Med Educ Online: April 2013. Available at: http:// www.ncbi.nlm.nih.gov/pmc/articles/PMC3623946/. Accessed December 3, 2013. 30. Liaison Committee on Medical Education. Accreditation standards. Washington, DC: Association of American Medical Colleges and the American Medical Association. Available at: http://www.lcme.org/stan dard.htm. 31. Association of American Medical Colleges. Recent studies and reports on physician shortages in the US. Washington, DC: Association of American Medical Colleges. Available at: https://www.aamc.org/download/100598/ data/. Accessed October 25, 2012. 32. Kirch DG, Prescott JE. From rankings to mission. Academic Medicine 2013;88:1064–1066. 33. Wayne SJ, Kalishman S, Jerabek RN, Timm C, Cosgrove E. Early predictors of physicians’ practice in medically underserved communities: A 12year follow-up study of University of New Mexico School of medicine graduates. Academic Medicine 2010;85:513–6.
Copyright of Teaching & Learning in Medicine is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
