British Journal of Medico1 Educorion, 1975, 9, 86-90
Longitudinal study of medical students selected for admission to medical school by actuarial and Committee methods’ WILLIAM SCHOFIELD2 and JUDITH GARRARD University of Minnesota Medical School, Minneapolis, Minnesota, U.S.A. Key words *COLLEGE ADMISSION T€Sf *SCHOOLS, MEDICAL APTITUDE TESTS REGRESSION ANALYSIS PERSONALITY ATI-ITWDE OF HEALTH PERSONNEL FOLLOW-u~ STUDIES INTERNSHIP AND RESIDENCY MINNESOTA
In recent years many state-supported medical schools in the United States have experienced public pressure to produce more primary-care physicians and, hopefully, graduates who will practise in the state which provided their training. Members of medical school admissions committees, other considerations equal, have been persuaded in their case-by-case decisions by ‘clues’ to the probable medical career of the applicant, and seek to balance the ‘hard’ evidence of academic aptitude against ‘softer’ indications of interests, values, and personality. Individual members vary in their confidence as to how well they can read such signs from the applicant dossier or from the interview. In the process of committee review these individualized appraisals are shared, compared, debated, and defended. The h a 1 judgement, a committee vote, hopefully represents some distillate of wisdom which is more valid and more reliable than could be otherwise accomplished. With very large and growing numbers of applicants, the time-consuming and very expensive clinical process of committee judgements can be justified only if it can demonstrate a significant gain over what can be achieved by a more objective and efficient process.
In a sizeable accumulation of selectionprediction studies, an actuarial approach has been used in which selection is based on quantitative summary of objective indices. The overwhelming evidence from these studies is that the actuarial approach equals when it does not exceed the validity of the traditional or clinical, judgemental approach, and always has the advantages of efficiency, economy, and reliability (Sawyer, 1966).
In a previous paper by the senior author (Schofield, 1970), the short-term outcomes of selection of medical students at the University of Minnesota by an actuarial approach as compared with the customary methods of committee selection were reported. One-third (N=49) of the entering 1966 class at the University of Minnesota (Pool C ) was selected solely by rank-order on a previously developed predictor index, PI (Schofield and Merwin, 1966). The index was developed in research in which students’ grade point averages (GPA) based on college credits earned before entering medical school, and the total scores on the Medical College Aptitude Test (MCAT) were used in a multiple linear regression analysis to establish their relation with grades in the first year of medical school. (The resulting weights were : PI = (GPA x 1.76) (MCAT x 1.56) 4070.) The index, therefore, is used to predict ‘The data analysis for the study reported here were supported by a National Institutes of Health Giant. applicants’ academic success during the first year of medical school.
+
3Requests for reprints to Professor William Schofield, Box 393 Mayo Hospital, University of Minnesota Medical School, Minneapolis, Minnesota, 55455, U.S.A. 86
+
The remaining two-thirds of the 1966 class were selected by the usual process of committee
rl ctiruriol and committee methods of admission
review: one-third (Fool A; N=52) without committee knowledge of the PIS, and the other third (Pool B; N=49) with PIS reported to the committee. Departure from strictly actuarial selection of the Pool C subjects was made only to assure a proportion of female and nonresident students matching that of the remainder of the class. The committee review and selection was based on the total information provided in each applicant’s file. In addition to the usual demographic data, academic transcript, and MCATs, the files included information on extracurricular activities, avocational interests, work experience, Minnesota Multiphasic Personality Inventory (MMPI) scores, Strong Vocational Interest Blank (SVIB) profiles, and notes of any interviews which were conducted.
As previously reported, the three pools did not differ with respect to the number of selected students who matriculated nor in the number whose medical education was discontinued within the first two years. Furthermore, despite the possibility for extra-academic variables to influence committee selection, it was found that the committee selections were very similar to the experimental pool in terms of average MCAT scores and premedical GPAs. The three student pools were compared on the basis of class rank based on grades in basic science courses for the freshman year, for the sophomore year, and for the first two years combined. None of the differences among the mean achievement scores or among the score variances of the three samples proved statistically reliable. Inasmuch as committee selection appeared to
be undifferentiated from actuarial selection with respect to the apparent emphasis on academic achievement variables, it might be expected that the three pools would be undifferentiated with respect to short-term basic science achievement. I t could be argued, however, that committee selectivity with regard to indicators of motivation, interest, and personality would begin to become manifest as the students moved into their clinical experiences during the third year of medical school, as the pressures of medical school became more varied, and as critical choices leading to subsequent careers were
a7
made. The purpose of this paper is to describe similarities and differences between and among the three groups of students as they moved beyond their second year of medical school.
Method Various kinds of data were available at different points in the students’ subsequent academic and professional careers. These included the junior year class rank, results from the part I1 examination by the National Board of Medical Examiners, and follow-up information about the type of graduate training and professional plans beyond the first year of internship. The junior year class rank was computed by the medical school half-way through the fourth year and was the official cumulative score used by students in applying for internships. For purposes o€ data analysis, the Kruskal-Wallis analysis of variance by ranks procedure was used to examine differences among the three student groups (Hays, 1966). The National Board examination was administered in April of the senior year just before graduation in June. The examination consists of six subtests : medicine, obstetrics/gynaecology, preventive rnedicine/public health, paediatrics, surgery, and psychiatry. Seven scores are reported, one for each subtest and a total score; the criterion €or passing the part I1 examination is based on the total score only. Students were required to pass part I1 in order to graduate from medical school. These data were analysed by means of a one-way analysis of variance and the Scheffe test was used to examine pairwise differences between means. The data gathered following graduation from medical school were obtained from the Center for Health Services Research and Development of the American Medical Association (AMA) in June 1971? This information is routinely gathered by the AMA from all physicians in the United States. The non-parametric 2 test was used €or purposes of data analyses. Results Table 1 reports data for class rank at the end of three and one-half years post-matriculation. To whatever extent this variable reflects appraisal of clinical skills and personality attitudes observable during the clinical clerkships of the
‘More detailed tabular breakdowns and statistical analyses are available upon request to the authors.
88
William Schofield and Judith Garrard
junior year, it provides for possible manifestations of non-cognitive potentials which might have been appraised by the admissions committee and be reflected in their selections, in contrast to the PI which reflects solely academic aptitude. However, as the data show, the three pools are not reliably different in the distributions of their class ranks. Table 1. Class rank of students at midpoint of senior year of medical school Pool
Mean
Standnrd deviation
A (N=52) B (N=50) C (N=49)
76.04 75.65 77.79
45.84 43.18 42.65
Kruskal-Wallis x2 = 5.80; degrees of freedom = 150, P>505
One could reason that if committee review and selection in fact was responsive to information not contained in the PI, one might anticipate that variations in interest patterns could be reflected in differential performance on the part I1 specialty exams by the National Board. This was not the case, however. The comparative data for the seven scores yielded by the part I1 examination showed that the three
groups were not differentiated by any of the subtest scores or by the total score. The probability levels based on the overall F test from the analysis of variance are as follows: medicine (p =0.32); surgery (p= 056); obstetrics/ gynaecology (p =0.1 6); preventive medicine/ public health (p= 0.08); paediatrics (p=0.27); psychiatry (p=0-53); and total (p=O.15). The data in Tables 2-5 are descriptive of the three pools of students with regard to variables characterizing the early stages of their medical careers, i.e. the nature of their training and/or experiences during the first year after graduation from medical school. In summary, it may be stated that the actuarially selected students cannot be statistically discriminated from committee selected students with respect to type or locus of internship, with respect to practice or training status one year after graduation, or with respect to plans for specialization. Table 3. Locations of students’ internships
2 =
Pool
Minnesota
A (N=48) B (N=49) C (N=471
17 20 25
Other states 31 29 22
3.21; d.f. = 2; P = 0.20
Table 2. N u m b e r of students taking dioercnt types of internships Rotating
Pool
or mixed
Medicine
Paediatrics
Surgery
Pathology
A (N=48) B (N=49) C (N=47)
33 31 31
7 11
6 6 3
2 1
0 0
4
1
x2 = 6.28; d.f. = 8; P
8
= 0.62
Table 4. Training or practice status as of June 1971, of three groups of students Pool
Internship
Residency
Other training
Patient care
Unclassified
A (N=51)
12 17 17
25 20 15
1 0 0
5 5 5
8 5 10
B (N=47) C (N=47) ,y2
= 15.71; d.f. = 14; P = 0.35
89
Actirarial and committee merhods of admission
Table 5. Speciality status (active or planned) as of June 1971 of three groups of students Pool
Internal medicine
Family practice
A (N=49) B (N=47) C (N=46)
8 9 11
1 2 2
General practice 8 3 5
Paediatrics 5
5 5
General srtrgery 4 1 5
Psychiatry 2 3 1
Other‘ 21 24 17
X? = 21.71; d.f. = 32; P = 0.91
‘Included eleven other specialities each with total (N= 142) selection by less than 57!, of each pool. These include: Anaesthesiology, Dermatology, Neurology, Neurosurgery, Obstetrics/Gynaecology, Orthopaedic Surgery, Otorhinolaryngology, Pathology, Radiology, and Underwater Medicine.
Discussion The previous study (Schofield, 1970) concluded that with respect to the short-range criterion of academic performance in the first two years of medical school, the customary procedures of committee review of complete applicant files, including interview impressions, do not achieve a significant gain over what can be effected by the actuarial application of an appropriately weighted predictor index. The findings of the present study support and extend that conclusion to cover the characteristics of medical school graduates in regard to their performance on the part I1 examination of the National Board of Medical Examiners, and global aspects of their post-graduation choices of training and 1or practice.
The present findings offer further support for the conclusion of the previous study: that the expensive and valuable time of admissions committees should be reserved for the evaluation of borderline or otherwise special cases which are not differentiated by an actuarial index. Furthermore, committees can better devote their energies to tough issues of po!icy and philosophy. The present findings are fully consonant with the ‘successive hurdles’ approach suggested by Rosenberg (1973). The criterion or ‘outcome’ measures reported in this study are by no means the ultimate or complete results of the selection process. It might be argued that the eventual ‘quality of practice’ as reflected in the personality of the fully trained physician and in his relations with his patients will be expressive of variables perceived by the admissions committee and highly determinative of their choices, so that eventually their selections could be dernonstrated to be systematically superior to the
actuarially chosen subjects in terms of empathy, clinical judgement, sensitivity to patients’ extramedical needs, etc. When ‘quality of practice’ has been successfully translated into reliable measures, it is possib!e that these measures may reflect attributes of the physician’s personality present at the time of application to medical school. If SO. these too will yield better predictions if evaluated by objective measures in an acturial context (Dawes and Corrigan, 1974). Summary This study evaluates a ‘mid-period’ follow-up evaluation of the outcomes of selection of medical students by customary committee review procedures versus actuarial selection. One-third of a freshman class was selected solely on the basis of a predictor index which was a previously validated, optimally weighted combination of scores on the Medical College Aptitude Test and the premedical grade-point average. The remaining two-thirds were selected by committee decision based on review of the total application file which, in addition to the aptitude test scores and academic record, included basic demographic data, information on extracurricular activities, avocational interests, work experience, letters of recommendation, personality test profiles, and interview impressions. In a previous study, it was reported that the two groups of students were undifferentiated with respect to their academic standing at the close of their sophomore year. In the present study, the actuarially selected and committee selected students were compared on class rank at the end of the junior year, total and subtest scores on part I1 of the National Board Examinations administered toward the close of their senior year, and type and location of internship, and practice or training status one
William Schofield and Judith Gnrrard
90
year after graduation. The two groups were not reliably differentiated on any of these variables. Implications of the findings are discussed with respect to reliability, efficiency, and economy in the selection process and the function of the admissions committees with respect to ‘borderline’ cases and issues of school poIicy and philosophy. References Dawes, R. M., and Corrigan, B. (1974). Linear models in decision making. Psychological Bulletin, 81, 95-106.
Hays, Wm. L. (1966). Sfarisrics f o r Psychologists. Holt, Rinehart, and Winston: New York. Rosenberg, M. L. (1973). Increasing the efficiency of medical school admissions. lorrrnal of Medical Education, 48, 707-717. Sawyer, J. (1966). Measurement and predicticn, clinical and statistical. Psychological Bulletin, 66, 178-200. Schofield, W. (1970). A modified actuarial method in the selection of medical students. Jortrnal of Medical Education, 45, 740-744. Schofield, W., and Menvin, J. C. (1966). The use of scholastic aptitude, personality and interest test data in the selection of medical students. Jorirnal of Medical Education, 41, 502-509.
Longitudinal study of medical students selected for admission to medical school by actuarial and Committee methods’ WILLIAM SCHOFIELD2 and JUDITH GARRARD University of Minnesota Medical School, Minneapolis, Minnesota, U.S.A. Key words *COLLEGE ADMISSION T€Sf *SCHOOLS, MEDICAL APTITUDE TESTS REGRESSION ANALYSIS PERSONALITY ATI-ITWDE OF HEALTH PERSONNEL FOLLOW-u~ STUDIES INTERNSHIP AND RESIDENCY MINNESOTA
In recent years many state-supported medical schools in the United States have experienced public pressure to produce more primary-care physicians and, hopefully, graduates who will practise in the state which provided their training. Members of medical school admissions committees, other considerations equal, have been persuaded in their case-by-case decisions by ‘clues’ to the probable medical career of the applicant, and seek to balance the ‘hard’ evidence of academic aptitude against ‘softer’ indications of interests, values, and personality. Individual members vary in their confidence as to how well they can read such signs from the applicant dossier or from the interview. In the process of committee review these individualized appraisals are shared, compared, debated, and defended. The h a 1 judgement, a committee vote, hopefully represents some distillate of wisdom which is more valid and more reliable than could be otherwise accomplished. With very large and growing numbers of applicants, the time-consuming and very expensive clinical process of committee judgements can be justified only if it can demonstrate a significant gain over what can be achieved by a more objective and efficient process.
In a sizeable accumulation of selectionprediction studies, an actuarial approach has been used in which selection is based on quantitative summary of objective indices. The overwhelming evidence from these studies is that the actuarial approach equals when it does not exceed the validity of the traditional or clinical, judgemental approach, and always has the advantages of efficiency, economy, and reliability (Sawyer, 1966).
In a previous paper by the senior author (Schofield, 1970), the short-term outcomes of selection of medical students at the University of Minnesota by an actuarial approach as compared with the customary methods of committee selection were reported. One-third (N=49) of the entering 1966 class at the University of Minnesota (Pool C ) was selected solely by rank-order on a previously developed predictor index, PI (Schofield and Merwin, 1966). The index was developed in research in which students’ grade point averages (GPA) based on college credits earned before entering medical school, and the total scores on the Medical College Aptitude Test (MCAT) were used in a multiple linear regression analysis to establish their relation with grades in the first year of medical school. (The resulting weights were : PI = (GPA x 1.76) (MCAT x 1.56) 4070.) The index, therefore, is used to predict ‘The data analysis for the study reported here were supported by a National Institutes of Health Giant. applicants’ academic success during the first year of medical school.
+
3Requests for reprints to Professor William Schofield, Box 393 Mayo Hospital, University of Minnesota Medical School, Minneapolis, Minnesota, 55455, U.S.A. 86
+
The remaining two-thirds of the 1966 class were selected by the usual process of committee
rl ctiruriol and committee methods of admission
review: one-third (Fool A; N=52) without committee knowledge of the PIS, and the other third (Pool B; N=49) with PIS reported to the committee. Departure from strictly actuarial selection of the Pool C subjects was made only to assure a proportion of female and nonresident students matching that of the remainder of the class. The committee review and selection was based on the total information provided in each applicant’s file. In addition to the usual demographic data, academic transcript, and MCATs, the files included information on extracurricular activities, avocational interests, work experience, Minnesota Multiphasic Personality Inventory (MMPI) scores, Strong Vocational Interest Blank (SVIB) profiles, and notes of any interviews which were conducted.
As previously reported, the three pools did not differ with respect to the number of selected students who matriculated nor in the number whose medical education was discontinued within the first two years. Furthermore, despite the possibility for extra-academic variables to influence committee selection, it was found that the committee selections were very similar to the experimental pool in terms of average MCAT scores and premedical GPAs. The three student pools were compared on the basis of class rank based on grades in basic science courses for the freshman year, for the sophomore year, and for the first two years combined. None of the differences among the mean achievement scores or among the score variances of the three samples proved statistically reliable. Inasmuch as committee selection appeared to
be undifferentiated from actuarial selection with respect to the apparent emphasis on academic achievement variables, it might be expected that the three pools would be undifferentiated with respect to short-term basic science achievement. I t could be argued, however, that committee selectivity with regard to indicators of motivation, interest, and personality would begin to become manifest as the students moved into their clinical experiences during the third year of medical school, as the pressures of medical school became more varied, and as critical choices leading to subsequent careers were
a7
made. The purpose of this paper is to describe similarities and differences between and among the three groups of students as they moved beyond their second year of medical school.
Method Various kinds of data were available at different points in the students’ subsequent academic and professional careers. These included the junior year class rank, results from the part I1 examination by the National Board of Medical Examiners, and follow-up information about the type of graduate training and professional plans beyond the first year of internship. The junior year class rank was computed by the medical school half-way through the fourth year and was the official cumulative score used by students in applying for internships. For purposes o€ data analysis, the Kruskal-Wallis analysis of variance by ranks procedure was used to examine differences among the three student groups (Hays, 1966). The National Board examination was administered in April of the senior year just before graduation in June. The examination consists of six subtests : medicine, obstetrics/gynaecology, preventive rnedicine/public health, paediatrics, surgery, and psychiatry. Seven scores are reported, one for each subtest and a total score; the criterion €or passing the part I1 examination is based on the total score only. Students were required to pass part I1 in order to graduate from medical school. These data were analysed by means of a one-way analysis of variance and the Scheffe test was used to examine pairwise differences between means. The data gathered following graduation from medical school were obtained from the Center for Health Services Research and Development of the American Medical Association (AMA) in June 1971? This information is routinely gathered by the AMA from all physicians in the United States. The non-parametric 2 test was used €or purposes of data analyses. Results Table 1 reports data for class rank at the end of three and one-half years post-matriculation. To whatever extent this variable reflects appraisal of clinical skills and personality attitudes observable during the clinical clerkships of the
‘More detailed tabular breakdowns and statistical analyses are available upon request to the authors.
88
William Schofield and Judith Garrard
junior year, it provides for possible manifestations of non-cognitive potentials which might have been appraised by the admissions committee and be reflected in their selections, in contrast to the PI which reflects solely academic aptitude. However, as the data show, the three pools are not reliably different in the distributions of their class ranks. Table 1. Class rank of students at midpoint of senior year of medical school Pool
Mean
Standnrd deviation
A (N=52) B (N=50) C (N=49)
76.04 75.65 77.79
45.84 43.18 42.65
Kruskal-Wallis x2 = 5.80; degrees of freedom = 150, P>505
One could reason that if committee review and selection in fact was responsive to information not contained in the PI, one might anticipate that variations in interest patterns could be reflected in differential performance on the part I1 specialty exams by the National Board. This was not the case, however. The comparative data for the seven scores yielded by the part I1 examination showed that the three
groups were not differentiated by any of the subtest scores or by the total score. The probability levels based on the overall F test from the analysis of variance are as follows: medicine (p =0.32); surgery (p= 056); obstetrics/ gynaecology (p =0.1 6); preventive medicine/ public health (p= 0.08); paediatrics (p=0.27); psychiatry (p=0-53); and total (p=O.15). The data in Tables 2-5 are descriptive of the three pools of students with regard to variables characterizing the early stages of their medical careers, i.e. the nature of their training and/or experiences during the first year after graduation from medical school. In summary, it may be stated that the actuarially selected students cannot be statistically discriminated from committee selected students with respect to type or locus of internship, with respect to practice or training status one year after graduation, or with respect to plans for specialization. Table 3. Locations of students’ internships
2 =
Pool
Minnesota
A (N=48) B (N=49) C (N=471
17 20 25
Other states 31 29 22
3.21; d.f. = 2; P = 0.20
Table 2. N u m b e r of students taking dioercnt types of internships Rotating
Pool
or mixed
Medicine
Paediatrics
Surgery
Pathology
A (N=48) B (N=49) C (N=47)
33 31 31
7 11
6 6 3
2 1
0 0
4
1
x2 = 6.28; d.f. = 8; P
8
= 0.62
Table 4. Training or practice status as of June 1971, of three groups of students Pool
Internship
Residency
Other training
Patient care
Unclassified
A (N=51)
12 17 17
25 20 15
1 0 0
5 5 5
8 5 10
B (N=47) C (N=47) ,y2
= 15.71; d.f. = 14; P = 0.35
89
Actirarial and committee merhods of admission
Table 5. Speciality status (active or planned) as of June 1971 of three groups of students Pool
Internal medicine
Family practice
A (N=49) B (N=47) C (N=46)
8 9 11
1 2 2
General practice 8 3 5
Paediatrics 5
5 5
General srtrgery 4 1 5
Psychiatry 2 3 1
Other‘ 21 24 17
X? = 21.71; d.f. = 32; P = 0.91
‘Included eleven other specialities each with total (N= 142) selection by less than 57!, of each pool. These include: Anaesthesiology, Dermatology, Neurology, Neurosurgery, Obstetrics/Gynaecology, Orthopaedic Surgery, Otorhinolaryngology, Pathology, Radiology, and Underwater Medicine.
Discussion The previous study (Schofield, 1970) concluded that with respect to the short-range criterion of academic performance in the first two years of medical school, the customary procedures of committee review of complete applicant files, including interview impressions, do not achieve a significant gain over what can be effected by the actuarial application of an appropriately weighted predictor index. The findings of the present study support and extend that conclusion to cover the characteristics of medical school graduates in regard to their performance on the part I1 examination of the National Board of Medical Examiners, and global aspects of their post-graduation choices of training and 1or practice.
The present findings offer further support for the conclusion of the previous study: that the expensive and valuable time of admissions committees should be reserved for the evaluation of borderline or otherwise special cases which are not differentiated by an actuarial index. Furthermore, committees can better devote their energies to tough issues of po!icy and philosophy. The present findings are fully consonant with the ‘successive hurdles’ approach suggested by Rosenberg (1973). The criterion or ‘outcome’ measures reported in this study are by no means the ultimate or complete results of the selection process. It might be argued that the eventual ‘quality of practice’ as reflected in the personality of the fully trained physician and in his relations with his patients will be expressive of variables perceived by the admissions committee and highly determinative of their choices, so that eventually their selections could be dernonstrated to be systematically superior to the
actuarially chosen subjects in terms of empathy, clinical judgement, sensitivity to patients’ extramedical needs, etc. When ‘quality of practice’ has been successfully translated into reliable measures, it is possib!e that these measures may reflect attributes of the physician’s personality present at the time of application to medical school. If SO. these too will yield better predictions if evaluated by objective measures in an acturial context (Dawes and Corrigan, 1974). Summary This study evaluates a ‘mid-period’ follow-up evaluation of the outcomes of selection of medical students by customary committee review procedures versus actuarial selection. One-third of a freshman class was selected solely on the basis of a predictor index which was a previously validated, optimally weighted combination of scores on the Medical College Aptitude Test and the premedical grade-point average. The remaining two-thirds were selected by committee decision based on review of the total application file which, in addition to the aptitude test scores and academic record, included basic demographic data, information on extracurricular activities, avocational interests, work experience, letters of recommendation, personality test profiles, and interview impressions. In a previous study, it was reported that the two groups of students were undifferentiated with respect to their academic standing at the close of their sophomore year. In the present study, the actuarially selected and committee selected students were compared on class rank at the end of the junior year, total and subtest scores on part I1 of the National Board Examinations administered toward the close of their senior year, and type and location of internship, and practice or training status one
William Schofield and Judith Gnrrard
90
year after graduation. The two groups were not reliably differentiated on any of these variables. Implications of the findings are discussed with respect to reliability, efficiency, and economy in the selection process and the function of the admissions committees with respect to ‘borderline’ cases and issues of school poIicy and philosophy. References Dawes, R. M., and Corrigan, B. (1974). Linear models in decision making. Psychological Bulletin, 81, 95-106.
Hays, Wm. L. (1966). Sfarisrics f o r Psychologists. Holt, Rinehart, and Winston: New York. Rosenberg, M. L. (1973). Increasing the efficiency of medical school admissions. lorrrnal of Medical Education, 48, 707-717. Sawyer, J. (1966). Measurement and predicticn, clinical and statistical. Psychological Bulletin, 66, 178-200. Schofield, W. (1970). A modified actuarial method in the selection of medical students. Jortrnal of Medical Education, 45, 740-744. Schofield, W., and Menvin, J. C. (1966). The use of scholastic aptitude, personality and interest test data in the selection of medical students. Jorirnal of Medical Education, 41, 502-509.
