Validity of the In-Training Examination for Predicting American Board of Internal Medicine Certifying Examination Scores ROBERT S. GROSSMAN, MD, RUTH-MARIE E. FINCHER, MD, RICHARD D. LAYNE, MD, CHARLES B. SEELIG, MD, LEE R. BERKOWITZ, MD, MARK A. LEVINE, MD Objective: To determine whether the results o f the Internal Medicine In- Training Examination (ITE) can p r e d i c t subsequent p e r f o r m a n c e o n the A m e r i c a n B o a r d o f Internal Medicine certifying examination (ABIMCE). Design: Retrospective data r e v i e w . Setting: A mixture o f six community hospital a n d university-based internal medicine training p r o g r a m s in the Eastern United States. Subjects: 109 residents who f i r s t took the ABIMCE in 1988 o r 1989, a n d who had also taken at least one ITE. Measurements: S c o r e s f o r the composite a n d subspecialty sections o f the ITE were compared with those f o r the ABIMCE. A n R 2 was obtained to relate the scores on the two examinations. A cutoff score was derived to m a x i m i z e t h e ability o f the ITE to discriminate between residents who w e r e likely to p a s s a n d t h o s e w h o w e r e likely to f a i l the ABIMCE. Main results: ABIMCE scores were available f o r 109 residents w h o h a d also taken the ITE during PGY-2 (19), PGY-3 (50), o r both y e a r s (40). Composite s c o r e s o n the ABIMCE w e r e highly correlated with t h o s e o n the ITE-PGY-2 ( R 2 = 0.593) a n d the ITE-PGY-3 (U 2 ~ 0.677) ( p < O.O001for each). Most o f the subspeciaily sections o n the two examinations were significantly correlated, although less strongly (range o f R 2 ~ 0 . 0 4 1 t o 0.32) than w e r e t h e composite scores. A n empirically derived cutoff score o f t h e 35th percentile o n the I T E - PGY-2 had a positive predictive t,a!,.~e o f 89% (probability o f p a s s i n g ABIMCE) a n d a negative predictive value o f 83% (probability o f f a i l i n g ABIMCE). Conclusions: P e r f o r m a n c e o n the ITE can accurately predict a n d is highly correlated with p e r f o r m a n c e o n the ABIMCE. ITE results may therefore be useful in c o u n s e l i n g residents about their educational needs in p r e p a r a t i o n f o r the ABIMCE. Key words: certifying examination; in-training examination; education; prediction; residents. J GEN IN-,-zmN MED

1992;7:63-67.

IN 1988 the American College of Physicians, the Association of Professors of Medicine, and the Association of Program Directors in Internal Medicine sponsored the

Received from the Internal Medicine Training Program, Moses H. Cone Memorial Hospital, Greensboro, North Carolina (RSG); the Department of Internal Medicine, Medical College of Georgia, Augusta, Georgia (R-MEF); the Department of Medicine, West Virginia University, Morgantown, West Virginia (RDL); the Internal Medicine Training Program, New Hanover Memorial Hospital, Wilmington, North Carolina (CBS); the Department of Internal Medicine, University of North Carolina, Chapel Hill, North Carolina (RSG, CBS, LRB); and the Department of Internal Medicine, Geisinger Medical Center, Danville, Pennsylvania (MAL). Address correspondence and reprint requests to Dr. Grossman: Department of Medicine, Harrisburg Hospital, South Front Street, Harrisburg, PA 17101.

first national in-training examination (ITE) in internal medicine. This test was designed to assess the cognitive skills of internal m e d i c i n e residents, especially those in the second postgraduate year (PGY-2). With this assessment at the m i d p o i n t of training, residents and program directors should have sufficient time to utilize the results of the examination to focus educational efforts during the r e m a i n d e r of residency. While there is no " g o l d standard" for assessing the clinical abilities of an internist, certification b y the ABIM is a widely recognized standard of a c h i e v e m e n t for internists in the United States. Additionally, p e r f o r m a n c e on the ABIM certifying examination (ABIMCE) predicts s u b s e q u e n t clinical p e r f o r m a n c e as judged b y peers. 1 Because the internal m e d i c i n e ITE is relatively new, there is currently no p u b l i s h e d information relating results of the ITE to those of the ABIMCE. Although 18 other specialty societies or boards sponsor their o w n ITEs, z the ability to predict future p e r f o r m a n c e on board-certifying examinations (predictive validity) has b e e n d e m o n s t r a t e d only for family m e d i c i n e and surgery.2, 3 Construct validity, an indicator of h o w w e l l a test measures what it is intended to measure, has b e e n suggested for the internal m e d i c i n e ITE b y the observation that senior residents score higher than do junior residents in surgery, ophthalmology, orthopedics, and family medicine. 47 We e x a m i n e d the predictive validity of the ITE b y c o m p a r i n g the ITE scores with the ABIMCE scores for a g r o u p of internal m e d i c i n e residents w h o had taken b o t h examinations. We also e x a m i n e d construct validity of the ITE b y c o m p a r i n g second- and third-year scores for a g r o u p of residents w h o had taken the ITE b o t h years.

METHODS Subjects All internal m e d i c i n e residents at the participating institutions w h o had taken the ABIMCE in 1988 or 1989 w e r e eligible if they had released their scores to their program directors and if these results c o u l d also be c o m p a r e d with the results of at least one prior ITE. For the p u r p o s e of testing construct validity, residents w h o had taken the ITE in b o t h their second and third years of training w e r e eligible, including five additional residents w h o had not released their ABIMCE 63

64

OrossmanetaL, PREDICTIVEVALIDITY OF IN-TRAINING EXAM

scores to the p r o g r a m directors. Data w e r e gathered b e t w e e n May and N o v e m b e r 1990 from the following internal m e d i c i n e residency programs: Moses H. Cone Memorial Hospital, Greensboro, NC; Geisinger Medical Center, Danville, PA; New Hanover Hospital, Wilmington, NC; University o f North Carolina School of Medicine, Chapel Hill, NC; Medical College of Georgia, Augusta, GA; and West Virginia University School of Medicine, Morgantown, WV.

Data For the ABIMCE, percentile scores for the entire test as well as for each subspecialty section of the ABIMCE w e r e taken directly from the official examination results report; standardized 5 0 0 / 1 O0 scaled scores w e r e not available. For the ITE, the raw score for the entire examination was c o n v e r t e d into a p e r c e n t i l e score using a table specific for level of training and year (provided with the test for this p u r p o s e b y the American College of Physicians). The individual subspecialty scores, initially r e p o r t e d as " n u m b e r w r o n g , " w e r e converted to " p e r c e n t a g e c o r r e c t " using the n u m b e r of questions in each subspecialty section. This information was included with the test results for 1989, and was provided for the 1988 ITE by a personal c o m m u n i cation from Charles Rossi of the American College of Physicians. Percentile score data w e r e not available for the ITE subspecialty sections. For s o m e residents, the ITE scores w e r e available only from the second or third year of training, while for others b o t h scores w e r e available. Subspecialty scores on the ITE w e r e not available for every resident. Residents w h o w e r e not in the usual July through June cycle w e r e classified as second-year if they w e r e in their 13th to 24th months of training at the time of the ITE and as third-year if they w e r e b e t w e e n their 25th and 36th months. Since m e d i c i n e - p e d i a t r i c residents c o m p l e t e their programs in four years, those taking the ITE in their third year w e r e analyzed with second-year internal m e d i c i n e residents, and those in their fourth year w e r e analyzed w i t h third-year internal m e d i c i n e residents.

Analysis Using SPSS software, s an ordinary least squares regression m o d e l was constructed with the ABIM composite percentile score as the d e p e n d e n t variable and the ITE c o m p o s i t e percentile score as the i n d e p e n d e n t variable. We analyzed the scores for each level of training separately. We also constructed a regression m o d e l c o m p a r i n g the ABIMCE percentile scores for each subspecialty with the second-year resident ITE p e r c e n t a g e correct scores for each corresponding subspecialty section. Pearson correlation coefficients and associated coefficients of determination w e r e obtained for each of

the above relationships. Because of the nonlinear transformation of the scores resulting from o u r use of percentile scores in the regressions, we also c o m p u t e d Spearman's coefficients of rank order correlation for the above relationships. Due to a change in the definition of the reference g r o u p for the ABIMCE b e t w e e n 1988 and 1989, the m i n i m u m passing score was the 21st percentile for 1988 and the 31 st percentile for 1989. As there is no passing score listed for the ITE, sensitivities, specificities, and predictive values w e r e calculated for several ITE cutoff scores. A receiver operating characteristic (ROC) curve was d e v e l o p e d and a " p a s s i n g " cutoff score was empirically selected to maximize the ability of the second-year ITE to predict p e r f o r m a n c e on the ABIMCE.

RESULTS One hundred fifty residents c o m p l e t e d internal m e d i c i n e training at o n e of the six participating institutions and w e r e first eligible to take the ABIMCE in 1988 or 1989. One site did not administer the ITE to any of its 19 third-year residents in 1988, reducing the eligible p o o l of residents to 131. Twenty-two other residents w e r e e x c l u d e d from the study. Eight failed to take either ITE, and 14 w h o w e r e eligible to do so did not take the ABIMCE in 1988 or 1989 or did not release their scores to their p r o g r a m directors. O u r final study cohort consisted of the 109 residents w h o b o t h released their ABIMCE scores and took the ITE as second-year residents (n ---- 19), as third-year residents (n = 50), or as both second- and third-year residents (n -----40). The n u m b e r s of residents included from the six programs were: six, ten, 16, 17, 28, and 32. The overall ABIMCE pass rate for the subjects in o u r study was 72.5%. The ranges, means, and standard errors for the c o m p o s i t e p e r c e n t i l e scores w e r e calculated from individual resident scores (Table 1). The ABIMCE scores for the eight residents w h o had not taken the ITE w e r e not significantly different from the ABIMCE scores for the 109 residents included in this study. The ITE scores for the 14 residents w h o w e r e e x c l u d e d for lack of ABIMCE results w e r e not significantly different from those of the 109 residents included. Scores on the ABIMCE w e r e highly correlated with ITE scores w h e t h e r the examination was taken b y second-year residents (R 2 = 0.593, p < 0 . 0 0 0 1 ) or thirdyear residents (R 2 = 0.677, p < 0 . 0 0 0 1 ) . Subspecialty scores on the ITE taken during the second year were c o m p a r e d with the scores on the corr e s p o n d i n g subspecialty section of the ABIM examination (Table 2). While there w e r e statistically significant correlations for m a n y of the subspecialties, they w e r e w e a k e r than the correlations for the c o m p o s i t e scores.

JOURNALOF GENERALINTERNALMEDICINE, Volume 7 (January/February). 1992

The Spearman's rank order correlation coefficients were virtually identical to the Pearson correlation coefficients for all statistically significant relationships (Appendix A). An ROC curve was constructed to describe the ability of the 1988 ITE taken by second-year residents to predict the results o f the ABIMCE (Figure I). Using the 35th percentile on the second-year ITE as the m i n i m u m "passing" grade, the ITE score has a sensitivity of 95%, a specificity of 67%, a positive predictive value of 89%, and a negative predictive value of 83%. The ITE purports to measure acquisition of cognitive skills in internal medicine. One w o u l d e x p e c t that senior residents w o u l d have better cognitive skills than junior residents. If the ITE possesses construct validity, residents should answer a higher percentage of questions correctly in their third year than they do in their second year of training. Therefore, we c o m p a r e d (using a paired t-test) the percentages of correct ITE scores of

TABLE

(35)

1

n

Mean

SD*

Range

ITE PGY-2 1988 1989

59 0

57.5 --

25.7 --

13-99 --

ITE PGY-3 1988 1989 Combined 1988 and 1989

46 44 90

55.0 57.9 56.3

26.0 23.3 24.7

3 - 96 19 - 99 3-99

ABIMCE 1988 1989 Combined 1988 and 1989

41t 68 109

37.6 52.0 46.6

26.2 26.0 26.9

1-86 2 - 99 1-99

*SD = standard deviation. Wive residents first eligible to take the ABIMCE in 1988 delayed the examination until 1989.

~

, / ~

(19) •

=

:

(40)• 0.80

.>_ o3 t(/)

0.60

0.40,

/~ (68)

0.20

0.00 0.00

Average Percentile Scores on the In-Training Examinations (ITEs) and the American Board of Internal Medicine Certifying Examination (ABIMCE) for Residents Meeting Inclusion Criteria Examination

1.00 -

65

I

I

I

I

I

0.20

0.40

0.60

0.80

1.00

1 - Specificity

FIGURE 1. Receiveroperating characteristic (ROC) curve for InTraining Examination (ITE)vs. AmericanBoardof Internal Medicinecertifying examination (ABIMCE)scores. (Numbers in parentheses are cutoff scores for ITE-PGY-2).

45 second-year residents with the percentages of correct scores on the ITE taken by the same residents during their third year. The mean correct percentages were 68.6% for the second year and 71.6% for the same residents during the third year. The 95% confidence interval for the mean increase in raw scores from the second to the third year of residency was 2.5 - 3.4 percentage points.

DISCUSSION TABLE

2

Relationship between Scores on the Subspecialty Section (n = 55) of the In-Training Examination (ITE)* and Scores on the American Board of Internal Medicine Certifying Examination

Subspecialty Section

No. Items on the ITE

Rz

p Value

Cardiovascular Endocrinology Gastroenterology Hematology Infectious diseases Oncology Respiratory diseases Rheumatology Renal diseases

87 29 36 18 49 32 48 44 49

0.319 O. 161 0.041 O. 166 0.091 0.057 0.056 0.207 O. 147