Regular Articles

Residents' Satisfaction With the PRITE Kenneth L Matthews, M.D. Christopher B. Ticknor, M.D. Examinations are an integral part of resident and program evaluation, hut they are considered particularly stressful on residents. The department of psychiatry of the University of Texas Health Science Center at San Antonio administered the Psychiatry Resident-in-Training Examination (PRITE) every other year to minimize stress and anxiety among residents. When questioned about their satisfaction with the PRITE and its administration, the residents reported high levels of satisfaction and adesire to take the examination yearly. Dissatisfaction was limited to the physical environment in wh ich the exam was administered.

E

xaminations have always played a major part in medical and, more specificalIy, in psychiatrie education. Examinations range from the Medical College Admissions Test (MCAT) to course exams or the National Board of Medical Examiners tests required to graduate from medical school, to clinical exams in residency training, and finally to the written and oral exams required for board certification. All exams offer an assessment in some area, hopefully in the area that the exam is designed to test. This assessment occurs with some costs, including multiple stresses on the examinee; examiner stress, preparation, and delivery time; and loss of service delivery time. The usefulness of exams is shown by the report that residents from medical schools that give grades were considered superior to those from schools that graded pass/fall (1). Dr. Matthews is director of residency training in the department of psychiatry at the University of Texas Health Science Center (UTHSC) at San Antonio, Floyd Curl Drive, San Antonio, Texas 78284-7792. Dr. Ticknor is in private practice in San Antonio and was chief resident in the department of psychiatry at UTHSC, San Antonio, when this paper was written. Copyright «> 1989 Academic Psychilltry.

Few exams have elicited as much attention as the oral portion of the American Board of Psychiatry and Neurology Certification Examination. Langsley expressed concern about the effects of the exam on staff, patients, and examinees (2). Brown reported that knowledge of board objectives and test-taking principles helps one to pass, despite what he perceives as ambiguous crlteria, lack of reliability, and absence of feedback (3). Others openly called for its demise (4), reported casualty figures (5), or shared strategies in avoiding the exam but maintaining self-esteem (6). Veloski and Gonnella addressed factors affecting performance relative to the National Board of Medical Examiners, Part ill, exam (7). Werkman described the value of the Psychiatrie Knowledge and Skills SelfAssessment Program in stimulating selfevaluation and learning without posing the risks associated with other exams (8). Leichner and Kalin discussed similar benefits to Canadian residents of a self-assessment exam (9). The Canadian psychiatrie literature, paralleling the American experience, contains articles outlining positive goals for a \'

.',

\

:

:'

.

\.

certifying exam, expressing concems about who passes and why, and suggesting changes in the exams (10-14). In Canada, the certification process requires the training director to complete an In-Training Evaluation Report (ITER) (15) based in part on an intraining examination. The training director recommends pass or fall for the candidate based on his or her residency performance. Shakun et al. demonstrate that the ITER falls to discriminate good candidates from poor ones for the Canadian certifying exam (16). In fact, they showed no correlation between candidates' performance on multiple-choice questions, oral exams, and the ITER. Klein and Bobineau report on the rational and irrational components of evaluation of trainees, including issues of feedback and consequences of evaluation (17). Kardener et al.' s article on residents' impressions of training experiences does not mention oral or written exams (18). SmalIlucidly argues that rather than any method's being universallyappropriate, oral exams, essays, and multiple-choice questions each have advantages and disadvantages, making each useful in certain situations (19). THE HISTORY OF PRITE

Chaisson discusses the utility of resident-intraining examinations in graduate medical education to measure cognitive information (20). Several psychiatry residency programs have had in-house examinations for residents (21,22). The development of anational Psychiatry Resident-in-Training Examination (PRITE) to replace the exams of individual programs is more recent, dating to 1979. This followed the trend of medical specialties, beginning in 1963 with orthopedics, to develop national in-training resident exams (20). Strauss et al. have examined the composition of the PRITE itself (22), and they compared it to the in-house psychiatry exam developed at the University of California, Los Angeles (UCLA) (23). They found a high

\'

I ~

11 \,

I

\

k..', t ) ,

correlation between the PRREand cognitive knowledge, detecting areasof weakness, cost, and resident satisfactiaft.'Negative eorrelations were found between the PRITE and the amount of informationo.and feedback to residents and programs an(! the amount of time between testing and·l'eSIIIlts. The more positive response of residents to the PRITE was attributed to the novelty of the new exam. The more positive response to the inhouse exam feedback relal!ed to its quicker return even though the resU1ts were less detailed. Strauss et al. noted"that the PRITE is not useful for all programs and named the Psychiatrie Knowledge aadSkills Self-Assessment Program (PKSAP).or the National Board of Medical Examiners'·(NBME) exam as appropriate altematives·{24). In 1985, Colenda reported that 90% of the ten psychiatry residency programs he surveyed used the PRITE, while 60 percent used mock oral boards, and ~had psychotherapy supervision and evaluation (25). Procei' s 1978 survey of 130 residency programs found that 25% used involuntary, nonanonymous written exams, and 8% used voluntary nonanonymous exams (26). Anonymous exams were given involuntarilyat 12% of the programs and voluntarily in 14% of the programs. The department of psychiatry at the University ofTexas HealthScieneeCenter at San Antonio has participated in the PRITE since 1979. The exam has been administered every other year so that residents would take the exam twice. It was feit that the biyearly schedule allowed the residents to gain experience but did not "overburden" them with a yearly exam. The exam is given in strict aecordanee with the guidelines for administration. However, residents are assured that the examination number code will not be broken and the results will remain anonymous, uniess residents choose to volunteer information about their SCOl'e5. Residents are required to take the test. Thus far, only highscoring residents have vohmteered information that would allow their names and

TABLEL Residents' responaes to a questionnaift about the PIUTE,. by number of residents

Qaestion

Highly Satisfied

Moderate1y Satisfied

Neutral or Undecided

Moderately Dissatisfied

Highly Dissatisfied

6

24

3

0

0

13

17

3

0

0

14

15

3

0

16

10

6

0

Definite Yes

Probable Yes

What is your overall reaction to this examination as an educational experience? How satisfied are you with the organization, oontenls, and study value of the itemby-item reference list? How satisfied are you with the type of statistical feedback you receive? How satisfied are you with ~emenls ID ensure co entiality and privacy for your PRITE resulls?

Do you want our residency training program to partidpate in the PRITE in future years? Do you think our program should take the PRITE yearly rather than every other year as we do now? Do you plan to use the PRITE as an educational opportunity to study at least some part of the item-by-item reference list? Do you plan to use the PRITE at some point in preparing for board certification exams? Would you like to have the option to discuss the results of the PRITE with your career supervisor or with training director?

Neutral

Probable No

Definite No

28

5

0

0

0

11

13

5

4

0

19

10

3

0

1

18

14

0

0

2

10

5

1

exams to be matched. The chairman and training director have been encouraged repeatedly to administer the PRITE on a yearly basis. Out of concem for stress and anxiety endured by residents, the San Antonio program has continued biyearly administration. In 1986, when the issue of a yearly exam was raised again, one of the authors (Cl') suggested the novel approach of asking the residents their opinions about the PRITE.

15

THE QUESTIONNAIRE

A ten-item questionnaire was developed and distributed to the 40 residents. Responses were obtained from 33 residents. Since the questionnaire, like the PRITE, was answered anonymously, nonresponders could not be identified. While obtaining the residents' views about taking the exam yearly was the primary focus of the survey, we also inquired about their satisfaction 1[,\1 1

·'l\II.I,.

~

with the exam and its expected uses. Table 1 summarizes responses to the first nine questions. The tenth question asked respondents what could be done to make the PRITE a better educational experience at our institution. Seven respondents asked for coffee, cookies or donuts. One resident requested softer chairs, and one requested a general meeting to discuss the results as they related to the strengths and weaknesses of the residency program.

DISCUSSION The respondents were satisfied with the organization of the test, the statistical feedback received, and the anonymous confidentiality arrangements we employ. The only areas in which any respondent reported feeling highly dissatisfied involved feedback and confidentiality, and both responses were given by the same resident. The residents' overall satisfaction with the exam is reinforced by the respondents' suggestions for changing the exam. Of those responses, 21 % were requests for food, 3% were for better environment, and 3% were for more detailed feedback. All in all, residents' satisfaction was high. Surprisingly, our results found that the residents wanted more exams rather than fewer. They preferred yearly administration

of the tests. The expected stress and anger or self-esteem challenge was not as high as expected by the faculty, particularly in terms of the PRITE. The required but anonymous participation helps moderate some stress and self-esteem issues. Thus all residents wanted to continue the PRITE, and all but four desired, or said they would be content with, yearly administration. Twenty-nine residents (88%) indicated they intended to study at least part of the results of the PRITE as an educational activity. This confirms the test's validity as an educational experience, as it is intended to be, in addition to it value as an evaluative exercise. More surprisingly, 32 residents (97%) intended to use the PRITE results to prepare for board certification. The high percentage of residents anticipating board certification was unexpected. When presented with the option of discussing individually the results of PRITE with the training director, only 12 residents (36%) expressed interest. Six residents (18%) said they would refuse the option. We assume that this division parallels resid~ts' performance; those whose score~ were above the mean performance would probably be receptive to a discussion, and those who scored below the mean would tend to be unreceptive.

Refermces 1. Carmel H, Amini F: Comparison of the performance of psychiatrie residents from pass/fai! versus graded medical schools. J Med Educ 1979; 54:901902

2. Langsley DC: Changing patterns of psychiatry specialtycertification in the English-speaking countries. AmJ Psychiatry 1981; 138:493--497 3. Brown BM: An examinee's perspective on Board certification. AmJ Psychiatry 1977; 134:1261-1264 4. Gardos G: The Oral Board Examination: a call for its demise. JClin Psychiatry 1980; 41 :422-424 5. Lipp MR: Experiences of psychiatrie board exam casualties: a survey report. Am J Psychiatry 1976;

133:279-283

6. Patterson DY: How to avoid taking the boards but save face. Am JPsychiatry 1975; 132:79-80 7. Veloski J, Gonnella J5: Performance on Part mof the National Boards-the effect of residency training. Annual Conference on Resident Medical Education 1980; 19:142-147 8. Werkman 5L: An opportunity to stimulate self-education and learning. Am JPsychiatry 1976; 133:215216

9. Leichner PP, Kalin R: Results of the first Canadian psychiatrie knowledgeself-assessment forresidents. Can J Psychiatry 1980; 25:281-290 10. McLean P, Fleming J: Evaluation of clinical competence in psychiatry by the Royal College Examina-

tion. CanadianPsychiatrie Association Journal 1978; 23:521-529 11. Leichner PP: 1'he certification examination; viewpoints of past cudidates. Canadian Psychiatrie AssociationJournaH978; 23:531-540 12. Lowy FH, Jon~ HO: The Canadian eertification examination in psychlatry. I: historical roles. Canadian Psychiatrie AssociationJournal1979; 24:275-284 13. Lowy FH, Dongier M: The Canadian certification examination in psychiatry, ß: who passes and who fails.CanadianPsychiatrieAssociationJournaI1979; 24:284-292 14. Persad E, Garfinkel P: Practicing psyehiatrists' views of the certification examination in psyehiatry. Canadian Psychiatrie Association Journal 1979; 24:303-307 15. Lowy FH, Prosen H: The Canadian certification examination in psychiatry, ill: toward better certification techniques. Canadian Psychiatrie Association Iournall979; 24:292-301 16. Shakun EN, Wilson DR, Taylor WC: A comparison of performance on multiple ehoice tests, oral examinations, and in-training evaluation reports. Annals of Royal College of Physicians and Surgeons of Canada 1977; 10:284-285 17. Klein RF, Babineau R: Evaluating the eompetence of trainees: It's nothing personal. Am J Psychiatry 1974; 131:788-791

18. Kardener SH, Fuller M, Mench I, et al: The trainee' s viewpoint of psychiatrie residency. Am J Psychiatry 1970; 126:1132-1138 19. Small SM: Limitations and values of evaluation techniques in psychiatrie education. Am I Psychiatry 1975; 132:52-55 20. Chaisson GM: The development and utilization of in-training examinations in graduate medical education. I Med Edue 1978; 53:502-504 21. Woods SM: A computerized self-assessment examinationforresidents. AmI Psychiatry 1974; 131:1283-

1286

22. Strauss GD, Vager J, Strauss GE: Assessing assessment: the content and quality of the psychiatrie intraining examination. Am J Psyehiatry 1982; 139:85-88 23. Strauss GD, Vager I, Liston EH: A eomparison of national and in-house examinations of psychiatrie knowledge. Am I Psychiatry 1984; 141 :882-884 24. Strauss GD, Vager I, Liston EH, et al: Testing psychiatrie knowledge with in-house examinations. Am I Psychiatry 1981; 138:636-641 25. Colenda CC: A survey of ten psychiatrie residency programs on procedures for evaluating residents. I Med Edue 1985; 60:886-888 26. Procci W, Friedmann C, Prizant G, et al: Anational survey of resident assessment methods. Am I Psychiatry 1978; 135:845-847

l

'

i

•

\

\ 1 1 r j,'

•

I

-