A Study of the Board Certification Examination in Child and Adolescent Psychiatry PETER E. TANGUAY , M.D., JOHN F. McDERMOTI, JR., M.D., IRVING PHILIPS, M.D .

AND

Abstract: In 1977, training directors of 55 child psychiatry training programs agreed to provide the Committee on Certification in Child Psychiatry of the American Board of Psychiatry and Neurology , Inc . with confidential data about the clinical performance of each of their graduates in 1978, 1979, and 1980. Information about 267 graduates from 37 programs was received. Almost all programs were university affiliated and located in medical centers. By 1986, 112 of these graduates had taken the child aboard examination. Training director ratings were found to be statistically correlated to a significant degree with actual pass or fail results on the written and on the overall examination. J.Am. Acad. Child Adole sc . Psychiatry, 1990, 29, 5:821-827 . Key Words: child psychiatry, board certification, validity, recertification .

In April 1976, at a conference convened by the American Board of Psychiatry and Neurology at Silverado, in Napa, California, representatives from academic and professional groups met to discuss ways in which the curricula for training child and adolescent psychiatrists could be improved. In reviewing the continuum of medical education from specialty training to board certification and possible recertification, it was proposed by one of the authors (IP) that a study be done on the degree to which training directors could correctly predict how their graduates would fare on the certification examination in child and adolescent psychiatry. Though initially seen as a study of how training directors might become part of the certification process, it later became apparent that the project might serve as a measure of the validity of the examination itself. It was realized that the measure would be imperfect, since training directors might be biased in favor of their graduates. It was hoped that the results could at least serve as a benchmark against which the success of future attempts to improve the exam could be measured. There have been many previous studies of the validity of AcceptedJanuary4.1990. Dr . Tanguay is Professor of Psychiatry and Associate Director . Division of Child and Adolescent Psychiatry, UCLA Neuropsy chiatric Institute, Los Angeles, California . Dr . McDermott is Professor ofPsychiatry and Chairman , Department ofPsychiatry, University of Hawaii School of Medicine , Hon olulu, Hawaii. Dr. Philips is Professor of Psychiatry, University of Californ ia Medical Center , San Francisco , California . Portions ofthis paper were presented at a Research Conference ofthe American Board ofMedical Specialists in Chicago in September 1986 . The results were also presented to the Directors ofthe American Board of Psychiatry and Neurolo gy (ABPN) meeting in Los Angeles in June 1987, and at the ABPN Conference on Lifetime Learning, Certification and Recertification held at Salishan Lodge , Oregon , in July 1989. The opinions in this report are those of the authors and do not necessarily reflect the views of the American Board of Psychiatry and Neurology , Inc . The authors thank the staff of the ABPNfor their help in assembling the questionnaires and providing the examination results and Donald Guthrie, Ph.D. .forstatistical consultation. Reprint requests to Dr. Tanguay, UCLA Neuropsychiatric Institute, 760 Westwood Plaza, Los Angeles , CA 90024-1759. 0890-8567/90/2905-08211$02 .00/0 © 1990 by the American Academy of Child and Adolescent Psychiatry.

821

the board examination process . Member boards of the American Board of Medical Specialties (ABMS) have spent considerable money and effort in these investigations. In 1986, seven of the 23 member boards were in the process of carrying out such studies (ABMS Committee on the Study of Examination Procedures Conference, 1986), several of which were quite ambitious. The American Board of Internal Medicine (Webster et al. , 1985) compared the (written) examination scores of 4,590 first-time candidates to program director ratings of cand idate skills and attributes. Candidates scoring in the upper range of the "overall competence" category were found to have higher mean exam scores than those who scored in the lower range. A study by the American Board of Colon and Rectal Surgery (Boggs and Dolch, 1985) asked program directors to evaluate candidates on 21 items. Only one item, the candidates "overall knowledge score," had a high correlation (0.41) with exam performance. A study by the American Board of Pediatrics (ABP) (Brownlee and Butzin, 1985) compared program directors' ratings oftheir graduates with the results of the ABP's written exam. Ninety-one percent of the 1,864 candidates who had been rated "9" passed the written exam, while only 52% or less of those rated as "5" or below did so. Some highly rated candidates received very low scores on the exam, and conversely, some poorly rated candidates did very well. A reanalysis of a sample of the data suggested that two factors may have played a role in this discrepancy. Candidates who received very low scores on the exam came from programs whose graduates tended to do badly on the exam . It was postulated that the training director's high ratings in such instances may have been a reflection of their low standards and expectations. Some graduates who had been highly rated but who did badly were foreign born. They may have had good clinical skills, but they might have done poorly on the written exam because of language difficulties. Of more immediate interest to the present study was an investigation carried out by Langsley and Lloyd (1985) for the ABMS, using data provided by the American Board of Psychiatry and Neurology (ABPN). Because the ABPN does not ask program directors for detailed information about their

TANGUAY ET AL

graduates, a questionnaire was used to collect data for the project. Questions were asked about the skills, knowledge, and attitudes thought to be important in the practice of general psychiatry. With the help of the American Association of Directors of Psychiatric Residency Training, performance ratings were obtained on 476 graduating residents in 56 different training programs. Candidates were rated on 26 items, including physician-patient interaction, capacity to conduct a clinical exam, capacity to elicit clinical data, formulation and differential diagnosis, and case management skills. By the early 1980s, when data analysis was begun, 364 of the graduates had taken the ABPN exam. Several shortcomings in the data limit the conclusions that can be drawn from the study. Training director questionnaires were complete for only 65 examinees, with the remaining questionnaires being reasonably but not fully complete. The ratings on all 26 items were averaged into a single score. Only the results of the written examination were studied. Candidates' performances on the examination were astonishingly high: greater than 90% of the candidates passed. Candidate scores were also tightly clustered together. It is possible that it was mostly training programs of excellence that had been sampled. The authors do not comment on this issue. The high pass rate and tight clustering of scores made it less likely that training director ratings would distinguish between levels of exam performance. Despite this, the authors were able to show that when candidates were grouped into four levels of estimated skill, the average exam scores and the percentage of candidates who passed rose from the lowest to the highest groups. A reanalysis by the authors of the raw data given in the paper suggested that the group differences were not statistically significant. The Child Psychiatry Board Certification Examination

In order to understand the present study in context, it is important to have an understanding of the nature of the certification exam itself. Since its inception in 1959, the child and adolescent examination of the ABPN has undergone substantial changes aimed at increasing its reliability and validity. In 1970, the Child Psychiatry Examination Committee of the ABPN, with the help of the Center For Educational Development at the University of Illinois College of Medicine began a systematic evaluation ofthe examination process (McDermott, 1975). This led to the replacement of two of the oral exams with a multiple-choice written exam. The three remaining oral examinations were improved by better defining the behavioral objectives to be measured, reducing unintended overlaps between exams, and improving examiner techniques. Equally important was the introduction of preexamination training sessions for examiners, and the use of standard audio-visual case presentations for groups of candidates. In 1973, a second study was begun, this one aimed at improving the relevance of the examination to clinical practice (McDermott et al., 1977). A stratified random sample of 506 practicing child psychiatrists (50% of the membership of the American Academy of Child Psychiatry at the time) was sent a questionnaire requesting detailed information

822

about their professional activities in a typical work week. The psychiatrists were also asked to provide two descriptions of , 'critical incidents": one in which they (or a colleague) had done something that they considered particularly effective; the other, an incident in which they or a colleague had done something that they considered clinically ineffective. Respondents were asked to describe the setting in which the incident occurred, exactly what was done, and why they considered it effective or ineffective. Roughly 60% of those surveyed responded. In addition, 227 pediatricians and 100 juvenile court judges, selected to reflect the geographic distribution of American child psychiatrists, were asked what types of assistance they may have sought and received from child psychiatrists, and whether the consultation had been helpful. They were also asked to provide two critical incidents describing such consultations, one in which the child psychiatrist had been helpful, and one where he or she had not. Forty percent of the pediatricians and 22% of the judges responded. The definition of essential skills derived from this exercise have been described in a previous monograph (McDermott et al., 1976). Many of the skills were already being emphasized in the adolescent and the audio-visual exams. Based upon the critical incidents reported, written vignettes describing specific school, court, or community agency requests for consultation were introduced as stimulus materials into the consultation-liaison examination as described below. The current Board Certification Examination in Child and Adolescent Psychiatry consists of written and oral components. A library of questions for the written examination is maintained. Library questions are categorized along several dimensions: what years were they used, what percent of candidates passed the item each time it was used, and how well the item distinguished between those who passed and failed the entire exam. Library questions and new questions are used to construct the exam each year. Library questions are reviewed in terms of their current accuracy, timeliness, and usefulness in distinguishing the more knowledgeable from less knowledgeable candidates in the past. Questions that fail to meet these criteria are dropped from the library. In scoring the written exam, performance of the candidates on each of the pool questions is assessed, and the pass/fail cutoff is set so that the examination will have the same statistical difficulty as did previous examinations. Initially, in the late 1970s, the pass/fail cutoff was arbitrarily set so that the lowest 15% of scores would be listed as' 'fail. " Since then, using the method described above for automatically generating a pass/fail cut point, the failure rate has fallen steadily and has been as little as 6% of candidates some years. At the time of the present study, four oral examinations were given each candidate. One involved a 30-minute evaluation of an adolescent patient, following which candidates were asked to summarize the salient features of the case, formulate a differential diagnosis, and discuss treatment. Two additional examinations used videotapes of individual children as stimulus materials; one presenting a preschool child, and the other a school-aged child. Each child had been chosen to demonstrate a specific psychiatric diagnosis or syndrome. A fourth oral exam presented candidates with written vignettes describing a consultation request from a l.Am.Acad. Child Adolesc.Psychiatry, 29:5, September 1990

CHILD AND ADOLESCENT CERTIFICATION EXAM TABLE

1. High Pass/Pass Fail Prediction by TD versus Exam Result

Gamma Written exam Entire exam

0.467 0.838

ASE 0.169 0.080

z 2.76 10.40

p

A study of the Board Certification Examination in Child and Adolescent Psychiatry.

In 1977, training directors of 55 child psychiatry training programs agreed to provide the Committee on Certification in Child Psychiatry of the Ameri...
7MB Sizes 0 Downloads 0 Views