Interns' Performances with Simulated Patients at the Beginning and the End of the Intern Year J. JILL GORDON, MBBS, PhD, NICHOLAS A. SAUNDERS, MD, DEBORAH HENNRIKUS, BA, MA, PhD, ROBERT W. SANSON-FISHER, BPsych (Hons), MPsych, PhD Objective: To determine whether interns'performances o f technical, preventive, a n d communication aspects o f patient care improve during the intern year. Design: A descriptive study. A t the beginning a n d end o f the intern year, interns" consultations with three sin~.:!ated ( s t a n d a r d i z e d ) patients were videotaped a n d scored according to explicit criteria set by an expertpaneL Problems simulated were urinary tract infection, bronchitis, a n d headache. Setting: The casualty outpatient department in a g e n e r a l teaching hospital in New South Wales, Australia~ Participants: Twenty-eight interns rotated to ate ctzsualty
departmen~ Results: Little improvement over the intern y e a r in technical competence o r preventive care was o b s e r ~ d , even though initial levels o f compliance with criteria were quite low f o r some items. Greater improvement was apparent in the area o f communication skills. Conclusions: The results suggest that the internship should be restructured to more adequately teach the skills required f o r p r i m a r y care. Key words: internships; simulated patients; preventive c a r e ; p h y s i c i a n - p a t i e n t coiiii~iunication,.performance assessment. J GEN INiratN MED 1992;7:57-62.
IN NEW SOUTH WALES, Australia, the attainment of an undergraduate degree in m e d i c i n e is followed b y a c o m p u l s o r y year of internship. Internship represents the c o m p l e t i o n of basic medical education; its p u r p o s e is to afford the recent graduate balanced clinical experience with increasing responsibility for the care of patients, t Tasks of the intern year are intended to dev e l o p cognitive, procedural, interpersonal, counseling, and administrative skills. 2 By the end of the year, interns have s o m e e x p e r i e n c e in medicine, surgery, and a m b u l a t o r y care. They are then registered for indep e n d e n t practice and eligible for specialty training. In a previous study involving 56 interns w h o saw covert simulated patients in a casualty setting, substantial deficiencies in the interns' preventive care and c o m m u n i c a t i o n skills w e r e identified. 3 Suggested explanations for these findings w e r e that the interns had not b e e n taught these skills as undergraduates, that Received from the Royal Australian College of General Practitioners, Macquarie Hospital, North Ryde (JJG), and the Faculty of Medicine, The University of Newcastle (NAS, DH, RWS-F), New South Wales, Australia. Supported by grants from the Australian Institute of Health and the NSW State Cancer Council. Address correspondence and reprint requests to Dr. SansonFisher: Behavioural Science in Relation to Medicine, Faculty of Medicine, University of Newcastle, New South Wales, 2308, Australia.
these skills w e r e suppressed in the first w e e k s of the internship w h e n the study took place, that the w o r k e n v i r o n m e n t discouraged these aspects of patient care, and that the interns themselves c o u l d not see the value o f these aspects o f care. If the intern year is fulfilling its p u r p o s e of completing the process of basic medical education, then measurable i m p r o v e m e n t should o c c u r b e t w e e n the beginning and the end of the year. Despite the crucial role of the intern year in the transition from undergraduate status to i n d e p e n d e n t practice, w e are unaware of studies that have a t t e m p t e d to d o c u m e n t the differential i m p a c t of the internship on p e r f o r m a n c e s of technical, preventive, and c o m m u n i c a t i o n skills. Stillman et al. 4, 5 have d e v e l o p e d the t e c h n i q u e of using overt standardized patients to assess the clinical skills of residents in internal medicine. T h e y have demonstrated the validity and reliability of this t e c h n i q u e and have suggested that the m e t h o d also provides opportunities to test skills, such as c o m m u n i c a t i o n skills, not easily assessed by other means w h i l e controlling for factors such as case difficulty. Simulated patients have also demonstrated their utility in the a m b u l a t o r y care setting w h e n used covertly. 68 The aims of the present study w e r e therefore to measure changes b e t w e e n the beginning and the end of internship in the interns' technical, preventive, and c o m m u n i c a t i o n skills, using covert simulated patients.
SETTING, SUBJECTS, AND METHODS The setting for the study was the Casualty Departm e n t of the Royal Newcastle Hospital, a 6 5 0 - b e d general teaching hospital located in the inner city. The region that it serves has a p o p u l a t i o n of a p p r o x i m a t e l y 350,000. The hospital is s u r r o u n d e d b y low-cost rental a c c o m m o d a t i o n , and m a n y of the patients w h o use the hospital as a source of p r i m a r y care live in the immediate area. It is the policy of the casualty d e p a r t m e n t to provide treatment for a m b u l a t o r y patients, but to encourage t h e m to consult a p r i m a r y care physician for ongoing management. All interns starting at the hospital at the beginning of year w e r e asked to participate in a study that involved the videotaping of consultations with simulated patients. The interns did not k n o w h o w m a n y patients w e r e to be presented but they w e r e aware that videotaping w o u l d o c c u r within a two-day period w h e n they $7
$8
Gordon etaL, PERFORMANCEAT BEGINNINGAND END OF INTERNSHIP
were specially rotated to the casualty department in the first and last ten weeks of their internships. Despite these cues, the interns were not able to accurately identify the simulators from among genuine patients. They also reported that the study generally had little effect on their usual consulting behaviors.9 Thirty-five of 41 interns at the hospital were available during the study period; all 35 consented to participate at the beginning of the year. Thirty-two (91%) consented to repeat the study at the end of the year. Four of these interns were unable to participate because of transfers to other hospitals, leaving 28 subjects for w h o m beginning- and end-of-year comparisons c o u l d be made. There was no significant difference in the ages, gender distributions, or universities of graduation between those w h o were available and consented to repeat the study and those w h o were not. At both the beginning and the end of the intern year, three consultations with simulated patients were videotaped during the two successive days of the casualty attachment. Each consulting cubicle in the casualty department had an unobtrusive camera placed close to the ceiling, and directed at the desk at w h i c h the intern and the patient sat during the interview. Simulated patients were recruited from the Commonwealth Employment Service and from the local Old Aged Pensioners' Association. Those w h o were selected had clinical histories that resembled most closely the patients w h o m they were to portray. Training followed the m e t h o d that was described by Barrows, lo with three one-hour training sessions. The patient's o w n medical and social history was used as m u c h as possible. Each simulated patient pretended to have a clinical p r o b l e m that was selected on the basis of being comm o n and potentially preventable and that required neither referral nor invasive diagnostic or management procedures. The patients were instructed to provide each intern with clearly defined information about their clinical problems. Their reliability in presenting this information was assessed by a research assistant during the videorecording, and later from a review of the videotapes. The interns were presented with the same three simulations at the end of the year, in order to standardize the evaluation. The simulators portraying the patients were changed at the end of the year to avoid the possibility of recognition. The n e w simulators were trained using the same m e t h o d as had been used for the first g r o u p at the beginning o f the year. Abbreviated versions of the patients' histories illustrate the opportunities that were provided for interns to demonstrate clinical, interpersonal, and preventive skills:
Case 1: u r i n a r y t r a c t i n f e c t i o n ( U T I ) i n a 2 2 - y e a r - o l d w o m a n . This patient presented with a past history of
cystitis. Her symptoms of dysuria and frequency of mic-
turition commenced a few days after the return of her boyfriend, who had been away for several months. She was unemployed, smoked 15 cigarettes each day, and drank to the point of feeling intoxicated once or twice each week. She took paracetamol for headaches and menstrual pains. She had no regular family doctor. Case 2: h e a d a c h e in a 1 9 - y e a r - o l d m a n . This patient presented with symptoms that were typical of tension headache that had lasted for the past few months. He was unemployed and lived in the inner city. He smoked 20 or more cigarettes each day, used marijuana regularly, and drank alcohol as often as he could afford to - - usually a few times per week. He was having problems in his relationship with his girlfriend and appeared lonely and depressed. He had no regular family doctor. Case 3: b r o n c h i t i s i n a 6 7 - y e a r . o l d m a n . This patient presented with chronic airflow limitation as a result of smoking in excess of 20 cigarettes per day for 50 years. He presented with increasing cough and breathlessness and had lost about 4 kg in weight in the last six months. He admitted to drinking at least 40 g of alcohol per day. He lived alone and had no family doctor.
Criteria for Care An expert panel was c o n v e n e d to develop explicit criteria for the management of each patient by an intern in a casualty department. The six-member panel comprised two family physicians, two internists, and two housestaff m e m b e r s - - one m e m b e r in his second year and one m e m b e r in his fourth year of training. Panel members were given full patient profiles on w h i c h to base their decisions. Each began by modifying a criterion list that was c o m p i l e d by us from standard sources; then they met to review each other's responses. Further individual modifications were made after the meeting and final decisions then were reached by consensus at a s e c o n d meeting. The panel divided the criteria into essential and nonessential items. Essential features of the consultation were used to establish a minimal level of performance. Three groups of behaviors were considered by the panel. The first was technical care related to historytaking, investigations, and management. The second was preventive care related to the eliciting of relevant items in the history or to the provision of specific advice c o n c e r n i n g the patient's presenting problems: for example, antismoking advice for the patient with chronic airway obstruction. Opportunities for providing preventive advice that were not related to the problem (for example, advice about smoking to the w o m a n with a UTI) also were noted. Finally, c o m m u n i c a t i o n skills were based on behaviors that have been shown to influence patient satisfaction and c o m p l i a n c e with treatment. Seven behaviors considered essential for effective d o c t o r - p a t i e n t c o m m u n i c a t i o n were identified.
JOURNALOFGENERALINTERNALMEDICINE,Volume 7 (January/February), 1992
The videotapes were scored by two trained raters, both of w h o m were nurse educators. The explicit criteria that they used for scoring included examples of statements that the interns might use. Each behavior was marked present or absent. To check interrater reliability, each rater scored a one-in-four random sample of the other rater's videotapes. The percentage of consultations in which the observers agreed u p o n w h e t h e r the behavior had o c c u r r e d was calculated for each behavior.
RESULTS Videotapes for both the beginning and the end of the year were obtained for 25 interns consulting with the simulators with UTI, 23 interns consulting with the simulators with headache, and 25 interns consulting with the simulators with bronchitis.
Reliability of Patients' Presentations The reliability of each simulated patient's presentation was checked on every occasion by the research assistant, w h o viewed the videorecording while the consultation was in progress. The check on reliability of performance consisted of a check that the simulated patient portrayed the essential features of each case. The reliability o f performance was 100%.
Reliability of Videotape Coding All items were c o d e d with acceptable reliability. Agreement b e t w e e n the two observers was at least 80% for all 83 items rated. One hundred percent agreement was achieved for 33 items and 90% agreement or better for 72 items.
Competence in Technical Care Table 1 summarizes the criteria determined by the expert panel to be essential for each of the three patients' problems. Tables 2, 3, and 4 highlight those technical aspects of care considered essential by the expert panel but p o o r l y adhered to by the interns. Items in w h i c h the interns performed well at both the beginning and the end of the intern year have been omitted for clarity of presentation. Generally, the interns demonstrated good ability in obtaining an adequate history of each patient's current p r o b l e m during both testing periods. Overall, the interns' compliance with the essential criteria for technical care did not change significantly for any of the three simulated patient problems. Isolated changes occurred in each problem. For the patients with UTIs, all of the interns ordered urinalysis and cultures at the end of the year, c o m p a r e d with only 64% at the beginning. More interns included an inquiry about the sleep pattern of the patient with tension headache, suggesting that the clinical possibility of depression was being explored, and more provided appropriate treatment of this patient at the end of the year than at the beginning. The e x p e r t panel accepted as appropriate treatment in this case an explanation of the nature of the headache along with suggestions for stress management a n d / o r management of the patient's depressive symptoms. The prescribing of narcotic analgesics was considered inappropriate treatment. By contrast, fewer interns made adequate arrangements for the follow-up care of the patient with bronchitis at the end of the year. Significant omissions o c c u r r e d at both the beginning and the end of the internships in taking histories concerning current medications from the patients with UTIs and tension headaches, obtaining adequate past
TABLE 1 Summary of Expert Panel's Essential Criteria for TechnicalCompetence in Interns' Management of Three Problems Urinary Tract Infection History
Main complaint Duration Pain
Past history Medications
Tension Headache
Bronchitis
Main complaint Duration Headache length Location Quality Context Migraine history Sleep pattern Past history Medications
Main complaint Duration Cough Sputum Breathlessness Weight loss Cardiovascular symptoms Last chest x-ray Past history Medications
Diagnostic tests
Urinalysis Culture and sensitivity
None should be done
Spirometry
Treatment
Appropriate antibiotic
Lifestyle counselingby self or referral
A range of symptomatic treatment options approved by the panel (e.g., alteration of salbutamol dose) Further investigation is mandatory
Follow-up
59
60
Gordon eta/.. PERFORMANCEAT BEGINNINGAND END OF INTERNSHIP
TABLE 2 Numbers (%) of InternsWho Satisfiedthe Minimum Criteria for TechnicalCare, Beginningvs. End of Internship: Urinary Tract Infection (n = 25) Criterion
Beginning
End
Past history Medications Investigations Follow-up
11 (44%) 6 (24%) 16 (64%) 19 (76%)
15 (60%) 7 (28%) 25 (100%) 21 (84%)
TABLE 3 Numbers (%) of Interns Who Satisfiedthe Minimum Criteria for TechnicalCare, Beginningvs. End of Internship: TensionHeadache (n = 23) Criterion
Beginning
End
Headachelength Headachecontext Sleep pattern Past history Medications Investigations Treatment Follow-up
11 (48%) 15 (65%) 7 (30%) 7 (30%) 8 (35%) 13 (57%) 10 (43%) 10 (43%)
18 (78%) 20 (87%) 15 (65%) 9 (39%) 6 (26%) 18 (78%) 17 (74%) 15 (65%)
TABLE 4 Numbers (%) of InternsWho Satisfiedthe Minimum Criteria for TechnicalCare, Beginningvs. End of Internship:Bronchitis(n = 25) Criterion
Beginning
End
Breathlessness Weight loss Cardiovascularsymptoms Last chestx-ray Past history Medications Investigations Follow-up
17 (68%) 1B (60%) 19 (72%) 12 (48%) 17 (68%) 21 (84%) 4 (16%) 24 (96%)
17 (68%) 14 (56%) 13 (52%) 14 (56%) 13 (52%) 18 (72%) 2 (8%) 17 (68%)
histories in all three cases, and measuring respiratory function (spirometry) in the patient w i t h chronic respiratory disease. Competence in Preventive Care Performance for preventive advice for the patients w i t h UTIs did not change (Table 5), suggesting that the interns did not take u p s o m e o f the practical preventive strategies identified by the e x p e r t panel. In particular, the interns failed to give the patient advice on h o w to deal w i t h a r e c u r r e n c e of this p r o b l e m . There was an increase in the n u m b e r of interns w h o dealt w i t h the p r o b l e m of depression as it presented in the y o u n g man with tension headache (Table 6). Only one intern in 23 had addressed this p r o b l e m at the beginning of the year, but seven interns a t t e m p t e d to offer h e l p in this area in the end-of-year study. Nevertheless, 70% of interns at the end of the year made no
a t t e m p t to deal w i t h an issue that was the key to the patient's somatic symptom. More interns p r o v i d e d advice a b o u t smoking for the patient with bronchitis at the end of the year, and m o r e interns suggested ways of stopping smoking (Table 7). Nevertheless, given the central role of smoking in the pathogenesis of this patient's clinical problem, it was disappointing to see that antismoking advice did not a p p e a r in every consultation. Overall, the n u m b e r s of interns satisfying the essential criteria w e r e low. Fewer than 30% satisfied any of the preventive care criteria for UTI and f e w e r than 40% satisfied any of the preventive care criteria for tension headache. The only simulated patient for w h o m more than half of the interns a p p r o a c h e d a satisfactory standard was the patient with chronic airflow limitation, for w h o m smoking was a central risk factor. However, fewer than 25% of the interns p r o v i d e d adequate advice in relation to c o p i n g with recurrences in the end-of-year consultations. Competence in Communication Skills While changes in c o m m u n i c a t i o n skills w e r e few, there was a trend toward greater c o m p l i a n c e with the criteria in the end-of-year study (Tables 8, 9, and 10). The most notable i m p r o v e m e n t was in explaining the pathogeneses of the patients' problems. Whereas only 40%, 30%, and 20% of the interns p r o v i d e d explanations at the beginning of the year, by the end of the year these percentages had increased to 64%, 61%, and 100%, respectively. Interns w e r e also m o r e likely to explain the plan of m a n a g e m e n t to the simulator with UTI. They w e r e m o r e likely to provide the simulator w h o had tension headache w i t h an explanation for their decisions not to order unnecessary investigations and to explain treatment decisions. There w e r e two areas of persistent weakness: s u m m a r i z i n g the p r o b l e m and ensuring patient understanding.
DISCUSSION The study of interns' technical, interpersonal, and preventive care skills at the beginning of the intern year identified a n u m b e r of deficiencies that m a y have reflected failings in their undergraduate education. Because the p e r f o r m a n c e assessment c o v e r e d c o m m o n a m b u l a t o r y care conditions, and because the essential criteria carefully avoided esoteric items, these deficiencies w e r e important. A year of internship offered s c o p e for the interns to i m p r o v e their skills. This study found that technical proficiency did not change significantly over the course of the year. While proficiency was s h o w n in s o m e areas at b o t h the beginning and the end of internship, there w e r e persistent deficits in others: for e x a m p l e , in obtaining an adequate past history and information a b o u t m e d i c a t i o n
JOURNALOFGENERALINTERNALMEDICINE,Volume 7 (January/February), 1992
use. Arrangements to ensure follow-up o f all three patients were poor. This is perhaps not surprising in an environment that discourages continuity of care, but it is p o o r preparation for the responsibilities of independent medical practice. Preventive care, w h i c h had the greatest scope for improvement, was persistently poor. The only case in w h i c h it approached adequacy was the case of the simulator with bronchitis, where the cause of the chronic respiratory impairment was obvious and the impact of continued smoking made successful treatment difficult. TABLE 5
Numbers (%) of Interns Who Satisfied the Minimum Counseling Criteria for Preventive Care, Beginningvs. End of Internship: Urinary Tract Infection (n = 25) Counseling Criterion
Beginning
Fluid intake Voiding frequency Sexual behavior Managing recurrences
4 (16%) 5 (20%) B (20%) 4 (16%)
End 6 7 6 6
(24%) (28%) (24%) (24%)
61
1. A c o m m i t m e n t by h o s p i t a l s to i n t e r n educat i o n as w e l l as service. One of the major problems with
internship training in Australia is the dominance of hospitals' service needs over their obligation to provide training. 11 Few intern training programs offer planned curricula that address the interns' needs to acquire technical, preventive, or interpersonal care skills.12 2. A r e o r i e n t a t i o n to the h e a l t h needs o f the pat i e n t s r e c e i v i n g care. Thirty years ago, White et al. ~3
estimated that the patients w h o receive treatment in teaching hospitals represent only a tiny p r o p o r t i o n of those w h o are ill. Modern teaching hospitals have moved even further in the direction of short-stay admissions for investigative and therapeutic procedures or admissions to intensive care units for the critically ill. They are therefore less and less representative of the real health needs of the communities that they serve. By contrast, patients presenting to hospital casualty departments, e m e r g e n c y rooms, and primary care clinics have a m u c h wider variety of health care needs. Using
TABLE 6
TABLE 8
Numbers (%) of Interns Who Satisfied the Minimum Counseling Criteria for Preventive Care, Beginningvs. End of Internship: Tension Headache (n = 23)
Numbers (%) of Interns Who Satisfied the Minimum Criteria for Doctor- Patient Communication, Beginningvs. End of Internship: Urinary Tract Infection (n = 25)
Counseling Criterion
Beginning
End
Criterion
Beginning
End
Stress reduction Alcohol use Depression Use of drugs Recurrences
10 (43%) 2 (9%) 1 (4%) 0 3 (13%)
9 (39%) 3 (13%) 7 (30%) 2 (9%) 4 (17%)
Introduction/identification Summarizing the problem Explaining investigations Explaining diagnosis Explaining pathogenesisof problem Explaining management plan Ensuring patient understanding
21 (84%) 7 (28%) 19 (76%)
24 (96%) 6 (24%) 23 (92%)
I 5 (60%)
18 (72%).
10 (40%) 20 (80%) 4 (16%)
16 (64%) 24 (96%) 4 (16%)
TABLE 7
Numbers (%) of Interns Who Satisfied the Minimum Counseling Criteria for Preventive Care, Beginning vs. End of Internship: Bronchitis (n = 25) Counseling Criterion
Beginning
End
Stopping smoking Ways of stopping smoking Recurrences
11 (44%) 8 (32%) S (20%)
18 (72%) 15 (60%) 6 (24%)
Two areas of persistent weakness in communication skills were in summarizing the problem and ensuring patient understanding. Deficits in these related behaviors suggest that interns do not know h o w to close the communication loop; they provide items of information for their patients but they do not "tie u p " their consultations and make sure that their patients have overviews of the interactions that have taken place. This study has identified a n u m b e r of unfulfilled educational needs among interns. Meeting these needs will involve educational and structural changes in the internship and in the way that teaching hospitals provide ambulatory care services:
TABLE 9
Numbers (%) of Interns Who Satisfied the Minimum Criteria for Doctor-Patient Communication, Beginning vs. End of Internship: Tension Headache(n = 23) Criterion
Beginning
End
Summarizing the problem Explaining investigations Explaining pathogenesisof problem Explaining management plan Ensuring patient understanding
6 (26%) I0 (43%) 7 (30%)
7 (30%) 17 (74%) 14 (61%)
I 0 (43%)
17 (74%)
5 (22%)
10 (43%)
TABLE 10
Numbers (%) of Interns Who Satisfied the Minimum Criteria for Doctor-Patient Communication, Beginning vs. End of Internship: Bronchitis (n = 25) Criterion
Beginning
End
Summarizing the problem Explaining investigations Explaining diagnosis Explaining pathogenesisof problem Ensuring patient understanding
5 (20%) 18 (72%) 4 (I 6%) 5 (20%) 6 (24%)
5 (20%) 24 (96%) I Z (48%) 25 (100%) 9 (36%)
6Z
~Ol'dOD e t a J., PERFORMANCEAT BEGINNING AND END OF ]NTERNSHIP
ambulatory patients with common, preventable illnesses as a basis for teaching w o u l d assist interns to obtain a more balanced perspective on the type of care they are likely to be providing in the future. Education in this area w o u l d necessitate a better understanding of epidemiology, health risk detection, and prevention.
3. Practical experience of ambulatory care under appropriate supervision. The performance of the interns at the beginning of the year suggests that medical schools do not yet provide adequate teaching in the specific skills required for early interventions to promote health and prevent illness. However, the results of this study suggest that interns are not encouraged to take advantage of the opportunity to gain experience in preventive care strategies after graduation even though the patients presenting in the casualty department provide ample opportunity for this experience. The absence of primary care practitioners from teaching hospitals in Australia limits the e x p e r i e n c e of both medical students and graduates, t ~, t4 There are two ways of redressing the balance between education for generalist practice and education for specialty practice: 1) to introduce more generalists into clinical and teaching positions within the primary care facilities of the hospital and 2) to provide medical students and interns with community-based experience outside the hospital, in c o m m u n i t y health clinics or in private practices. Ideally, a mixture of these experiences should occur. In 1988, a major review of medical education in Australia made a strong case for this shift in emphasis. H To date, there have b e e n few signs that the recommendations in this report will be implemented.
CONCLUSIONS The findings of this study suggest that the internship should be restructured to more adequately teach
the skills required for primary care. Hospitals should be providing planned curricula that address interns' educational needs as well as the needs of patients with the c o m m o n preventable diseases typically seen in ambulatory practice.
REFERENCES 1. General Medical Council of the United Kingdom. Education committee recommendations on basic medical education, London: Greater Medical Council, 1980. 2. Larkins R. Aims of the intern year. In: Review of the intern year in Victoria. Melbourne: Victorian Medical Postgraduate Foundation, 1986. 3. Gordon JJ, Saunders NA, Sanson-Fisher RW. Evaluating interns' performance using simulated patients in a casualty department. Med J Austr. 1989;151:18-21. 4. Stillman PL, Swanson DB, Smee S, et al. Assessingclinical skills of residents with standardized patients. Ann Intern Med. 1986; 105:762-71. 5. Stillman PL, Swanson DB, Regan MB, et al. Assessment of clinical skills of residents utilizing standardized patients: a followup study and recommendations for application. Ann Intern Med. 1991;114:393-401. 6. Rethans jOE, van Boven CPA. Simulated patients in general practice: a different look at the consultation. Br Med J. 1987; 294:809-12. 7. Norman GR, Neufeld VR, Walsh A, Woodward CA, McConvey GA. Measuring physicians' performances by using simulated patients. J Med Educ. 1985;60:925-34. 8. O'HaganJJ, Davies LJ, Pears RK. The use of simulated patients in the assessment of actual clinical performance in general practice. N Z MedJ. 1986;99:948-51. 9. Gordon J, Sanson-Fisher R, Saunders N. Identification of simulated patients by interns in a casualty setting. Med Educ. 1988;22:533-8. 10. Barrows HS. Simulated patients (programmed patients). Springfield, IL: Charles C. Thomas, 1971. 11. Doherty R. Report of the Committee of Inquiry into Medical Education and Medical Workforce. Australian medical education and workforce into the 21st century. Canberra: Australian Government Printing Service, 1988. 12. Rotem A, Craig P, Cox KR, Ewan CE. The organization and management of medical education in Australia. Health Policy Educ. 1981;2:177-206. 13. White KL, Williams TF, Greenberg BG. The ecology of medical care. N EngJ Med. 1961;265:885-92. 14. Metcalfe D. The mismatch between undergraduate education and the medical task. In: Pendelton D, Hasler J (eds). Doctorpatient communication. London: Academic Press, 1983.
departmen~ Results: Little improvement over the intern y e a r in technical competence o r preventive care was o b s e r ~ d , even though initial levels o f compliance with criteria were quite low f o r some items. Greater improvement was apparent in the area o f communication skills. Conclusions: The results suggest that the internship should be restructured to more adequately teach the skills required f o r p r i m a r y care. Key words: internships; simulated patients; preventive c a r e ; p h y s i c i a n - p a t i e n t coiiii~iunication,.performance assessment. J GEN INiratN MED 1992;7:57-62.
IN NEW SOUTH WALES, Australia, the attainment of an undergraduate degree in m e d i c i n e is followed b y a c o m p u l s o r y year of internship. Internship represents the c o m p l e t i o n of basic medical education; its p u r p o s e is to afford the recent graduate balanced clinical experience with increasing responsibility for the care of patients, t Tasks of the intern year are intended to dev e l o p cognitive, procedural, interpersonal, counseling, and administrative skills. 2 By the end of the year, interns have s o m e e x p e r i e n c e in medicine, surgery, and a m b u l a t o r y care. They are then registered for indep e n d e n t practice and eligible for specialty training. In a previous study involving 56 interns w h o saw covert simulated patients in a casualty setting, substantial deficiencies in the interns' preventive care and c o m m u n i c a t i o n skills w e r e identified. 3 Suggested explanations for these findings w e r e that the interns had not b e e n taught these skills as undergraduates, that Received from the Royal Australian College of General Practitioners, Macquarie Hospital, North Ryde (JJG), and the Faculty of Medicine, The University of Newcastle (NAS, DH, RWS-F), New South Wales, Australia. Supported by grants from the Australian Institute of Health and the NSW State Cancer Council. Address correspondence and reprint requests to Dr. SansonFisher: Behavioural Science in Relation to Medicine, Faculty of Medicine, University of Newcastle, New South Wales, 2308, Australia.
these skills w e r e suppressed in the first w e e k s of the internship w h e n the study took place, that the w o r k e n v i r o n m e n t discouraged these aspects of patient care, and that the interns themselves c o u l d not see the value o f these aspects o f care. If the intern year is fulfilling its p u r p o s e of completing the process of basic medical education, then measurable i m p r o v e m e n t should o c c u r b e t w e e n the beginning and the end of the year. Despite the crucial role of the intern year in the transition from undergraduate status to i n d e p e n d e n t practice, w e are unaware of studies that have a t t e m p t e d to d o c u m e n t the differential i m p a c t of the internship on p e r f o r m a n c e s of technical, preventive, and c o m m u n i c a t i o n skills. Stillman et al. 4, 5 have d e v e l o p e d the t e c h n i q u e of using overt standardized patients to assess the clinical skills of residents in internal medicine. T h e y have demonstrated the validity and reliability of this t e c h n i q u e and have suggested that the m e t h o d also provides opportunities to test skills, such as c o m m u n i c a t i o n skills, not easily assessed by other means w h i l e controlling for factors such as case difficulty. Simulated patients have also demonstrated their utility in the a m b u l a t o r y care setting w h e n used covertly. 68 The aims of the present study w e r e therefore to measure changes b e t w e e n the beginning and the end of internship in the interns' technical, preventive, and c o m m u n i c a t i o n skills, using covert simulated patients.
SETTING, SUBJECTS, AND METHODS The setting for the study was the Casualty Departm e n t of the Royal Newcastle Hospital, a 6 5 0 - b e d general teaching hospital located in the inner city. The region that it serves has a p o p u l a t i o n of a p p r o x i m a t e l y 350,000. The hospital is s u r r o u n d e d b y low-cost rental a c c o m m o d a t i o n , and m a n y of the patients w h o use the hospital as a source of p r i m a r y care live in the immediate area. It is the policy of the casualty d e p a r t m e n t to provide treatment for a m b u l a t o r y patients, but to encourage t h e m to consult a p r i m a r y care physician for ongoing management. All interns starting at the hospital at the beginning of year w e r e asked to participate in a study that involved the videotaping of consultations with simulated patients. The interns did not k n o w h o w m a n y patients w e r e to be presented but they w e r e aware that videotaping w o u l d o c c u r within a two-day period w h e n they $7
$8
Gordon etaL, PERFORMANCEAT BEGINNINGAND END OF INTERNSHIP
were specially rotated to the casualty department in the first and last ten weeks of their internships. Despite these cues, the interns were not able to accurately identify the simulators from among genuine patients. They also reported that the study generally had little effect on their usual consulting behaviors.9 Thirty-five of 41 interns at the hospital were available during the study period; all 35 consented to participate at the beginning of the year. Thirty-two (91%) consented to repeat the study at the end of the year. Four of these interns were unable to participate because of transfers to other hospitals, leaving 28 subjects for w h o m beginning- and end-of-year comparisons c o u l d be made. There was no significant difference in the ages, gender distributions, or universities of graduation between those w h o were available and consented to repeat the study and those w h o were not. At both the beginning and the end of the intern year, three consultations with simulated patients were videotaped during the two successive days of the casualty attachment. Each consulting cubicle in the casualty department had an unobtrusive camera placed close to the ceiling, and directed at the desk at w h i c h the intern and the patient sat during the interview. Simulated patients were recruited from the Commonwealth Employment Service and from the local Old Aged Pensioners' Association. Those w h o were selected had clinical histories that resembled most closely the patients w h o m they were to portray. Training followed the m e t h o d that was described by Barrows, lo with three one-hour training sessions. The patient's o w n medical and social history was used as m u c h as possible. Each simulated patient pretended to have a clinical p r o b l e m that was selected on the basis of being comm o n and potentially preventable and that required neither referral nor invasive diagnostic or management procedures. The patients were instructed to provide each intern with clearly defined information about their clinical problems. Their reliability in presenting this information was assessed by a research assistant during the videorecording, and later from a review of the videotapes. The interns were presented with the same three simulations at the end of the year, in order to standardize the evaluation. The simulators portraying the patients were changed at the end of the year to avoid the possibility of recognition. The n e w simulators were trained using the same m e t h o d as had been used for the first g r o u p at the beginning o f the year. Abbreviated versions of the patients' histories illustrate the opportunities that were provided for interns to demonstrate clinical, interpersonal, and preventive skills:
Case 1: u r i n a r y t r a c t i n f e c t i o n ( U T I ) i n a 2 2 - y e a r - o l d w o m a n . This patient presented with a past history of
cystitis. Her symptoms of dysuria and frequency of mic-
turition commenced a few days after the return of her boyfriend, who had been away for several months. She was unemployed, smoked 15 cigarettes each day, and drank to the point of feeling intoxicated once or twice each week. She took paracetamol for headaches and menstrual pains. She had no regular family doctor. Case 2: h e a d a c h e in a 1 9 - y e a r - o l d m a n . This patient presented with symptoms that were typical of tension headache that had lasted for the past few months. He was unemployed and lived in the inner city. He smoked 20 or more cigarettes each day, used marijuana regularly, and drank alcohol as often as he could afford to - - usually a few times per week. He was having problems in his relationship with his girlfriend and appeared lonely and depressed. He had no regular family doctor. Case 3: b r o n c h i t i s i n a 6 7 - y e a r . o l d m a n . This patient presented with chronic airflow limitation as a result of smoking in excess of 20 cigarettes per day for 50 years. He presented with increasing cough and breathlessness and had lost about 4 kg in weight in the last six months. He admitted to drinking at least 40 g of alcohol per day. He lived alone and had no family doctor.
Criteria for Care An expert panel was c o n v e n e d to develop explicit criteria for the management of each patient by an intern in a casualty department. The six-member panel comprised two family physicians, two internists, and two housestaff m e m b e r s - - one m e m b e r in his second year and one m e m b e r in his fourth year of training. Panel members were given full patient profiles on w h i c h to base their decisions. Each began by modifying a criterion list that was c o m p i l e d by us from standard sources; then they met to review each other's responses. Further individual modifications were made after the meeting and final decisions then were reached by consensus at a s e c o n d meeting. The panel divided the criteria into essential and nonessential items. Essential features of the consultation were used to establish a minimal level of performance. Three groups of behaviors were considered by the panel. The first was technical care related to historytaking, investigations, and management. The second was preventive care related to the eliciting of relevant items in the history or to the provision of specific advice c o n c e r n i n g the patient's presenting problems: for example, antismoking advice for the patient with chronic airway obstruction. Opportunities for providing preventive advice that were not related to the problem (for example, advice about smoking to the w o m a n with a UTI) also were noted. Finally, c o m m u n i c a t i o n skills were based on behaviors that have been shown to influence patient satisfaction and c o m p l i a n c e with treatment. Seven behaviors considered essential for effective d o c t o r - p a t i e n t c o m m u n i c a t i o n were identified.
JOURNALOFGENERALINTERNALMEDICINE,Volume 7 (January/February), 1992
The videotapes were scored by two trained raters, both of w h o m were nurse educators. The explicit criteria that they used for scoring included examples of statements that the interns might use. Each behavior was marked present or absent. To check interrater reliability, each rater scored a one-in-four random sample of the other rater's videotapes. The percentage of consultations in which the observers agreed u p o n w h e t h e r the behavior had o c c u r r e d was calculated for each behavior.
RESULTS Videotapes for both the beginning and the end of the year were obtained for 25 interns consulting with the simulators with UTI, 23 interns consulting with the simulators with headache, and 25 interns consulting with the simulators with bronchitis.
Reliability of Patients' Presentations The reliability of each simulated patient's presentation was checked on every occasion by the research assistant, w h o viewed the videorecording while the consultation was in progress. The check on reliability of performance consisted of a check that the simulated patient portrayed the essential features of each case. The reliability o f performance was 100%.
Reliability of Videotape Coding All items were c o d e d with acceptable reliability. Agreement b e t w e e n the two observers was at least 80% for all 83 items rated. One hundred percent agreement was achieved for 33 items and 90% agreement or better for 72 items.
Competence in Technical Care Table 1 summarizes the criteria determined by the expert panel to be essential for each of the three patients' problems. Tables 2, 3, and 4 highlight those technical aspects of care considered essential by the expert panel but p o o r l y adhered to by the interns. Items in w h i c h the interns performed well at both the beginning and the end of the intern year have been omitted for clarity of presentation. Generally, the interns demonstrated good ability in obtaining an adequate history of each patient's current p r o b l e m during both testing periods. Overall, the interns' compliance with the essential criteria for technical care did not change significantly for any of the three simulated patient problems. Isolated changes occurred in each problem. For the patients with UTIs, all of the interns ordered urinalysis and cultures at the end of the year, c o m p a r e d with only 64% at the beginning. More interns included an inquiry about the sleep pattern of the patient with tension headache, suggesting that the clinical possibility of depression was being explored, and more provided appropriate treatment of this patient at the end of the year than at the beginning. The e x p e r t panel accepted as appropriate treatment in this case an explanation of the nature of the headache along with suggestions for stress management a n d / o r management of the patient's depressive symptoms. The prescribing of narcotic analgesics was considered inappropriate treatment. By contrast, fewer interns made adequate arrangements for the follow-up care of the patient with bronchitis at the end of the year. Significant omissions o c c u r r e d at both the beginning and the end of the internships in taking histories concerning current medications from the patients with UTIs and tension headaches, obtaining adequate past
TABLE 1 Summary of Expert Panel's Essential Criteria for TechnicalCompetence in Interns' Management of Three Problems Urinary Tract Infection History
Main complaint Duration Pain
Past history Medications
Tension Headache
Bronchitis
Main complaint Duration Headache length Location Quality Context Migraine history Sleep pattern Past history Medications
Main complaint Duration Cough Sputum Breathlessness Weight loss Cardiovascular symptoms Last chest x-ray Past history Medications
Diagnostic tests
Urinalysis Culture and sensitivity
None should be done
Spirometry
Treatment
Appropriate antibiotic
Lifestyle counselingby self or referral
A range of symptomatic treatment options approved by the panel (e.g., alteration of salbutamol dose) Further investigation is mandatory
Follow-up
59
60
Gordon eta/.. PERFORMANCEAT BEGINNINGAND END OF INTERNSHIP
TABLE 2 Numbers (%) of InternsWho Satisfiedthe Minimum Criteria for TechnicalCare, Beginningvs. End of Internship: Urinary Tract Infection (n = 25) Criterion
Beginning
End
Past history Medications Investigations Follow-up
11 (44%) 6 (24%) 16 (64%) 19 (76%)
15 (60%) 7 (28%) 25 (100%) 21 (84%)
TABLE 3 Numbers (%) of Interns Who Satisfiedthe Minimum Criteria for TechnicalCare, Beginningvs. End of Internship: TensionHeadache (n = 23) Criterion
Beginning
End
Headachelength Headachecontext Sleep pattern Past history Medications Investigations Treatment Follow-up
11 (48%) 15 (65%) 7 (30%) 7 (30%) 8 (35%) 13 (57%) 10 (43%) 10 (43%)
18 (78%) 20 (87%) 15 (65%) 9 (39%) 6 (26%) 18 (78%) 17 (74%) 15 (65%)
TABLE 4 Numbers (%) of InternsWho Satisfiedthe Minimum Criteria for TechnicalCare, Beginningvs. End of Internship:Bronchitis(n = 25) Criterion
Beginning
End
Breathlessness Weight loss Cardiovascularsymptoms Last chestx-ray Past history Medications Investigations Follow-up
17 (68%) 1B (60%) 19 (72%) 12 (48%) 17 (68%) 21 (84%) 4 (16%) 24 (96%)
17 (68%) 14 (56%) 13 (52%) 14 (56%) 13 (52%) 18 (72%) 2 (8%) 17 (68%)
histories in all three cases, and measuring respiratory function (spirometry) in the patient w i t h chronic respiratory disease. Competence in Preventive Care Performance for preventive advice for the patients w i t h UTIs did not change (Table 5), suggesting that the interns did not take u p s o m e o f the practical preventive strategies identified by the e x p e r t panel. In particular, the interns failed to give the patient advice on h o w to deal w i t h a r e c u r r e n c e of this p r o b l e m . There was an increase in the n u m b e r of interns w h o dealt w i t h the p r o b l e m of depression as it presented in the y o u n g man with tension headache (Table 6). Only one intern in 23 had addressed this p r o b l e m at the beginning of the year, but seven interns a t t e m p t e d to offer h e l p in this area in the end-of-year study. Nevertheless, 70% of interns at the end of the year made no
a t t e m p t to deal w i t h an issue that was the key to the patient's somatic symptom. More interns p r o v i d e d advice a b o u t smoking for the patient with bronchitis at the end of the year, and m o r e interns suggested ways of stopping smoking (Table 7). Nevertheless, given the central role of smoking in the pathogenesis of this patient's clinical problem, it was disappointing to see that antismoking advice did not a p p e a r in every consultation. Overall, the n u m b e r s of interns satisfying the essential criteria w e r e low. Fewer than 30% satisfied any of the preventive care criteria for UTI and f e w e r than 40% satisfied any of the preventive care criteria for tension headache. The only simulated patient for w h o m more than half of the interns a p p r o a c h e d a satisfactory standard was the patient with chronic airflow limitation, for w h o m smoking was a central risk factor. However, fewer than 25% of the interns p r o v i d e d adequate advice in relation to c o p i n g with recurrences in the end-of-year consultations. Competence in Communication Skills While changes in c o m m u n i c a t i o n skills w e r e few, there was a trend toward greater c o m p l i a n c e with the criteria in the end-of-year study (Tables 8, 9, and 10). The most notable i m p r o v e m e n t was in explaining the pathogeneses of the patients' problems. Whereas only 40%, 30%, and 20% of the interns p r o v i d e d explanations at the beginning of the year, by the end of the year these percentages had increased to 64%, 61%, and 100%, respectively. Interns w e r e also m o r e likely to explain the plan of m a n a g e m e n t to the simulator with UTI. They w e r e m o r e likely to provide the simulator w h o had tension headache w i t h an explanation for their decisions not to order unnecessary investigations and to explain treatment decisions. There w e r e two areas of persistent weakness: s u m m a r i z i n g the p r o b l e m and ensuring patient understanding.
DISCUSSION The study of interns' technical, interpersonal, and preventive care skills at the beginning of the intern year identified a n u m b e r of deficiencies that m a y have reflected failings in their undergraduate education. Because the p e r f o r m a n c e assessment c o v e r e d c o m m o n a m b u l a t o r y care conditions, and because the essential criteria carefully avoided esoteric items, these deficiencies w e r e important. A year of internship offered s c o p e for the interns to i m p r o v e their skills. This study found that technical proficiency did not change significantly over the course of the year. While proficiency was s h o w n in s o m e areas at b o t h the beginning and the end of internship, there w e r e persistent deficits in others: for e x a m p l e , in obtaining an adequate past history and information a b o u t m e d i c a t i o n
JOURNALOFGENERALINTERNALMEDICINE,Volume 7 (January/February), 1992
use. Arrangements to ensure follow-up o f all three patients were poor. This is perhaps not surprising in an environment that discourages continuity of care, but it is p o o r preparation for the responsibilities of independent medical practice. Preventive care, w h i c h had the greatest scope for improvement, was persistently poor. The only case in w h i c h it approached adequacy was the case of the simulator with bronchitis, where the cause of the chronic respiratory impairment was obvious and the impact of continued smoking made successful treatment difficult. TABLE 5
Numbers (%) of Interns Who Satisfied the Minimum Counseling Criteria for Preventive Care, Beginningvs. End of Internship: Urinary Tract Infection (n = 25) Counseling Criterion
Beginning
Fluid intake Voiding frequency Sexual behavior Managing recurrences
4 (16%) 5 (20%) B (20%) 4 (16%)
End 6 7 6 6
(24%) (28%) (24%) (24%)
61
1. A c o m m i t m e n t by h o s p i t a l s to i n t e r n educat i o n as w e l l as service. One of the major problems with
internship training in Australia is the dominance of hospitals' service needs over their obligation to provide training. 11 Few intern training programs offer planned curricula that address the interns' needs to acquire technical, preventive, or interpersonal care skills.12 2. A r e o r i e n t a t i o n to the h e a l t h needs o f the pat i e n t s r e c e i v i n g care. Thirty years ago, White et al. ~3
estimated that the patients w h o receive treatment in teaching hospitals represent only a tiny p r o p o r t i o n of those w h o are ill. Modern teaching hospitals have moved even further in the direction of short-stay admissions for investigative and therapeutic procedures or admissions to intensive care units for the critically ill. They are therefore less and less representative of the real health needs of the communities that they serve. By contrast, patients presenting to hospital casualty departments, e m e r g e n c y rooms, and primary care clinics have a m u c h wider variety of health care needs. Using
TABLE 6
TABLE 8
Numbers (%) of Interns Who Satisfied the Minimum Counseling Criteria for Preventive Care, Beginningvs. End of Internship: Tension Headache (n = 23)
Numbers (%) of Interns Who Satisfied the Minimum Criteria for Doctor- Patient Communication, Beginningvs. End of Internship: Urinary Tract Infection (n = 25)
Counseling Criterion
Beginning
End
Criterion
Beginning
End
Stress reduction Alcohol use Depression Use of drugs Recurrences
10 (43%) 2 (9%) 1 (4%) 0 3 (13%)
9 (39%) 3 (13%) 7 (30%) 2 (9%) 4 (17%)
Introduction/identification Summarizing the problem Explaining investigations Explaining diagnosis Explaining pathogenesisof problem Explaining management plan Ensuring patient understanding
21 (84%) 7 (28%) 19 (76%)
24 (96%) 6 (24%) 23 (92%)
I 5 (60%)
18 (72%).
10 (40%) 20 (80%) 4 (16%)
16 (64%) 24 (96%) 4 (16%)
TABLE 7
Numbers (%) of Interns Who Satisfied the Minimum Counseling Criteria for Preventive Care, Beginning vs. End of Internship: Bronchitis (n = 25) Counseling Criterion
Beginning
End
Stopping smoking Ways of stopping smoking Recurrences
11 (44%) 8 (32%) S (20%)
18 (72%) 15 (60%) 6 (24%)
Two areas of persistent weakness in communication skills were in summarizing the problem and ensuring patient understanding. Deficits in these related behaviors suggest that interns do not know h o w to close the communication loop; they provide items of information for their patients but they do not "tie u p " their consultations and make sure that their patients have overviews of the interactions that have taken place. This study has identified a n u m b e r of unfulfilled educational needs among interns. Meeting these needs will involve educational and structural changes in the internship and in the way that teaching hospitals provide ambulatory care services:
TABLE 9
Numbers (%) of Interns Who Satisfied the Minimum Criteria for Doctor-Patient Communication, Beginning vs. End of Internship: Tension Headache(n = 23) Criterion
Beginning
End
Summarizing the problem Explaining investigations Explaining pathogenesisof problem Explaining management plan Ensuring patient understanding
6 (26%) I0 (43%) 7 (30%)
7 (30%) 17 (74%) 14 (61%)
I 0 (43%)
17 (74%)
5 (22%)
10 (43%)
TABLE 10
Numbers (%) of Interns Who Satisfied the Minimum Criteria for Doctor-Patient Communication, Beginning vs. End of Internship: Bronchitis (n = 25) Criterion
Beginning
End
Summarizing the problem Explaining investigations Explaining diagnosis Explaining pathogenesisof problem Ensuring patient understanding
5 (20%) 18 (72%) 4 (I 6%) 5 (20%) 6 (24%)
5 (20%) 24 (96%) I Z (48%) 25 (100%) 9 (36%)
6Z
~Ol'dOD e t a J., PERFORMANCEAT BEGINNING AND END OF ]NTERNSHIP
ambulatory patients with common, preventable illnesses as a basis for teaching w o u l d assist interns to obtain a more balanced perspective on the type of care they are likely to be providing in the future. Education in this area w o u l d necessitate a better understanding of epidemiology, health risk detection, and prevention.
3. Practical experience of ambulatory care under appropriate supervision. The performance of the interns at the beginning of the year suggests that medical schools do not yet provide adequate teaching in the specific skills required for early interventions to promote health and prevent illness. However, the results of this study suggest that interns are not encouraged to take advantage of the opportunity to gain experience in preventive care strategies after graduation even though the patients presenting in the casualty department provide ample opportunity for this experience. The absence of primary care practitioners from teaching hospitals in Australia limits the e x p e r i e n c e of both medical students and graduates, t ~, t4 There are two ways of redressing the balance between education for generalist practice and education for specialty practice: 1) to introduce more generalists into clinical and teaching positions within the primary care facilities of the hospital and 2) to provide medical students and interns with community-based experience outside the hospital, in c o m m u n i t y health clinics or in private practices. Ideally, a mixture of these experiences should occur. In 1988, a major review of medical education in Australia made a strong case for this shift in emphasis. H To date, there have b e e n few signs that the recommendations in this report will be implemented.
CONCLUSIONS The findings of this study suggest that the internship should be restructured to more adequately teach
the skills required for primary care. Hospitals should be providing planned curricula that address interns' educational needs as well as the needs of patients with the c o m m o n preventable diseases typically seen in ambulatory practice.
REFERENCES 1. General Medical Council of the United Kingdom. Education committee recommendations on basic medical education, London: Greater Medical Council, 1980. 2. Larkins R. Aims of the intern year. In: Review of the intern year in Victoria. Melbourne: Victorian Medical Postgraduate Foundation, 1986. 3. Gordon JJ, Saunders NA, Sanson-Fisher RW. Evaluating interns' performance using simulated patients in a casualty department. Med J Austr. 1989;151:18-21. 4. Stillman PL, Swanson DB, Smee S, et al. Assessingclinical skills of residents with standardized patients. Ann Intern Med. 1986; 105:762-71. 5. Stillman PL, Swanson DB, Regan MB, et al. Assessment of clinical skills of residents utilizing standardized patients: a followup study and recommendations for application. Ann Intern Med. 1991;114:393-401. 6. Rethans jOE, van Boven CPA. Simulated patients in general practice: a different look at the consultation. Br Med J. 1987; 294:809-12. 7. Norman GR, Neufeld VR, Walsh A, Woodward CA, McConvey GA. Measuring physicians' performances by using simulated patients. J Med Educ. 1985;60:925-34. 8. O'HaganJJ, Davies LJ, Pears RK. The use of simulated patients in the assessment of actual clinical performance in general practice. N Z MedJ. 1986;99:948-51. 9. Gordon J, Sanson-Fisher R, Saunders N. Identification of simulated patients by interns in a casualty setting. Med Educ. 1988;22:533-8. 10. Barrows HS. Simulated patients (programmed patients). Springfield, IL: Charles C. Thomas, 1971. 11. Doherty R. Report of the Committee of Inquiry into Medical Education and Medical Workforce. Australian medical education and workforce into the 21st century. Canberra: Australian Government Printing Service, 1988. 12. Rotem A, Craig P, Cox KR, Ewan CE. The organization and management of medical education in Australia. Health Policy Educ. 1981;2:177-206. 13. White KL, Williams TF, Greenberg BG. The ecology of medical care. N EngJ Med. 1961;265:885-92. 14. Metcalfe D. The mismatch between undergraduate education and the medical task. In: Pendelton D, Hasler J (eds). Doctorpatient communication. London: Academic Press, 1983.
