Programme Evaluation :An Anthropological Look at the Clerkship

Med Teach Downloaded from informahealthcare.com by University of Sydney on 01/02/15 For personal use only.


Members of the Departments of Medical Education and Medicine sought to design a study that would produce a n accurate, detailed description of the nineweek medicine clerkship. An anthropological research approach was adopted to develop a richly detailed, composite picture of the clerkship experience from the student’s perspective. T h e results indicate the value of this approach under selected conditions, as discussed i n the article. While it is impossible to prove a direct causal relationship, several problems identified during the study have since been alleviated. Interested in learning how well their perceptions matched those of their students, faculty members of the Department of Medicine at Southern Illinois University School of Medicine enlisted the assistance of the Department of Medical Education to plan and conduct an in-depth study of the nine-week medicine clerkship. That was in 1976, and since then a number of changes have transpired with regard to the clerkship, some of which might be partially attributed to the results of that study. Members of the clinical faculty were primarily concerned with learning how students spent their time and what learning resources they used while participating in the clerkship. Additionally, it was hoped that such a project would provide information useful for the evaluation and subsequent improvement of the organization and content of the clerkship. A team of four from the Department of Medical Education met with the Director of the Medicine Clerkship to design a study which would produce an accurate, detailed description of the clerkship experience from the student’s perspective. Following lengthy discussions of how best to study the clinical life of a medicine clerk, the group decided to adopt the anthropological research approach that has also been termed ‘responsive evaluGerry R. Schemerhorn. MA, is Instructional Development Specialist, Department of Medical Education, Southern Illinois University School of Medicine; Reed G Williams, PH.D,is Professor. Department of Medical Education: Robert H . Miller. MD, FACP. Glynn Brunswick Hospital, Brunswick, Georgia 31520. is former Assoriate Professor, Departmcnt of Medicine; Lynn M . Kienzler. MA, and Susan K . Wilson, R N . BS. are Instructional Development Specialists. Department of Medical Education, at the Southern Illinois University School of Medicine, Springfield. Illinois 62708.

Medical Teacher V o l 2 No 4 1980

ation’ (Stake 1976), or ‘illuminative evaluation’ (Parlett and Hamilton 1972). Like anthropologists going into the field, the medical education team would follow and observe individual medicine clerks as they progressed through the cardiology and general medicine rotations that constituted the medicine clerkship. Data would be collected regarding activities in which the clerks participated, patient contact experienced, level of responsibility assumed, and learning resources utilized. The main research goal was to provide faculty mem bers of the Department of Medicine with a thorough understanding of the complex combination of activities, interactions, responsibilities, expectations, and attitudes of their students. This objective seemed well-suited to the responsive or anthropological model as described (Stake 1976): “The responsive approach is an attempt to respond to the natural ways in which people assimilate information and arrive at understanding. Direct personal experience is an efficient, comprehensive, and satisfying way of creating understanding, but a way not usually available to our audiences. The best substitute for direct experience probably is vicarious experience -increasingly better when the evaluator uses attending and conceptualizing styles similar to those which members of the audience use. Such styles are not likely to be those of the specialist in measurement or theoretically minded social scientist. Vicarious experience often will be conceptualized in terms of persons, places, and events. ’’ Method Initially, the observer team devised a checklist which was to expedite observer note-taking. While the actual study was still in the planning phase, a pilot probe was conducted to familiarize the observers with clerkship routines and to test the checklist. Each observer followed one student for one day. Independently, each observer soon realized that the checklist was far too rigid to reflect adequately the interwoven complex of planned and u n planned activities and personal interactions that constituted the total learning experience. As an alternative, 171

Med Teach Downloaded from informahealthcare.com by University of Sydney on 01/02/15 For personal use only.

each observer recorded copious notes regarding the student’s actions and interactions. The notes formed detailed time logs showing how long the student was engaged in each activity and included subjective observations of the student’s attitude. The observers also performed on-the-spot interviews with the students concerning their opinions of the clerkship in general, the learning resources and experiences found most useful, and relationships with faculty and teaching residents. The observer team decided to maintain this open-ended form of data collection in the actual study thinking that such a flexible system would provide answers to questions of immediate concern and questions which might arise later. Four of the 13 students rotating through medicine at the time of the study were randomly selected as a sample for close observation. The rapport generated by observers working individually with students throughout the nine weeks enabled the students to act more naturally and answer interview questions more frankly. T o minimize the influence of personal rater bias on the results, a system of cross-checks was built in whereby each observer followed and reported on one of the students normally observed by another observer. The students selected were willing to be observed and cooperated fully throughout the study. They were assured that their anonymity would be protected and that the data gathered would have no effect on their own evaluations. The remaining eight students were followed less closely and were informally interviewed throughout the clerkship: one student refused to be observed. Because many clerkship activities varied according to the day of the week, each observer followed a student on at least six different weekdays. Observation days totalled 32 out of the 54 days of the clerkship and were distributed throughout the nine-week period. Three observers followed students from hospital arrival to departure for approximately two days every three weeks. Another observer performed a more intensive observation by following his assigned student for five consecutive days. Each observer also went ‘on call’ with a student at least once during the clerkship and stayed overnight at the hospital. Throughout the clerkship, informal interviews with the students and their assigned residents were conducted, focusing on opinions about clerkship activities and experiences. Such interviews were often conducted while travelling between hospitals or enjoying a meal. As is typical for responsive evaluation projects, issues tended to evolve as the evaluators became more cognizant of the programme’s strengths and weaknesses. Further exploration led to additional issues, and identification of new issues provided new directions to explore. The inherent flexibility of the responsive approach is one of its strengths (Stake, 1976). The raw data resulting from more than 250 hours of direct observation time was initially categorized as patient contact, interpersonal relations, learning materials used, participation in activities, clinical procedures performed, and comments about the clerkship. Categorized data were further digested to offer summary 172

Figure 1. The average student’s day: perceztage of time spent in various activities. information for the main clerkship issues. The comprehensive nature of the data derived presumably would allow future reference to explore additional issues. Findings A summary report of the findings was presented to all faculty members of the Department of Medicine. Limited copies of the full report were made available to staff for reference purposes. Because the findings of this study were specific to the Southern Illinois University medicine clerkship, no claim could be made for the generalizability of the results. Nevertheless, they might be of interest to medical educators and clinicians at other institutions. A selection of abbreviated findings is provided both as points of interest and as illustrations of the type of information the anthropological evaluation approach yields.

The Average Day The student’s average day in the hospital lasted seven hours and was spent as shown in Figure 1.

Night Call Five call nights were observed. Resulting reports indicated that use of time while on call varied considerably depending upon the number of newly admitted patients, the student’s relationship with the assigned resident, and the student’s own level of assertiveness regarding patient care.

Opportunitiesf o r Clinical Work During the 32 observation days, the observation team witnessed 15 physical examinations: of these, students

Medical Teacher Vol2 No 4 I980

Med Teach Downloaded from informahealthcare.com by University of Sydney on 01/02/15 For personal use only.

performed seven independently. Most other physical examination procedures were performed by a resident or attending physician with the student observing. Students performed 13 diagnostic procedures and observed a resident or attending physician perform 16 procedures. Regarding management, on five occasions a student developed and ordered a patient management plan. In six other cases, the student participated in preparing and ordering the management plan. Finally, in 21 cases, the observed student discussed a plan for management with the assigned resident, but did not participate in ordering it. Two students indicated dissatisfaction with the number of opportunities to practise clinical procedures during the clerkship. Each stated that the assigned resident performed most of the clinical procedures with the student observing only.

Observation suggested that assertive students received more learning opportunities. For example, only one student performed any bone marrow aspirations during the study, and this student performed two. In both cases, the student had requested permission to perform the procedure. Also, assertive students appeared to present more patients during rounds. Certain students took more initiative in creating learning opportunities for themselves.

municating accurate, useful information to those responsible for deciding the future of the programme, one might wish to measure how well the anthropological approach fulfils this intention. In a subsequent study (Schermerhorn and Williams 1979), reaction to the anthropological study report prepared for members of the Department of Medicine was measured against audience reaction to a report based on a ‘standard’ questionnaire administered 10 the students at the completion of the same medicine clerkship. Overall, respondents preferred the anthropological evaluation approach to the preordinate approach. Respondents reported that the anthropological evaluation gave a more accurate impression of the clerkship experience, offered more valuable information, and was more interesting. The two reports were rated as equally efficient and objective in presenting information. Certainly, one of the most important indications of the value of an evaluation is the ultimate programme improvements it inspires. While it is impossible to prove a direct causal relationship, a significant number of problems identified in the 1976 study of the medicine clerkship have since been alleviated. Such modifications have included the addition of certain scheduled activities such as renal and neurology rounds, movement toward improved matching of students to assigned residents, efforts to improve the teaching skills of house officers, establishment of a faculty mentor system, and provision for more frequent feedback to students on their academic progress. (The clerkship is now 11 weeks in length, but this change has been planned before the 1976 study.) Another measure of the perceived value of a n evaluation approach might be the number of subsequent requests for similar studies. Since the 1976 study, the Department of Medical Education has been asked by members of the Department of Medicine to conduct an anthropological study of the four-week gastroenteroloe course. Also, a similar study of the emergency medicine service has been completed, and plans are underway for a study of the psychiatry clerkship. Not surprisingly, the anthropological evaluacion approach is generally expensive in terms of personnel erne. In addition to the initial observation. the cime log repor. are time-consuming to prepare. They. in turn. !ield a vast amount of information which must be organized and processed by hand. However. [he indi\idudc \.tho s e d as observers worked with [he medicine clerkship on a continuing basis and reported that the observation itself was of great value in helping &em to gain a better understanding of the clerkship. Severtheless, contern plation of similar evaluatix-e projects should include consideration of this relatively large time commitment with the expected derived benefits.

Evaluating the Evaluation


While it might appear obvious that this type of evaluation naturally yields a rich composite picture of the educational programme studied, there might be some concern about the objectivity of the resulting report. If one accepts the purpose of programme evaluation as comMedical Teacher VoL 2 N o 4 I980

In summary, the anthropological methodology provides a rich composite portrait of a set of activities and their perceived effects on participants. Proper controls in designing the evaluation ensure an accurate represent ation of the programme. In addition, we believe that the

Interactions with Medicine Faculty Members Students were observed interacting with faculty members, clinical associates and attending physicians 37 times regarding patient- or medicine-related topics and three times concerning topics unrelated to patients or medi cine. These encounters ranged in length from under one minute to 45 minutes, with the median interaction lasting six minutes. Over half of the interactions occurred because the student approached the physician about a patient, disease or procedure. Five interactions took place as the physician performed a procedure, such as a gastroscopy or proctoscopy .

Variation According to Assigned Resident The most clearly observed difference in this study was the variation in interaction between residents and students regarding the degree to which each student and resident worked together, who did the work, the resident’s approach to instruction and the degree to which the resident encouraged the student. Such variation influenced the learning experience, assuming that observation as a mode of learning differs from the opportunity to practise clinical decision-making ,

Assertiueness and Learning Opportunities


nature of the report communicates more effectively and has more impact on readers whose native language is English than do statistics. This form of evaluation also has potential for answering questions which had not been posed at the time the data were collected. Finally, the experience of conducting the evaluation is a highly productive source of insight when the people involved continue to work with the programme.

Med Teach Downloaded from informahealthcare.com by University of Sydney on 01/02/15 For personal use only.

References Parlett, Malcolm and Hamilton, David. Evaluatzon as Illumznatzon A New A p p o a c h to the Study of Innovatory Programmes, Occasional Paper 9, Center for Rescarch in the Frlucational Sciences, University of Edinburgh. 1972. Schermerhorn, Gerry R. and Williams, Reed G , An empirical corn. parison of responsive and preordinate approaches to program evaluation, Educational Evaluation and Polzcy Analysis, 1979. 1 , 53 60 Stake, Robert E . , To evaluate an arts program, Journal of Aesthetzc Mucatzon, 1976, 10, 115-133

Assessment in Medical Education A course on assessment will be held at the Centre for Medical Education, Dundee from 8 to 12 September. It is aimed at staff concerned with undergraduate, postgraduate and continuing medical education and for teachers of para -medical workers. The course reflects the belief that individuals involved in medical education profit from meeting together to discuss and evaluate the teaching of medicine. It provides a focus for an examination of the functioning, effectiveness and rationale of different educational practices. The course provides a structured framework within which the complexities and constraints of existing edu cational practice can be examined and discussed. The emphasis throughout will be on practical application of ideas and during the course participants will discuss the implications of some new and old techniques in their own situation, inspect examples, talk with teachers who have implemented new ideas and prepare teaching or assessment material for their own situation. The topics covered in the course will include: testing knowledge: testing clinical and practical skills; assessing attitudes; when students should be assessed: who should assess students: and the role of assessment in medical education. At the end of the course participants should be able to examine critically techniques of teaching which are cur rently in use in their own situation; suggest alternative techniques and offer some guidance on the implications of adopting other techniques; and examine critically their own teaching methods and identify areas for improvement. Fees for the course (excluding travel and accommodation) are El25 for United Kingdom participants and f250 for overseas participants. Full details are available from Professor R . M. Harden, Director, Centre for Medical Education, Level 8, Ninewells Hospital and Medical School, Dundee DDl 9SY, Scotland (Tel. 0382-23181, Ext. 617). 174

American Students’ Loans in Jeopardy The Carter Administration’s proposed phase-out of loans and capitation grants for health professions students will have a severe negative impact on the quality of e d u cational programmes and the ability to health professions schools to respond to health goals and priorities of Congress, according to Anthony J . McNevin, chairman of the Federation of Associations of Schools of the Health Professions. The Federation of 10 associations for the education of students in the health professions is concerned about the serious implications of the proposed reductions in capit ation grants for financial year 1979 and the complete phase-out of the capitation programme for financial year 1980, elimination of loans to health professions students, and severe cutbacks in support for nurse training. T h e capitation proposal includes a decrease of $76.7 million for medical, osteopathic and dental schools, $23 million for nursing schools, and additional large reduc tions for veterinary, optometric, pharmacy and podiatric schools. The proposed rescission for nursing schools and training programmes, depleting funds for research and research training, follows the President’s promise that nursing programmes would not be hurt by his veto of earlier nurse training legislation. The programme of federal capitation support was introduced in 197 1, when the Federal government made a policy decision to provide direct institutional assistance to eligible health professions schools. Over the past seven years this money has been used to improve significantly the health care system of the USA by addressing national priorities, training more practitioners, improving cur ricula, recruiting minorities and women, and maintain. ing quality educational programmes at a time of rapidly escalating costs. The Federation emphasizes that reductions in capit ation place increasing fiscal constraints on health pro fessions schools. These reductions will have the effect of raising tuition and other educational expenses, and will impede compliance with the recently announced tuition containment effort by the Department of Health, Edu cation and Welfare. In this light, reductions in loans to health professions students are of particular concern. These loans represent one of the few viable options available to many students faced with increasing edu cational costs. Thus, there is an inconsistency in the reduction of important capitation funds at the same time as the reduction of loans which could assist students in meeting the increased costs of education. While the amount of capitation money has been tremendously varied and has been seriously eroded by inflation, the constituency represented by this coalition wants the Congress to understand how important this flexible money has been to the operation of the institutions, and how vital loans for health professions students have been in assisting needy students to meet educational costs. Medical Teacher V o l 2 No 4 1980 ~

Original research: programme evaluation: an anthropological look at the clerkship.

Members of the Departments of Medical Education and Medicine sought to design a study that would produce an accurate, detailed description of the nine...
565KB Sizes 3 Downloads 0 Views