COMPUTERS
AND
Medical
BIOMEDICAL
RESEARCH
25, 43-55 (1992)
Students Using Grateful Searches and a Six-Month
Med: Analysis of Failed Follow-up Study
JOYCE A. MITCHELL, E. DIANEJOHNSON,JOHN AND VIRGINIA K. PROUD University
Of Missouri,
School
of Medicine,
Columbia,
E. HEWETT,
Missouri
65211
Received December 20, 1990
Medical students at the University of Missouri were taught to search MEDLINE using the Grateful Med software. The traffic files of student searches were analyzed for search behavior. This paper reports on two specific aspects of these analyses: (1) failed searches where retrieval was either too large or too small (zero hits); and (2) a six month followup study which focused on error rates and retention of skills. The two major reasons for retrieving too many citations were entering a phrase on a single subject line and a search of only one concept. Of the zero hit searches, the most commonly occurring errors were MeSH (medical subject headings) not used, other MeSH errors, misspelling or keyboard errors, and faulty logic. During the six month follow-up study, the error rates of students declined in spite of the fact that they had not used the software in six months. Furthermore, a larger percentage of the searches showed a more sophisticated use of MeSH than previously employed. 0 1992 Academic PESS, IIIC.
INTRODUCTION
A crisis exists in medical education because of the time constraints of the curriculum and the ever increasing amount of information to be covered in both the basic science and the clinical curriculum. One part of the solution to this information explosion is to teach students the skills of information access and to teach them how to pose a clinical or research problem in terms which can be addressed by access to bibliographic sources, databases or other knowledge bases. Studies have shown that neither students nor attending physicians can adequately access the literature to make patient care decisions. Most students and many faculty rely on their own, out-of-date, personal reference library to answer the majority of patient care questions (Z-S). This traditional lack of training in information retrieval carries over into the medical practice behavior of physicians in later years. Office-based physicians usually do not perform literature searches on bibliographic databases even when they own the equipment and software to do so (4). This behavior persists even though a large percentage of 43 OOlO-48O9/92 $3.00 Copyright 0 1992 by Academic Press, Inc. All rights of reproduction in any form reserved.
44
MITCHELL
ET AL
patient-related questions are left unanswered; questions which could have altered patient management decisions (5). The opportunities for end-user searching are increasing rapidly because of the dramatic improvements in the availability and user-friendliness of end-user systems for literature retrieval (6, 7). Studies abound in the methods and relative ease of using different systems (8-11). One must be concerned, however, that the end-users are able to retrieve the available information needed to assist in decision making. Some searches performed by end-users clearly do not retrieve all of the relevant literature; the concern is that this lack of complete information may adversely affect clinical decisions (12). To investigate this effect. searches run in duplicate or triplicate by different groups of searchers have found that there were always some unique, relevant articles retrieved by each of the groups including the novice searchers (13-15). although trained librarians retrieve a larger percentage of relevant articles. Clearly, the literature and the language are so complex and voluminous that it is impossible to guarantee complete retrieval even by expert searchers. It appears likely that total recall is not as important to a busy clinician as currency and precision. Time pressures would not permit the utilization of “complete” information even if it could be defined. The obvious solution to this information explosion and the proliferation of end-user systems is to teach physicians to search the literature and access databases as part of their academic training and to develop methods to access the efficacy of the teaching. The process of learning to use computerized databases effectively entails four steps: (1) being aware that the database exists: (2) acquiring some basic searching skills; (3) recognizing when it is appropriate to use the database: and (4) being motivated to actually use the database. The teaching methods must include the heuristics of searching and problem solving. Once information gathering techniques are mastered, they can permanently shape thinking and patient care (16, 17). Various reports (6, 14, 18-22), including our own recent work (23, 24). have shown that medical students can be trained to use Grateful Med or other bibliographic systems. The research in this article is on two topics related to teaching medical students to use Grateful Med to search the medical literature and a study of the efficacy of such training: (1) an in-depth analysis of failed searches during the time immediately after initial training during a genetics exercise in their first year biochemistry course. and (2) a follow-up study of the same students six months later when they searched the literature again for a genetics exercise in the second year pathology course. METHODS
The project began as a pilot program to incorporate informatics techniques into medical genetics education. Both of these disciplines are taught at the University of Missouri in a series of lectures or exercises and not as a complete course. In the biochemistry course, we taught first year students to use Grateful
MEDICAL STUDENTS USING GRATEFUL MED MEDSDAOZ MEDSDA02 MEDSDAOZ MEDSDAOZ MEDSDAOZ MEDSDAOP MEDSDAOZ MEDSDAOZ MEDSDAOZ MEDSDA02 MEDSDAOZ MEDSDAOZ MEDSDAO:! MEDSDAOZ MEDSDAOZ MEDSDAOZ MEDSDAOZ MEDSDAOZ MEDSDAO2 MEDSDAO? MEDSDAOZ MEDSDAOZ MEDSDAOZ
06B 06B 06B 06B 06B 06B 065 06B 06B 06B 06B 068 063 06B 06B 06B 06B 06B 06B 06B 06B 06B 06B
89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 FIG.
1.
1649223 1649224 1649225 1649297 1649310 1649364 1649366 1649407 1649409 1649443 1649446 1649490 1649521 1649572 1650022 1650022 1650023 1650202 1650203 1650203 1650203 1650335 1650355/
45
/LOGIN a.~sDAoz /TIGER ITI WELCOME TO THE NATIONAL LIBRARY OF MEDICINE'S E YOU ARE NOW CONNECTED TO THE MEDLINE (1988 - 89 USERS GMF4MEDLINE PROG: 32 USERS LOGGED IN PRESENTLY. SS 1 / XERODERMA PIGMENTOSU?l (MH) PROG: SS (1) PSTG (44) SS 2 /C? USER: PHYSIOLOGY (SH) PROG: SS (2) PSTG (28859) SS 3 /C? USER: lAND2 PROG: SS (3) PSTG (3) SS 4 /C? USER: 3 AND HOT FOR (LA) PROG: SS (4) PSTG (3) SS 5 /C? USER: PRINT INCLUDE AD.MH.PS.LA AD - Department of Molecular Oncology, Faculty + PROG: 1 UI *. 88288298 AU - Arita I AU USER: Y PROG: 3 UI *. 88094464 AU - Lambert MW A X treated DNA. AD - Department of Patholo MH - Xeroderma Pigmentosum/*ENZYMOLOGY/GENETIC STOP Y GMMEDLINE 000220010000 3 34 " PROC: THE ESTIMATED TOTAL ONLINE GOOD-BYE!
One traffic file record for one Grateful Med search.
Med to search MEDLINE for an assignment on a specific genetic disease with a biochemical basis. The students were to find articles from the literature and write a brief abstract of the disorder which would focus on what was new about that disease. The students presented their findings in small group discussions. The students were given five weeks of non-billed computer time and encouraged to use Grateful Med to search MEDLINE for other topics in addition to completing their assignments. Data analysis was based on a questionnaire with descriptive information from the students and the analysis of NLM traffic files of the actual searches. A traffic file for one search on the topic of xeroderma pigmentosum is shown in Fig. 1 with the corresponding coded data form (1990 version) in Fig. 2. The coding involved a translation of the Elhill command language to the use of the Grateful Med form screen. Each traffic file was coded by two of the researchers (J.A.M. and E.D.J.) to ensure uniformity of interpretation and translation. The data was stored in a dBase III file and uploaded to the mainframe for analysis with SAS. A more complete description of the methods and an analysis of the basic biochemistry searches have been published elsewhere (23, 24). Analysis of Failed Searches (Biochemistry) A detailed analysis was conducted on successful subject searches conducted during the biochemistry study. For the purposes of this analysis, a failed search was defined as a subject search in the current MEDLINE file which either retrieved nothing at all or retrieved in excess of 300 citations. Two hundred eight (208) of the 713 analyzable searches of the current file met these criteria (see Fig. 3). Of these, 188 were “zero hits” searches, and the remaining 20 retrieved greater than 300 items. Searches belonging to this subset were an-
MITCHELL
46
U Line Line Line Lme
1: 2: 3: 4:
$2:: Y Y
Cit; ations Current. Backfiles
Other Logic
Other _ Search
_ _ _ _ Other __ _
d FlO
Used
N N
1986 1983 1960 ,077 ,-,_ 1966
Used DN Y Y Y Y i’
ET AL.
Text Word Y N Y N Y N Y N
Used
Retrwed 3
Forward Slash Y N Y N Y N Y N
Printed
3
N N N hl i
Flies. Error (Y/N)’ Specific :
9,j __ _ _
Phrase entered on one Redundant Search Other (specify):
Errors (YIN) Author Truncation Error Misspelling Forward Slash used incorrectly Non- MeSH term Other (specify). Features Used (Y/N) AIM ALL Override Attached Subheading Author Citation Pearl Same as prewous search Same search but printed Other (specify) : Notes (YIN) Computer/ MechanIcal Form Screen Override Librarian Assisted
line
Subhead Error. _ Subhead Error - Not QuaIlfled _ Subhead Error - Entered Dashes _ Starring Subheadmg
more
Problem
J _ - 4 _
Engksh Explosion Floating Subhead OR used Pre - Explode
_ _ _
_
Same
search
_ _
Store Unable
Postmgs Overflow to completely analyze
Review Smgle Journal Starring Trtle Truncation
but on backfiles
Notes:
L
_
Search
Okay Missed Missed Missed Missed
Opportunity Opportunity Opportunity Opportunity
Ouakty
-
MeSH term omitted from disk MeSH elsewhere on disk FlO not used Fatled to Explode
p (Exce:lent) FIG.
3 (Average)
_
_
4
Too Restnctwe - Review Too Restrictive AIM Too Restrictwe - Eng Specialty term vs dtsease
5 (Abysmal)
2. Coded section of traffic file corresponding to Fig. 1
term
MEDICAL
STUDENTS
USING GRATEFUL
0 HITS
47
SUBJECT 188
0 HITS
FIG.
MED
AUTHOR 26
400 3. Deriving a subset from current file searches: Biochemistry (n = 713)
alyzed in depth in an attempt to identify commonly recurring errors and problems in search construction as well as “missed opportunities” (25), where bypassing MeSH might have affected the outcome of the search. The search in Table 1 will serve to illustrate some of the guidelines used in the in-depth analysis. The first line, nitrates, was obviously not selected from the MeSH disk, even though it is in fact a MeSH term. But since Grateful Med searches single word terms both as MeSH headings and as text words, the student did not lose anything by not selecting “nitrates” from FlO. Therefore, this would not be classified as an error. On the second line, however, the outcome of the search may have been affected by not pressing FlO and locating the proper MeSH term: Dogs. This was counted as a missed opportunity: FlO not used. The third line would also be tagged as a missed opportunity. The subheading “physiopathology” would be the MeSH equivalent, but since the student entered “pathophysiology,” he would not have found the term even if he had pressed FlO. The MeSH term he needed was located in another part of the alphabetical MeSH list. This would be coded as a missed opportunity; MeSH term elsewhere in the FlO list. On the fourth line, the forward slash was used with a word that is neither a MeSH term nor a cross reference. It would be coded as a forward slash error.
TABLE
1
GRATEFULMEDSAMPLESEARCH
Nitrates (TW) or nitrates (MH) Dog (TW) or dog (MH) Pathophysiology (TW) or pathophysiology (MH) IVenodilation
48 Pathology
MITCHELL
Search Follo~~p
ET Al
Study
A follow-up study was performed during the second year of medical school in the pathology course, which focused on specific genetic disorders in a case discussion format. The same students who had used Grateful Med the previous spring were studied for their use of the system in pathology. Once again nonbilled access to MEDLINE was provided and they were invited to use Grateful Med to search other medical topics in other courses during a six week period. They were given a two-page handout describing their most common errors on using Grateful Med in biochemistry and some suggestions for overcoming those errors. The students were given six topics as part of case workups and invited to break into groups of four students to gather the requisite information and discuss the results. Since the questions had multiple subtopics. not all of which required literature searches, the student groups usually parcelled out the subparts and then met to organize the overall case discussion. Usually only one of the four students did a search on the group’s topic, although multiple students might search on subtopics. Even though the four students worked as a group to gather all of the case materials, the students usually performed the Grateful Med searching independently. Each student used an individual password and his/her searching behavior was analyzed on an individual basis. The traffic file data were analyzed with the same procedures as in the biochemistry study for basic parameters. In order to compare the errors and use of Grateful Med features during the two exercises (biochemistry and pathology) we calculated a summary score for each student to allow tracking students through time. We could not directly compare the results of the two class expertments because each student contributed a different number of searches to the total and thus the groups of searches were not independent. We compared the two experiments by: (1) calculating a mean score for the two experiments for each student, (2) calculating a difference in the mean score between experiment two (pathology) minus experiment one (biochemistry), and (3) testing the hypothesis that the difference in mean score is different from zero (although the hypothesis may be directional depending upon the assumptions) using the Wilcoxon signed rank test. For example. we calculate an error rate per student which is the proportion of searches with errors and test the hypothesis that the error rate declined from biochemistry to pathology. If the data was dichotomous. then McNemar’s test was used to compare proportions.
Analysis
oj’Failed
Searches
(Biochemistry)
Of the 208 searches in the subset. 20 retrieved over 300 citations, ranging from 309 to 13,842 hits. In nine of these searches, the student had entered a phrase on a single subject line, causing the search words to be ORed rather than ANDed together. These search failures can be attributed to a misunderstanding of the logical underpinnings of the Grateful Med form screen, compounded by
MEDICAL
STUDENTS
USING GRATEFUL
TABLE FAILED
SEARCH
ANALYSIS:
NO-HITS
49
MED
2
SEARCHES
IN BIOCHEMISTRY
(n = 188)
Error/problem
Number
Percent
No FlO used No errors MeSH elsewhere Forward slash error Logic error Too narrow: Review Misspelling Choice of term Redundant search Too narrow: English Too narrow: Title Too narrow: AIM
53 43 37 26 19 19 15 12 11 8 I 4
28 23 20 14 10 10 8 6 6 4 4 2
the Version 3 “enhancement” which permitted the user to remove the ORs which appear each time the space bar is pressed while on a subject line. This editing on the users’ part made no difference in the processing of the search; Grateful Med simply reinserted the ORs when the search was run. The remaining 11 searches retrieved over 300 items because the student had entered only a single concept. Overall, however, relatively few of the searches failed by retrieving over 300 citations. A far more common problem was the search which retrieved no articles at all. One hundred eighty-eight (188) of the searches in the subset were “zero hits” searches. Of these, 23% (43) contained no discernible errors or problems. On the remaining searches, the most common problems were the missed opportunities (see Table 2). Twenty-eight percent (28%) of the searches might have retrieved something if the student had searched using a MeSH term; and they would have quickly located the MeSH equivalent for their text word had they only pressed FIO. On 20% of the searches, one or more of the text words that the student chose had a MeSH equivalent, but it would not have been immediately accessible via FlO using their text word as an entry point. The MeSH term would have been in another part of the alphabetical list. For example, while many of the students were searching“nitroglycerin” for a physiology assignment, few of them made it to “glyceryl trinitrate in the MeSH list.” The third most common error (14% of the searches) was the use of the forward slash with a non-MeSH term. We did not want to teach beginning searchers to use the forward slash override on the Grateful Med form screen; however, the version of Grateful Med in use in 1988 (Version 3) did not have the complete MeSH on the disk. The biochemical disorders which the students were searching are commonly characterized by enzyme deficiencies; the MeSH headings for many of those enzymes were omitted from the MeSH disk, and thus had to be entered on the Grateful Med search form preceded by a forward slash.
SO
MITCHELL
ET AI
Misspellings of these terms were common along with the actual forward slash error. As with the >300 hits searches, logic errors accounted for some of the failures (10% of the searches), when the searcher entered a phrase on a single subject line, causing the words to be ORed rather than ANDed. This error (which usually causes too many hits) was caused by picking two or more terms which individually had zero hits. Some of the searches yielded nothing simply because they were overly restrictive. Most often this occurred when the searcher attempted limiting to only review articles (10% of the searches), although limiting to title words, English language, or AIM journals accounted for some of the “no hits” searches t2-4% of the searches). It is likely that the choice of MeSH term affected the outcome in 6% of the failed searches. Most often the searcher selected a specialty term like “cardiology” instead of the corresponding disease term”heart diseases.” Puthology
Search Follow-up
Study
A total of 446 searches were completed by 53 students during the follow-up study. Fifty percent of the searches were on the assigned genetics topics and 50% were on other topics. Because the assignment did not require each student to search independently, it was not surprising that only a subset of the students used Grateful Med. The most interesting data concerned comparisons between the errors in the entire set of searches (not just failed searches) and use of Grateful Med features during the two exercises (biochemistry and pathology). Eleven percent of the pathology searches had at least one error, while 20% of the biochemistry searches had at least one error, a significant decline in error rate per student. Data were analyzed to determine whether the students who made the most errors in biochemistry simply did not search the literature in the pathology exercise, but this was not statistically significant. However, there was strong significance (p < ,002) when one considers those students who searched and made errors in biochemistry and then consider their error rate in pathology; the error rate declined. Furthermore, a larger percentage of the searches in the pathology study used MeSH exclusively than during the biochemistry study. This decline in error rate may be partially attributable to the handout which contained a discussion of the most common error (entering a phrase on one line) and how to avoid it. Other routes of access to MEDLINE were not readily available to these students in 1988 so the decline in error rate was probably not from other independent searching. The staff of the health sciences library did report a considerable increase in student search requests during the six month hiatus, demonstrating an increased awareness and interest in computer mediated searching. However, a very few students would have been able to search independently and practice their search skills without Grateful Med codes.
MEDICAL
STUDENTS
USING
GRATEFUL
MED
51
We examined the change in specific types of errors in the entire set of searches. Whereas the trend appears to be an improvement from the previous experiment, there were no statistically significant differences between the specific types of errors in the two experiments. It appears that novice users can return to use the Grateful Med searching software after six months of disuse and recall the basic search techniques and correct use of the software. Most of the other end-user systems require frequent use to maintain low error rates. We did not do an in-depth study of failed searches in the pathology searches, but have some genera1 information on the magnitude of the problems. The overall magnitude of “zero hits” in the current file, non-author searches was 20%. The greatest difficulty arose because the student did not use the FlO key to select a MeSH term when an appropriate term might have improved retrieval. The percentage of searches retrieving greater than 300 citations had increased to 7%, attributed to the broader subject areas for the search topics compared to the previous exercise. This serves to emphasize that trainers must teach troubleshooting of searches with special attention to recovery from zero hits. When one compares the genetics searches versus the non-genetics searches, some interesting trends appear. We graded the searches on overall quality on a scale from 1 (excellent) to 5 (abysmal). The students had better search quality and fewer missed opportunities on genetics searches than when searching on other topics. They searched the backfiles more frequently on subjects other than genetics. As was found in the biochemistry study, the searches on nongenetics topics used more lines and more text words compared to the genetics topics. This could be interpreted as more complex searches on other topics or, alternatively, less focused search statements since MeSH was used less frequently when they searched independently on non-genetics topics. Overall, the students were doing very complex searches compared to other end-user studies (25,26) which showed a high percentage of single concept searches with most end-users, even after considerable experience. Figure 4 is a graph of individuals in the class who searched the literature in both biochemistry and pathology. There was no correlation between the number of searches each student performed in biochemistry and pathology. It was surprising that some individuals who did no independent searches in the first exercise were frequent searchers in the second exercise. Over the two studies, a large number of the students were frequent searchers. Developing a profile of attributes which characterize the best or the most frequent searchers is elusive. We analyzed the total (biochemistry plus pathology) number of searches, the total search time, the number of search sessions, the percentage of searches with no errors compared to the background statistics and previous training of the students as determined from the biochemistry questionnaire. Once again, there were no statistically significant correlations; all regression equations showed no significant predictors of performance. Of the four students with no prior computer experience and no independent searches in the biochemistry experiment, there was only one student of the group who still had not touched the computer independently after the pathology assignments.
m
Biochemistry
FIG. 4. This graph shows the number of independent searches done by each of I I.5students during the biochemistry exercise and the pathology exercise.
DISCUSSION
These studies lead to some interesting conclusions when compared to previous studies of end-user searching of literature databases. The students appear to be interested in searching independently, and a few search quite frequently. This phenomenon of a few highly motivated searchers is in concordance with other studies (9, 14, IS, 25, 27). A unique result of this work is the longitudinal follow-up which shows that the most frequent searchers in biochemistry were not the same people as the most frequent searchers in pathology. One of the surprises of our research was that the students took the opportunity to do searches on other genetics topics as well as on other medical topics. This may demonstrate the perception of the students that computer mediated literature searching is an efficient way to use their time. Furthermore, they were better prepared for their discussion sessions because they investigated the overlap between their assigned disease and similar diseases assigned to other students. The error analysis would contribute to the impression that students can effectively use the MeSH vocabulary if trained in the proper techniques. Furthermore, the ease of use of the FIO key to access the appropriate vocabulary is one key reason that the students can return to Grateful Med and use it with fewer errors after a six month hiatus. The continued improvements in Grateful Med will ensure that the software is easy to use.
MEDICALSTUDENTSUSINGGRATEFULMED
53
There are substantial practical applications for Grateful Med trainers which arise from our studies of errors and failed searches. We have made several changes in the content and emphasis in subsequent training sessions based upon the traffic file analysis. First, since logic errors appeared on 13% of the 208 failed searches (and 11% of the total 829 searches), additional emphasis was placed upon principles of Boolean logic and the implementation of Boolean operators on the Grateful Med form screen. Prior to the traffic file analysis, we did not belabor Boolean principles in training sessions, since it seemed that the Grateful Med form screen made Boolean logic rather transparent (an erroneous assumption). An extra effort was made to stress the value and uses of subheadings in subsequent training sessions. Even though subheadings were addressed in the initial training, they were still a source of confusion and accounted for many errors and missed opportunities. No systematic coding was attempted to classify various problems with subheadings, but there were many failed attempts to employ them. In addition, we began to highlight the difference between specialty and disease terms whenever the opportunity arose. Although this was not a common problem, it often made a critical difference in the outcome of the search when the wrong term was selected. Grateful Med programmers began to address this distinction in Version 4 through the use of scope notes on many of the specialty terms. With 30% of all searches resulting in zero hits, we strongly recommend that trainers provide novice searchers with some hints for recovering from a no hits search. The Found column incorporated into Grateful Med Version 4 and higher versions should help the user to recognize and correct overly specific searches; we have incorporated a sample search illustrating the use of the Found column in our training sessions. We also provide some general guidelines for broadening and narrowing searches through the addition and deletion of lines on the form screen. Beginning medical students can become relatively efficient and frequent users of Grateful Med to search MEDLINE. We have analyzed the major patterns of uses, errors, and patterns of behavior associated with student use of Grateful Med, although we have not studied the effectiveness of the searches. It appears that this group of students has some initial skills and the movtivation to use Grateful Med; it took remarkably little time from the traditional curriculum. However, the extent to which this skill is integrated into the daily practice of clinical medicine remains to be determined.
ACKNOWLEDGMENTS This work was supported in part by a grant from the Mizzou Alumni Association and with nonbilled codes granted by the National Library of Medicine.
MITCHELL
54
ET AL
REFERENCES I. DEROSA. D. A., MAST, T. A., DAWSON-SAUNDERS. B.. MAZUR. J.. RANSEY. D. E.. AND FOLSE. J. R. A study of the information-seeking skills of medical students and physician faculty. .I. Med. Educ. 58, 45 (1983). 17. NORTHRUP. D. E., MOORE-WEST, M.. SKIPPER, B.. AND T~AF. S. R. Characteristics of clinical information-searching: Investigation using critical incident technique. J. Med. Edrrc,. 58, 873 (1983). .q. STINSON. E. R.. AND MUELIXR. D. A. Survey of health professional!,’ informatIon habits and needs. Am. Med. Assoc. 243, No. 2. 140 (1980). 4. L. HARRIS AND ASSOCIATES “The Future of Information Systems for the Medical Sciences; A Study Conducted for the New York Academy of Medicine.” New York Academy of Medicine. New York, 1988. 5. COVELL, D. G.. UMAN. G. C.. AND MANNING, P. R. Information needs in office practice: Plre they being met? Ann. Infern. Med. 103, 596 (1985). 6. WELBORN, V., AND KUEHN. J. J. End-user programs in medical school libraries: A survey. Bull. Med. Lihr. Assoc. 76, No. 2. 137 (1988). 7. SOBEN, P., AND GREEN. E.W. GRATEFUL MED searching: A hospital library perspective. Med. Rqf. Scruiws Qnurteriy 7. No. 3. 35 (1988). 8. PORTER, D.. WIG~~N. R. S.. REIDELBACH, M. A.. BL.EICH. H. L.. AND SI.A~~. W. V. Selfservice computerized bibliographic retrieval: a comparison of Colleague and PaperChase. programs that search the MEDLINE data base. Camp. und Biomed. Rex. 21:488-501 ( 1988). 9. BOHHAM. M. D.. AND NELSON. L. L. An evaluation of four end-user systems for searching MEDLINE. Bull. Med. Lihr. Assoc. 76( 1):22-31 (1988). 111. MILLER, N.. KIRBY M.. AND ETHELDRA, T. MEDLINE on CD-ROM: end user searching in a medical school library. Med. Ref. Sew. Q. 7, No. 3. 1 (1988). II. SIMON, F. A. A comparison of two computer programs for zearching the medical literature. J. Med.
Educ.
63, No.
4. 331 (1988).
12. BURROUGHS. C. M. Clinicians’ satisfaction with Grateful Med: An exploratory study. B&L. Med. Libr. Assoc. 77, No. 1. 56 (1989). I3. SARACEVIC. T.. AND KANTOR. P. A study of information seeking and retrieving. 111.Searcher\. searches. and overlap. J. Am. SOC. fnf. Sri. 39. 197 (1988). 14. MCKIBBON, K. A.. WALKER. C. J., RYAN. N. C.. FI~ZGERAI.D. D.. AND HAYNES, K. B. .4 study of MEDLINE in clinical settings: Design and preliminary results. In “SCAMC Proceedings” (R. A. Greenes. Ed.). pp. 526-519. Computer Society Press. Los Angeles, CA. 1988. 15. HAYNES. R. B.. MCKIBBON. K. A.. WALKER, C. J., RYAN. N.. FITZ~;ERAI.D. D.. .+.lu~ RAFTSDEN, M. F. Online access to MEDLINE in clinical settings-A study of use and usefulness. Ann. Intern. Med. 112, 78 (1990). 16. PERKINS, D. N. The fingertip effect: How information-processing technology shapes thinking. Ed/cc. Rpx. pp. I l-17, (AugiSep 1985). 17. HAYNES. R. B., RAMSDEN, M., MCKIBBON, K. A.. WALKER. C. J.. AND RYAN. N. C. A review of medical education and medical informatics. Acad.Med. 64, No. 4, 207 (1989,. 18. ZAROUKIAN, M. H.. HAF, J. J.. AND ALFANO, M. A. MEDLINE computerized literature searching using GRATEFUL MED: A pilot instructional prqject for entering medical students. At,rrd. Med.. submitted. 19. BRADIGAN. P. S., AND MULARSKI. C. A. End-user searching in a medical school curriculum: An evaluated modular approach. Bull. Med. Lihr. .4.~so~~. 77. No. 4. 348 (1989). 20. BURROWS. S.. GINN. D. S.. LOVE. N.. AND WILLIAIIS. I‘. L. A strategy for curriculum integration of information skills instruction. Bull. Med. Lihr. Assoc. 77, No. 3. 245 (1989). 21. KIMMEL. S. Teaching third-year medical students to search MEDLINE. Med. Rtf Sew. Q. 8. No. 3. 69 (1989). 22. RODNICK, J. E., SIMRIN, S. M., YANG. M. G., AND ALTMAN, D. F. Teaching medical students to do bibliographic searching. J. Med. Educ. 63, 728 (1988). 23. PROUD. V. K., SCHMIDT. F. J.. JOHNSON, E. D.. AND MITCHELI, J. A. Teaching human genetics in biochemistry by computer literature searching. Am J. Hwn. Gencr. 44, 597 (1989).
MEDICAL
STUDENTS
USING
GRATEFUL
MED
55
MITCHELL, J. A., JOHNSON, E. D., AND PROUD, V. K. New thoughts about medical students as effective searchers of MEDLINE. Acad. Med. 65, No. 7, 434 (1990). 25. SEWELL, W., AND TEITELBAUM, S. Observations of end-users online searching behavior over eleven years. J. Am. Sm. Zn$ Sci. 37, No. 4, 234 (1986). 26. KIRBY, M., AND MILLER. N. MEDLINE searching on Colleague: reasons for failure or success of untrained end-users. Med. Ref. Serv. Q. 5, No. 3, 17 (1986). 27. WALKER, G. “End-user searching: A selection of the literature for 1983-1988.” An ERIC synthesis paper. ERIC Clearinghouse on Information Resources, Syracuse, NY, December 1988. 24.
AND
Medical
BIOMEDICAL
RESEARCH
25, 43-55 (1992)
Students Using Grateful Searches and a Six-Month
Med: Analysis of Failed Follow-up Study
JOYCE A. MITCHELL, E. DIANEJOHNSON,JOHN AND VIRGINIA K. PROUD University
Of Missouri,
School
of Medicine,
Columbia,
E. HEWETT,
Missouri
65211
Received December 20, 1990
Medical students at the University of Missouri were taught to search MEDLINE using the Grateful Med software. The traffic files of student searches were analyzed for search behavior. This paper reports on two specific aspects of these analyses: (1) failed searches where retrieval was either too large or too small (zero hits); and (2) a six month followup study which focused on error rates and retention of skills. The two major reasons for retrieving too many citations were entering a phrase on a single subject line and a search of only one concept. Of the zero hit searches, the most commonly occurring errors were MeSH (medical subject headings) not used, other MeSH errors, misspelling or keyboard errors, and faulty logic. During the six month follow-up study, the error rates of students declined in spite of the fact that they had not used the software in six months. Furthermore, a larger percentage of the searches showed a more sophisticated use of MeSH than previously employed. 0 1992 Academic PESS, IIIC.
INTRODUCTION
A crisis exists in medical education because of the time constraints of the curriculum and the ever increasing amount of information to be covered in both the basic science and the clinical curriculum. One part of the solution to this information explosion is to teach students the skills of information access and to teach them how to pose a clinical or research problem in terms which can be addressed by access to bibliographic sources, databases or other knowledge bases. Studies have shown that neither students nor attending physicians can adequately access the literature to make patient care decisions. Most students and many faculty rely on their own, out-of-date, personal reference library to answer the majority of patient care questions (Z-S). This traditional lack of training in information retrieval carries over into the medical practice behavior of physicians in later years. Office-based physicians usually do not perform literature searches on bibliographic databases even when they own the equipment and software to do so (4). This behavior persists even though a large percentage of 43 OOlO-48O9/92 $3.00 Copyright 0 1992 by Academic Press, Inc. All rights of reproduction in any form reserved.
44
MITCHELL
ET AL
patient-related questions are left unanswered; questions which could have altered patient management decisions (5). The opportunities for end-user searching are increasing rapidly because of the dramatic improvements in the availability and user-friendliness of end-user systems for literature retrieval (6, 7). Studies abound in the methods and relative ease of using different systems (8-11). One must be concerned, however, that the end-users are able to retrieve the available information needed to assist in decision making. Some searches performed by end-users clearly do not retrieve all of the relevant literature; the concern is that this lack of complete information may adversely affect clinical decisions (12). To investigate this effect. searches run in duplicate or triplicate by different groups of searchers have found that there were always some unique, relevant articles retrieved by each of the groups including the novice searchers (13-15). although trained librarians retrieve a larger percentage of relevant articles. Clearly, the literature and the language are so complex and voluminous that it is impossible to guarantee complete retrieval even by expert searchers. It appears likely that total recall is not as important to a busy clinician as currency and precision. Time pressures would not permit the utilization of “complete” information even if it could be defined. The obvious solution to this information explosion and the proliferation of end-user systems is to teach physicians to search the literature and access databases as part of their academic training and to develop methods to access the efficacy of the teaching. The process of learning to use computerized databases effectively entails four steps: (1) being aware that the database exists: (2) acquiring some basic searching skills; (3) recognizing when it is appropriate to use the database: and (4) being motivated to actually use the database. The teaching methods must include the heuristics of searching and problem solving. Once information gathering techniques are mastered, they can permanently shape thinking and patient care (16, 17). Various reports (6, 14, 18-22), including our own recent work (23, 24). have shown that medical students can be trained to use Grateful Med or other bibliographic systems. The research in this article is on two topics related to teaching medical students to use Grateful Med to search the medical literature and a study of the efficacy of such training: (1) an in-depth analysis of failed searches during the time immediately after initial training during a genetics exercise in their first year biochemistry course. and (2) a follow-up study of the same students six months later when they searched the literature again for a genetics exercise in the second year pathology course. METHODS
The project began as a pilot program to incorporate informatics techniques into medical genetics education. Both of these disciplines are taught at the University of Missouri in a series of lectures or exercises and not as a complete course. In the biochemistry course, we taught first year students to use Grateful
MEDICAL STUDENTS USING GRATEFUL MED MEDSDAOZ MEDSDA02 MEDSDAOZ MEDSDAOZ MEDSDAOZ MEDSDAOP MEDSDAOZ MEDSDAOZ MEDSDAOZ MEDSDA02 MEDSDAOZ MEDSDAOZ MEDSDAO:! MEDSDAOZ MEDSDAOZ MEDSDAOZ MEDSDAOZ MEDSDAOZ MEDSDAO2 MEDSDAO? MEDSDAOZ MEDSDAOZ MEDSDAOZ
06B 06B 06B 06B 06B 06B 065 06B 06B 06B 06B 068 063 06B 06B 06B 06B 06B 06B 06B 06B 06B 06B
89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 89055 FIG.
1.
1649223 1649224 1649225 1649297 1649310 1649364 1649366 1649407 1649409 1649443 1649446 1649490 1649521 1649572 1650022 1650022 1650023 1650202 1650203 1650203 1650203 1650335 1650355/
45
/LOGIN a.~sDAoz /TIGER ITI WELCOME TO THE NATIONAL LIBRARY OF MEDICINE'S E YOU ARE NOW CONNECTED TO THE MEDLINE (1988 - 89 USERS GMF4MEDLINE PROG: 32 USERS LOGGED IN PRESENTLY. SS 1 / XERODERMA PIGMENTOSU?l (MH) PROG: SS (1) PSTG (44) SS 2 /C? USER: PHYSIOLOGY (SH) PROG: SS (2) PSTG (28859) SS 3 /C? USER: lAND2 PROG: SS (3) PSTG (3) SS 4 /C? USER: 3 AND HOT FOR (LA) PROG: SS (4) PSTG (3) SS 5 /C? USER: PRINT INCLUDE AD.MH.PS.LA AD - Department of Molecular Oncology, Faculty + PROG: 1 UI *. 88288298 AU - Arita I AU USER: Y PROG: 3 UI *. 88094464 AU - Lambert MW A X treated DNA. AD - Department of Patholo MH - Xeroderma Pigmentosum/*ENZYMOLOGY/GENETIC STOP Y GMMEDLINE 000220010000 3 34 " PROC: THE ESTIMATED TOTAL ONLINE GOOD-BYE!
One traffic file record for one Grateful Med search.
Med to search MEDLINE for an assignment on a specific genetic disease with a biochemical basis. The students were to find articles from the literature and write a brief abstract of the disorder which would focus on what was new about that disease. The students presented their findings in small group discussions. The students were given five weeks of non-billed computer time and encouraged to use Grateful Med to search MEDLINE for other topics in addition to completing their assignments. Data analysis was based on a questionnaire with descriptive information from the students and the analysis of NLM traffic files of the actual searches. A traffic file for one search on the topic of xeroderma pigmentosum is shown in Fig. 1 with the corresponding coded data form (1990 version) in Fig. 2. The coding involved a translation of the Elhill command language to the use of the Grateful Med form screen. Each traffic file was coded by two of the researchers (J.A.M. and E.D.J.) to ensure uniformity of interpretation and translation. The data was stored in a dBase III file and uploaded to the mainframe for analysis with SAS. A more complete description of the methods and an analysis of the basic biochemistry searches have been published elsewhere (23, 24). Analysis of Failed Searches (Biochemistry) A detailed analysis was conducted on successful subject searches conducted during the biochemistry study. For the purposes of this analysis, a failed search was defined as a subject search in the current MEDLINE file which either retrieved nothing at all or retrieved in excess of 300 citations. Two hundred eight (208) of the 713 analyzable searches of the current file met these criteria (see Fig. 3). Of these, 188 were “zero hits” searches, and the remaining 20 retrieved greater than 300 items. Searches belonging to this subset were an-
MITCHELL
46
U Line Line Line Lme
1: 2: 3: 4:
$2:: Y Y
Cit; ations Current. Backfiles
Other Logic
Other _ Search
_ _ _ _ Other __ _
d FlO
Used
N N
1986 1983 1960 ,077 ,-,_ 1966
Used DN Y Y Y Y i’
ET AL.
Text Word Y N Y N Y N Y N
Used
Retrwed 3
Forward Slash Y N Y N Y N Y N
Printed
3
N N N hl i
Flies. Error (Y/N)’ Specific :
9,j __ _ _
Phrase entered on one Redundant Search Other (specify):
Errors (YIN) Author Truncation Error Misspelling Forward Slash used incorrectly Non- MeSH term Other (specify). Features Used (Y/N) AIM ALL Override Attached Subheading Author Citation Pearl Same as prewous search Same search but printed Other (specify) : Notes (YIN) Computer/ MechanIcal Form Screen Override Librarian Assisted
line
Subhead Error. _ Subhead Error - Not QuaIlfled _ Subhead Error - Entered Dashes _ Starring Subheadmg
more
Problem
J _ - 4 _
Engksh Explosion Floating Subhead OR used Pre - Explode
_ _ _
_
Same
search
_ _
Store Unable
Postmgs Overflow to completely analyze
Review Smgle Journal Starring Trtle Truncation
but on backfiles
Notes:
L
_
Search
Okay Missed Missed Missed Missed
Opportunity Opportunity Opportunity Opportunity
Ouakty
-
MeSH term omitted from disk MeSH elsewhere on disk FlO not used Fatled to Explode
p (Exce:lent) FIG.
3 (Average)
_
_
4
Too Restnctwe - Review Too Restrictive AIM Too Restrictwe - Eng Specialty term vs dtsease
5 (Abysmal)
2. Coded section of traffic file corresponding to Fig. 1
term
MEDICAL
STUDENTS
USING GRATEFUL
0 HITS
47
SUBJECT 188
0 HITS
FIG.
MED
AUTHOR 26
400 3. Deriving a subset from current file searches: Biochemistry (n = 713)
alyzed in depth in an attempt to identify commonly recurring errors and problems in search construction as well as “missed opportunities” (25), where bypassing MeSH might have affected the outcome of the search. The search in Table 1 will serve to illustrate some of the guidelines used in the in-depth analysis. The first line, nitrates, was obviously not selected from the MeSH disk, even though it is in fact a MeSH term. But since Grateful Med searches single word terms both as MeSH headings and as text words, the student did not lose anything by not selecting “nitrates” from FlO. Therefore, this would not be classified as an error. On the second line, however, the outcome of the search may have been affected by not pressing FlO and locating the proper MeSH term: Dogs. This was counted as a missed opportunity: FlO not used. The third line would also be tagged as a missed opportunity. The subheading “physiopathology” would be the MeSH equivalent, but since the student entered “pathophysiology,” he would not have found the term even if he had pressed FlO. The MeSH term he needed was located in another part of the alphabetical MeSH list. This would be coded as a missed opportunity; MeSH term elsewhere in the FlO list. On the fourth line, the forward slash was used with a word that is neither a MeSH term nor a cross reference. It would be coded as a forward slash error.
TABLE
1
GRATEFULMEDSAMPLESEARCH
Nitrates (TW) or nitrates (MH) Dog (TW) or dog (MH) Pathophysiology (TW) or pathophysiology (MH) IVenodilation
48 Pathology
MITCHELL
Search Follo~~p
ET Al
Study
A follow-up study was performed during the second year of medical school in the pathology course, which focused on specific genetic disorders in a case discussion format. The same students who had used Grateful Med the previous spring were studied for their use of the system in pathology. Once again nonbilled access to MEDLINE was provided and they were invited to use Grateful Med to search other medical topics in other courses during a six week period. They were given a two-page handout describing their most common errors on using Grateful Med in biochemistry and some suggestions for overcoming those errors. The students were given six topics as part of case workups and invited to break into groups of four students to gather the requisite information and discuss the results. Since the questions had multiple subtopics. not all of which required literature searches, the student groups usually parcelled out the subparts and then met to organize the overall case discussion. Usually only one of the four students did a search on the group’s topic, although multiple students might search on subtopics. Even though the four students worked as a group to gather all of the case materials, the students usually performed the Grateful Med searching independently. Each student used an individual password and his/her searching behavior was analyzed on an individual basis. The traffic file data were analyzed with the same procedures as in the biochemistry study for basic parameters. In order to compare the errors and use of Grateful Med features during the two exercises (biochemistry and pathology) we calculated a summary score for each student to allow tracking students through time. We could not directly compare the results of the two class expertments because each student contributed a different number of searches to the total and thus the groups of searches were not independent. We compared the two experiments by: (1) calculating a mean score for the two experiments for each student, (2) calculating a difference in the mean score between experiment two (pathology) minus experiment one (biochemistry), and (3) testing the hypothesis that the difference in mean score is different from zero (although the hypothesis may be directional depending upon the assumptions) using the Wilcoxon signed rank test. For example. we calculate an error rate per student which is the proportion of searches with errors and test the hypothesis that the error rate declined from biochemistry to pathology. If the data was dichotomous. then McNemar’s test was used to compare proportions.
Analysis
oj’Failed
Searches
(Biochemistry)
Of the 208 searches in the subset. 20 retrieved over 300 citations, ranging from 309 to 13,842 hits. In nine of these searches, the student had entered a phrase on a single subject line, causing the search words to be ORed rather than ANDed together. These search failures can be attributed to a misunderstanding of the logical underpinnings of the Grateful Med form screen, compounded by
MEDICAL
STUDENTS
USING GRATEFUL
TABLE FAILED
SEARCH
ANALYSIS:
NO-HITS
49
MED
2
SEARCHES
IN BIOCHEMISTRY
(n = 188)
Error/problem
Number
Percent
No FlO used No errors MeSH elsewhere Forward slash error Logic error Too narrow: Review Misspelling Choice of term Redundant search Too narrow: English Too narrow: Title Too narrow: AIM
53 43 37 26 19 19 15 12 11 8 I 4
28 23 20 14 10 10 8 6 6 4 4 2
the Version 3 “enhancement” which permitted the user to remove the ORs which appear each time the space bar is pressed while on a subject line. This editing on the users’ part made no difference in the processing of the search; Grateful Med simply reinserted the ORs when the search was run. The remaining 11 searches retrieved over 300 items because the student had entered only a single concept. Overall, however, relatively few of the searches failed by retrieving over 300 citations. A far more common problem was the search which retrieved no articles at all. One hundred eighty-eight (188) of the searches in the subset were “zero hits” searches. Of these, 23% (43) contained no discernible errors or problems. On the remaining searches, the most common problems were the missed opportunities (see Table 2). Twenty-eight percent (28%) of the searches might have retrieved something if the student had searched using a MeSH term; and they would have quickly located the MeSH equivalent for their text word had they only pressed FIO. On 20% of the searches, one or more of the text words that the student chose had a MeSH equivalent, but it would not have been immediately accessible via FlO using their text word as an entry point. The MeSH term would have been in another part of the alphabetical list. For example, while many of the students were searching“nitroglycerin” for a physiology assignment, few of them made it to “glyceryl trinitrate in the MeSH list.” The third most common error (14% of the searches) was the use of the forward slash with a non-MeSH term. We did not want to teach beginning searchers to use the forward slash override on the Grateful Med form screen; however, the version of Grateful Med in use in 1988 (Version 3) did not have the complete MeSH on the disk. The biochemical disorders which the students were searching are commonly characterized by enzyme deficiencies; the MeSH headings for many of those enzymes were omitted from the MeSH disk, and thus had to be entered on the Grateful Med search form preceded by a forward slash.
SO
MITCHELL
ET AI
Misspellings of these terms were common along with the actual forward slash error. As with the >300 hits searches, logic errors accounted for some of the failures (10% of the searches), when the searcher entered a phrase on a single subject line, causing the words to be ORed rather than ANDed. This error (which usually causes too many hits) was caused by picking two or more terms which individually had zero hits. Some of the searches yielded nothing simply because they were overly restrictive. Most often this occurred when the searcher attempted limiting to only review articles (10% of the searches), although limiting to title words, English language, or AIM journals accounted for some of the “no hits” searches t2-4% of the searches). It is likely that the choice of MeSH term affected the outcome in 6% of the failed searches. Most often the searcher selected a specialty term like “cardiology” instead of the corresponding disease term”heart diseases.” Puthology
Search Follow-up
Study
A total of 446 searches were completed by 53 students during the follow-up study. Fifty percent of the searches were on the assigned genetics topics and 50% were on other topics. Because the assignment did not require each student to search independently, it was not surprising that only a subset of the students used Grateful Med. The most interesting data concerned comparisons between the errors in the entire set of searches (not just failed searches) and use of Grateful Med features during the two exercises (biochemistry and pathology). Eleven percent of the pathology searches had at least one error, while 20% of the biochemistry searches had at least one error, a significant decline in error rate per student. Data were analyzed to determine whether the students who made the most errors in biochemistry simply did not search the literature in the pathology exercise, but this was not statistically significant. However, there was strong significance (p < ,002) when one considers those students who searched and made errors in biochemistry and then consider their error rate in pathology; the error rate declined. Furthermore, a larger percentage of the searches in the pathology study used MeSH exclusively than during the biochemistry study. This decline in error rate may be partially attributable to the handout which contained a discussion of the most common error (entering a phrase on one line) and how to avoid it. Other routes of access to MEDLINE were not readily available to these students in 1988 so the decline in error rate was probably not from other independent searching. The staff of the health sciences library did report a considerable increase in student search requests during the six month hiatus, demonstrating an increased awareness and interest in computer mediated searching. However, a very few students would have been able to search independently and practice their search skills without Grateful Med codes.
MEDICAL
STUDENTS
USING
GRATEFUL
MED
51
We examined the change in specific types of errors in the entire set of searches. Whereas the trend appears to be an improvement from the previous experiment, there were no statistically significant differences between the specific types of errors in the two experiments. It appears that novice users can return to use the Grateful Med searching software after six months of disuse and recall the basic search techniques and correct use of the software. Most of the other end-user systems require frequent use to maintain low error rates. We did not do an in-depth study of failed searches in the pathology searches, but have some genera1 information on the magnitude of the problems. The overall magnitude of “zero hits” in the current file, non-author searches was 20%. The greatest difficulty arose because the student did not use the FlO key to select a MeSH term when an appropriate term might have improved retrieval. The percentage of searches retrieving greater than 300 citations had increased to 7%, attributed to the broader subject areas for the search topics compared to the previous exercise. This serves to emphasize that trainers must teach troubleshooting of searches with special attention to recovery from zero hits. When one compares the genetics searches versus the non-genetics searches, some interesting trends appear. We graded the searches on overall quality on a scale from 1 (excellent) to 5 (abysmal). The students had better search quality and fewer missed opportunities on genetics searches than when searching on other topics. They searched the backfiles more frequently on subjects other than genetics. As was found in the biochemistry study, the searches on nongenetics topics used more lines and more text words compared to the genetics topics. This could be interpreted as more complex searches on other topics or, alternatively, less focused search statements since MeSH was used less frequently when they searched independently on non-genetics topics. Overall, the students were doing very complex searches compared to other end-user studies (25,26) which showed a high percentage of single concept searches with most end-users, even after considerable experience. Figure 4 is a graph of individuals in the class who searched the literature in both biochemistry and pathology. There was no correlation between the number of searches each student performed in biochemistry and pathology. It was surprising that some individuals who did no independent searches in the first exercise were frequent searchers in the second exercise. Over the two studies, a large number of the students were frequent searchers. Developing a profile of attributes which characterize the best or the most frequent searchers is elusive. We analyzed the total (biochemistry plus pathology) number of searches, the total search time, the number of search sessions, the percentage of searches with no errors compared to the background statistics and previous training of the students as determined from the biochemistry questionnaire. Once again, there were no statistically significant correlations; all regression equations showed no significant predictors of performance. Of the four students with no prior computer experience and no independent searches in the biochemistry experiment, there was only one student of the group who still had not touched the computer independently after the pathology assignments.
m
Biochemistry
FIG. 4. This graph shows the number of independent searches done by each of I I.5students during the biochemistry exercise and the pathology exercise.
DISCUSSION
These studies lead to some interesting conclusions when compared to previous studies of end-user searching of literature databases. The students appear to be interested in searching independently, and a few search quite frequently. This phenomenon of a few highly motivated searchers is in concordance with other studies (9, 14, IS, 25, 27). A unique result of this work is the longitudinal follow-up which shows that the most frequent searchers in biochemistry were not the same people as the most frequent searchers in pathology. One of the surprises of our research was that the students took the opportunity to do searches on other genetics topics as well as on other medical topics. This may demonstrate the perception of the students that computer mediated literature searching is an efficient way to use their time. Furthermore, they were better prepared for their discussion sessions because they investigated the overlap between their assigned disease and similar diseases assigned to other students. The error analysis would contribute to the impression that students can effectively use the MeSH vocabulary if trained in the proper techniques. Furthermore, the ease of use of the FIO key to access the appropriate vocabulary is one key reason that the students can return to Grateful Med and use it with fewer errors after a six month hiatus. The continued improvements in Grateful Med will ensure that the software is easy to use.
MEDICALSTUDENTSUSINGGRATEFULMED
53
There are substantial practical applications for Grateful Med trainers which arise from our studies of errors and failed searches. We have made several changes in the content and emphasis in subsequent training sessions based upon the traffic file analysis. First, since logic errors appeared on 13% of the 208 failed searches (and 11% of the total 829 searches), additional emphasis was placed upon principles of Boolean logic and the implementation of Boolean operators on the Grateful Med form screen. Prior to the traffic file analysis, we did not belabor Boolean principles in training sessions, since it seemed that the Grateful Med form screen made Boolean logic rather transparent (an erroneous assumption). An extra effort was made to stress the value and uses of subheadings in subsequent training sessions. Even though subheadings were addressed in the initial training, they were still a source of confusion and accounted for many errors and missed opportunities. No systematic coding was attempted to classify various problems with subheadings, but there were many failed attempts to employ them. In addition, we began to highlight the difference between specialty and disease terms whenever the opportunity arose. Although this was not a common problem, it often made a critical difference in the outcome of the search when the wrong term was selected. Grateful Med programmers began to address this distinction in Version 4 through the use of scope notes on many of the specialty terms. With 30% of all searches resulting in zero hits, we strongly recommend that trainers provide novice searchers with some hints for recovering from a no hits search. The Found column incorporated into Grateful Med Version 4 and higher versions should help the user to recognize and correct overly specific searches; we have incorporated a sample search illustrating the use of the Found column in our training sessions. We also provide some general guidelines for broadening and narrowing searches through the addition and deletion of lines on the form screen. Beginning medical students can become relatively efficient and frequent users of Grateful Med to search MEDLINE. We have analyzed the major patterns of uses, errors, and patterns of behavior associated with student use of Grateful Med, although we have not studied the effectiveness of the searches. It appears that this group of students has some initial skills and the movtivation to use Grateful Med; it took remarkably little time from the traditional curriculum. However, the extent to which this skill is integrated into the daily practice of clinical medicine remains to be determined.
ACKNOWLEDGMENTS This work was supported in part by a grant from the Mizzou Alumni Association and with nonbilled codes granted by the National Library of Medicine.
MITCHELL
54
ET AL
REFERENCES I. DEROSA. D. A., MAST, T. A., DAWSON-SAUNDERS. B.. MAZUR. J.. RANSEY. D. E.. AND FOLSE. J. R. A study of the information-seeking skills of medical students and physician faculty. .I. Med. Educ. 58, 45 (1983). 17. NORTHRUP. D. E., MOORE-WEST, M.. SKIPPER, B.. AND T~AF. S. R. Characteristics of clinical information-searching: Investigation using critical incident technique. J. Med. Edrrc,. 58, 873 (1983). .q. STINSON. E. R.. AND MUELIXR. D. A. Survey of health professional!,’ informatIon habits and needs. Am. Med. Assoc. 243, No. 2. 140 (1980). 4. L. HARRIS AND ASSOCIATES “The Future of Information Systems for the Medical Sciences; A Study Conducted for the New York Academy of Medicine.” New York Academy of Medicine. New York, 1988. 5. COVELL, D. G.. UMAN. G. C.. AND MANNING, P. R. Information needs in office practice: Plre they being met? Ann. Infern. Med. 103, 596 (1985). 6. WELBORN, V., AND KUEHN. J. J. End-user programs in medical school libraries: A survey. Bull. Med. Lihr. Assoc. 76, No. 2. 137 (1988). 7. SOBEN, P., AND GREEN. E.W. GRATEFUL MED searching: A hospital library perspective. Med. Rqf. Scruiws Qnurteriy 7. No. 3. 35 (1988). 8. PORTER, D.. WIG~~N. R. S.. REIDELBACH, M. A.. BL.EICH. H. L.. AND SI.A~~. W. V. Selfservice computerized bibliographic retrieval: a comparison of Colleague and PaperChase. programs that search the MEDLINE data base. Camp. und Biomed. Rex. 21:488-501 ( 1988). 9. BOHHAM. M. D.. AND NELSON. L. L. An evaluation of four end-user systems for searching MEDLINE. Bull. Med. Lihr. Assoc. 76( 1):22-31 (1988). 111. MILLER, N.. KIRBY M.. AND ETHELDRA, T. MEDLINE on CD-ROM: end user searching in a medical school library. Med. Ref. Sew. Q. 7, No. 3. 1 (1988). II. SIMON, F. A. A comparison of two computer programs for zearching the medical literature. J. Med.
Educ.
63, No.
4. 331 (1988).
12. BURROUGHS. C. M. Clinicians’ satisfaction with Grateful Med: An exploratory study. B&L. Med. Libr. Assoc. 77, No. 1. 56 (1989). I3. SARACEVIC. T.. AND KANTOR. P. A study of information seeking and retrieving. 111.Searcher\. searches. and overlap. J. Am. SOC. fnf. Sri. 39. 197 (1988). 14. MCKIBBON, K. A.. WALKER. C. J., RYAN. N. C.. FI~ZGERAI.D. D.. AND HAYNES, K. B. .4 study of MEDLINE in clinical settings: Design and preliminary results. In “SCAMC Proceedings” (R. A. Greenes. Ed.). pp. 526-519. Computer Society Press. Los Angeles, CA. 1988. 15. HAYNES. R. B.. MCKIBBON. K. A.. WALKER, C. J., RYAN. N.. FITZ~;ERAI.D. D.. .+.lu~ RAFTSDEN, M. F. Online access to MEDLINE in clinical settings-A study of use and usefulness. Ann. Intern. Med. 112, 78 (1990). 16. PERKINS, D. N. The fingertip effect: How information-processing technology shapes thinking. Ed/cc. Rpx. pp. I l-17, (AugiSep 1985). 17. HAYNES. R. B., RAMSDEN, M., MCKIBBON, K. A.. WALKER. C. J.. AND RYAN. N. C. A review of medical education and medical informatics. Acad.Med. 64, No. 4, 207 (1989,. 18. ZAROUKIAN, M. H.. HAF, J. J.. AND ALFANO, M. A. MEDLINE computerized literature searching using GRATEFUL MED: A pilot instructional prqject for entering medical students. At,rrd. Med.. submitted. 19. BRADIGAN. P. S., AND MULARSKI. C. A. End-user searching in a medical school curriculum: An evaluated modular approach. Bull. Med. Lihr. .4.~so~~. 77. No. 4. 348 (1989). 20. BURROWS. S.. GINN. D. S.. LOVE. N.. AND WILLIAIIS. I‘. L. A strategy for curriculum integration of information skills instruction. Bull. Med. Lihr. Assoc. 77, No. 3. 245 (1989). 21. KIMMEL. S. Teaching third-year medical students to search MEDLINE. Med. Rtf Sew. Q. 8. No. 3. 69 (1989). 22. RODNICK, J. E., SIMRIN, S. M., YANG. M. G., AND ALTMAN, D. F. Teaching medical students to do bibliographic searching. J. Med. Educ. 63, 728 (1988). 23. PROUD. V. K., SCHMIDT. F. J.. JOHNSON, E. D.. AND MITCHELI, J. A. Teaching human genetics in biochemistry by computer literature searching. Am J. Hwn. Gencr. 44, 597 (1989).
MEDICAL
STUDENTS
USING
GRATEFUL
MED
55
MITCHELL, J. A., JOHNSON, E. D., AND PROUD, V. K. New thoughts about medical students as effective searchers of MEDLINE. Acad. Med. 65, No. 7, 434 (1990). 25. SEWELL, W., AND TEITELBAUM, S. Observations of end-users online searching behavior over eleven years. J. Am. Sm. Zn$ Sci. 37, No. 4, 234 (1986). 26. KIRBY, M., AND MILLER. N. MEDLINE searching on Colleague: reasons for failure or success of untrained end-users. Med. Ref. Serv. Q. 5, No. 3, 17 (1986). 27. WALKER, G. “End-user searching: A selection of the literature for 1983-1988.” An ERIC synthesis paper. ERIC Clearinghouse on Information Resources, Syracuse, NY, December 1988. 24.
