This article was downloaded by: [Georgian Court University] On: 10 March 2015, At: 16:56 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Medical Reference Services Quarterly Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/wmrs20
RELM: Developing a Serious Game to Teach Evidence-Based Medicine in an Academic Health Sciences Setting Ann Whitney Gleason
a
a
Health Sciences Library, Stony Brook University, Stony Brook, New York, USA Published online: 22 Jan 2015.
Click for updates To cite this article: Ann Whitney Gleason (2015) RELM: Developing a Serious Game to Teach EvidenceBased Medicine in an Academic Health Sciences Setting, Medical Reference Services Quarterly, 34:1, 17-28, DOI: 10.1080/02763869.2015.986709 To link to this article: http://dx.doi.org/10.1080/02763869.2015.986709
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Downloaded by [Georgian Court University] at 16:56 10 March 2015
Conditions of access and use can be found at http://www.tandfonline.com/page/termsand-conditions
Medical Reference Services Quarterly, 34(1):17–28, 2015 Published with license by Taylor & Francis ISSN: 0276-3869 print=1540-9597 online DOI: 10.1080/02763869.2015.986709
RELM: Developing a Serious Game to Teach Evidence-Based Medicine in an Academic Health Sciences Setting ANN WHITNEY GLEASON Downloaded by [Georgian Court University] at 16:56 10 March 2015
Health Sciences Library, Stony Brook University, Stony Brook, New York, USA
Gaming as a means of delivering online education continues to gain in popularity. Online games provide an engaging and enjoyable way of learning. Gaming is especially appropriate for case-based teaching, and provides a conducive environment for adult independent learning. With funding from the National Network of Libraries of Medicine, Pacific Northwest Region (NN= LM PNR), the University of Washington (UW) Health Sciences Library, and the UW School of Medicine are collaborating to create an interactive, self-paced online game that teaches players to employ the steps in practicing evidence-based medicine. The game encourages life-long learning and literacy skills and could be used for providing continuing medical education. KEYWORDS Academic health sciences libraries, evidence-based medicine, gaming, library education, medical education, serious games
# Ann Whitney Gleason Received: August 28, 2014; Revised: October 22, 2014; Accepted: October 25, 2014. A poster was also presented at the Medical Library Association Annual Conference in May 20, 2014, entitled ‘‘RELM’’ (Research and Evidence Literacy in Medicine): Exploring Online Gaming in Medical Instruction. Address correspondence to Ann Whitney Gleason, Health Sciences Library, Stony Brook University, Health Sciences Tower, Level 3, Room 142, Stony Brook, NY 11994. E-mail: [email protected] Color versions of one or more of the figures in the article can be found online at www. tandfonline.com/wmrs. 17
18
A. W. Gleason
Downloaded by [Georgian Court University] at 16:56 10 March 2015
INTRODUCTION The use of gaming technology in education has been gaining mainstream popularity over the last few years. In the library arena, public libraries have been the most enthusiastic adopters of this technology, pulling underserved teens into the library through gaming tournaments and other events.1 Studies have shown that these programs are popular. Young people who come into the library for gaming events tend to return and use other library services. Could gaming technology be useful in a higher education academic library setting? This was the question asked at the University of Washington Health Sciences Library. Initially, faculty were skeptical that gaming could offer value in a library serving primarily upper-division students and faculty. It was thought that undergraduates would most likely embrace these kinds of programs, but graduate students and faculty wouldn’t have time for anything beyond serious studies and research. However, after exploring the literature on the applications of gaming in health sciences, it was surprising to find that gaming technology is already being used extensively in medical education and that academic libraries are already using this innovative technology to enhance library programs. The RELM project was launched to introduce this highly interactive mode of learning to the teaching of evidence-based medicine (EBM) in the University of Washington Health Sciences Library.
BACKGROUND AND DEFINITIONS Djaouti and colleagues, in a chapter on classifying educational games published in a 2011 book on improving learning and motivation, define a ‘‘serious game’’ as ‘‘any piece of software that merges a non-entertaining purpose (serious) with a video game structure (game).’’2 This definition distinguishes a serious game from other electronic video games where the purpose is primarily recreational. Top-selling video game consoles include Nintendo, Microsoft Xbox 360, and Sony Playstation 3. Many games are manufactured for these platforms targeted to players of all ages. Some K-12 titles are labeled as ‘‘educational,’’ but the majority of console games are recreational offerings. Computer-based games also offer a wide variety of recreational titles as well as educational games, some of which are commonly available to students in computer labs or libraries at K-12 schools. Gaming in higher education is still at the cutting edge of this technology. The Higher Education Video Game Alliance (HEVGA), a newly formed group of institutions and organizations interested in promoting video game technology in higher education, met for the first time in November 2014 to work on promoting video game technology in learning environments for the 21st century.3
Downloaded by [Georgian Court University] at 16:56 10 March 2015
RELM: A Serious Game for Evidence-Based Medicine
19
Simulations are another category of ‘‘serious games’’ that tend to be more realistic and have uses beyond recreation in business and educational training programs. Simulations range from simple, case study video scenarios to complex computer simulations that interface with lifelike manikins. One example of an online simulation is found in Second Life (SL). SL is an online 3D virtual world where people play games, explore, and learn through their online ‘‘avatars.’’ The Nintendo Wii console is another gaming technology that enhances the typical game experience by using a remote control with accelerometer and motion sensors to make games more realistic. Microsoft’s Kinect is a similar product that uses cameras to simulate movement when playing games on the Xbox 360. Simulation technology has a huge potential for use in graduate-level medical education. Simulations have actually been used for many years for educational purposes including in business, aerospace, manufacturing, and military training. Simulations for medical education have been controversial, with complaints that it is too ‘‘impersonal’’ and not transferrable to real-life situations.4 Despite this controversy, simulations in medical education have become very common in medical schools as well as other health professional programs. Simulations have the capability to provide medical training for many learners at the same time, helping learners to achieve proficiency without being dependent on a clinical setting where real-life human subjects are not always available for study. Simulations enable medical students to practice interviewing and medical skills before encountering real human patients. Studies have shown that repeated simulation practice actually improves medical students’ responses when encountering a real-life situation.5 Simulations in medical education can take many forms. Role-playing involves playing the part of a medical practitioner or patient in a medical situation and acting out the interaction that may occur in practice. These interactions can be recorded, and students can be assessed on their performances. One drawback of this method of simulation is that students may be self-conscious in front of other students and instructors. Another method, computer-based simulation, allows students to practice medical situations as many times as they need at their own pace. Some common medical examples include the basics of history taking and physical examination, listening to heart and lung sounds, and problem-based learning modules where students practice clinical reasoning and problem solving.4 Video-based simulations are also common for teaching physician skills such as interviewing, bedside manner, and counseling techniques. They are also used to model the discussion of familiar medical conditions and situations that physicians typically encounter, again without having to practice on real-life patients. New manikin-based simulators are now being developed that are even more promising for medical education. Early simulators still in use include Resusci Anne for CPR education and Harvey for teaching cardiac examination skills. As the cost for hardware components continues to fall, more
Downloaded by [Georgian Court University] at 16:56 10 March 2015
20
A. W. Gleason
complex and realistic simulators are being developed to teach medical procedures such as IV insertion, endoscopic surgery, arthroscopy, and cardiac surgery.4 The use of simulations for these procedures allows students to practice invasive techniques on a manikin before transferring those skills to real-life patients and enhances the student’s ability to successfully diagnose illnesses with fewer diagnostic tests needed.5 The University of Washington Medical School has a very complex simulator at the Institute for Simulation and Interprofessional Studies (ISIS) where students can practice anesthesiology techniques, administer medications, and monitor vital signs, all on a realistic manikin that simulates body functioning and responses to medication .6 Virtual online simulations such as those found in Second Life may be useful for medical education in the future. Current examples include a heart murmur simulation that enables students to listen to different types of cardiovascular sounds and learn to diagnose conditions; a virtual medical center; the gene pool which has many levels of genetic educational materials; emergency preparedness training simulations; and a medical library where students can search PubMed and browse patient educational materials. Many virtual classes are now also held in Second Life as well. Several library sites are staffed by real librarians who help others to navigate this complex virtual world, providing reference information just like in the real world.7 Very popular several years ago, Second Life continues to have enthusiasts; however, the creation and maintenance of virtual sites in SL is dependent on the continuing interest and time investment of volunteers. Many interesting projects have been initiated only to be abandoned due to lack of maintenance. Another drawback of SL is that it is populated by many commercial and entertainment sites. It is easy for users to waste a lot of time wandering around unless they are guided to specific sites hosted by educational and professional organizations. The University of Pennsylvania has some helpful SL tips on their website at . The development of the Nintendo Wii console in 2006 revolutionized video game technology by introducing motion sensing through the use of an accelerometer and an infrared camera.8 Games played on the Wii are much more interactive and approach the feeling of virtual reality. Unfortunately, the development of games for this new technology is complex and not much has been developed in the educational field. The Wii Sports package is very popular and allows realistic play of sports such as golf, tennis, and bowling. Unfortunately, there is a risk of repetitive motion injuries to video games enthusiasts, who may not usually be very active people, and who may injure themselves through overly enthusiastic movements when playing Wii Sports.9 Interesting medical uses of Wii technology include the improvement of surgeon laparoscopic skills by playing Wii games10 and the rehabilitation of stroke victims with Wii consoles in their own homes, which could
Downloaded by [Georgian Court University] at 16:56 10 March 2015
RELM: A Serious Game for Evidence-Based Medicine
21
improve chances for full recovery of movement, since it can be introduced at an earlier stage than traditional physical therapy.11 In a 2011 systematic review of gaming, several benefits were found for the use of serious games in medical education.12 The entertainment factor of games was found to hold the player’s attention with improved performance noted as a result of sustained attention. Competition was found to improve a player’s engagement with learning. Feedback during game play is also essential to learning and helps make meaningful connections to the real world. Learning theory suggests that independent learning is an important factor for adult learning. Gaming is not only a highly independent form of learning, but it also builds on past experiences, which is another important element of adult learning. Games also provide goal-orientated learning when they introduce case-based, real world scenarios, which must be successfully completed in order to progress in the game. Case-based, serious gaming is now considered a viable addition to educational programs and should be evaluated further for its effectiveness in medical training, and, ultimately, patient outcomes. In a 2010 study comparing traditional case-based learning with game-based learning, no difference was found between the effectiveness of the two methods.13 In fact, participants in the game-based learning cohort reported greater satisfaction with their learning experience. The study concluded that game-based learning was a viable alternative for providing CME opportunities to physicians. In his book Everything Bad is Good for You: How Today’s Popular Culture is Actually Making Us Smarter, Steven Johnson argues that digital gaming actually improves problem-solving ability and even IQ.14 Gaming technology offers new opportunities for motivating learners who are both familiar with and enthusiastic about gaming. It also offers new ways of reaching students with a more interactive learning style. In a 2013 randomized controlled trial with third-year medical students, researchers found that students who participated in a game-based learning activity had better retention of the subject matter than those who learned through traditional methods.15 A widely accepted pedagogical strategy of creative use of gaming for teaching and learning has yet to be developed that would help teachers successfully integrate gaming into higher education classes. Game development requires a collaborative approach among educators, game developers, and subject specialists. Drawbacks to game development are the amount of work that goes into creating game-based learning modules as well as the expense, although inexpensive development frameworks are becoming more available.
DEVELOPING A SERIOUS GAME FOR MEDICAL EDUCATION At the University of Washington Health Sciences Library, librarians are collaborating with School of Medicine education staff to create a web-based,
Downloaded by [Georgian Court University] at 16:56 10 March 2015
22
A. W. Gleason
serious game to teach evidence-based medicine. The University of Washington (UW) School of Medicine is the sole medical school for a fivestate region: Washington, Wyoming, Alaska, Montana, and Idaho (WWAMI). More than 900 students and thousands of faculty work and learn in diverse rural and urban settings across the entire area. The UW Health Sciences Library serves as the medical library, providing access to e-resources at the point of care for clinicians, administrators, instructors, and students, throughout the WWAMI region. At a meeting between administrators from both the library and School of Medicine, the idea of building a game to teach new clinical faculty how to practice EBM emerged. Since librarians at the UW Health Sciences Library are always striving to find new ways of teaching in this challenging, geographically dispersed regional setting, this idea was met with enthusiasm. Online gaming can potentially extend the reach of librarians offering a way to teach faculty as well as students across the WWAMI region in an engaging and interactive way. Two major goals drove the initial idea to create an interactive, online game for teaching EBM. The first goal was to increase the research literacy knowledge, and strengthen the skills of the University of Washington clinical faculty across the five-state region. Stronger skills among practicing physicians who work with students will improve patient care while setting an example for lifelong learning. The second goal was to establish gaming as a successful continuing medical education (CME) mechanism for practicing physicians in distributed areas. EBM integrates the best research evidence with clinical expertise and the patient’s unique circumstances. EBM involves asking clinical questions and finding the best answers from the literature in five steps:16 1. 2. 3. 4. 5.
Asking answerable search questions Performing evidence searches Critically appraising the evidence Integrating clinical expertise Evaluating the effectiveness of the search
EBM uses the PICO (Patient, Intervention, Comparison, Outcome) method as a tool for formulating clearly focused clinical questions before searching the evidence literature.17 The game developed at UW focuses on the first two steps of EBM, asking and acquiring evidence. Research and Evidence Learning in Medicine (RELM) was envisioned as a serious game for teaching and learning EBM. It is an interactive, self-paced, online game which incorporates case-based learning. RELM features elements of social websites. Players must create a login account in order to save their scores in the online leaderboard to compete with other players. Competition plays an important part in motivating learners to continue playing the game. Players who successfully complete all of the game levels have the opportunity to submit new cases for others to play. Funding from the National Network of
Downloaded by [Georgian Court University] at 16:56 10 March 2015
RELM: A Serious Game for Evidence-Based Medicine
23
Libraries of Medicine, Pacific Northwest Region (NN=LM PNR) was received for this project after the initial program specifications were outlined. After reviewing the literature, the reasons for pursuing a gaming project to teach EBM at the University of Washington emerged, stimulating a desire to learn, reinforcing learning through doing, and providing a visual mode of learning. Studies, such as the 2013 randomized controlled trial with third-year medical students, suggest that serious gaming is likely to be an effective training method. Also, gaming is engaging and fun, both attributes that can be lacking in the traditional models employed when teaching library skills; this will hopefully encourage physician participation in the program. Through the development process, the RELM game evolved into a set of medical cases that players engage with, answering clinical questions and finding evidence-based answers that lead to the correct diagnosis and treatment of a patient. The game begins in a fictional clinic (see Figure 1) and players are asked to choose from a selection of rooms, which contain different levels of patient cases from easy to hard. There are three game levels: .
Easy: The case is a commonly encountered scenario. The player is penalized very little for guessing a treatment plan, is given hints, and is prompted with helpful search terms.
FIGURE 1 RELM login screen.
24 .
Downloaded by [Georgian Court University] at 16:56 10 March 2015
.
A. W. Gleason
Standard: The case is a harder scenario. The player is penalized more for guessing, not given many hints, and few search terms are provided. Hard: The case presents a difficult-to-diagnose scenario. The player is heavily penalized for guessing, awarded more points for winning, and offered no hints or search help.
Each RELM case starts with a PICO matching exercise and formulation of a clinical question. Players then read the case overview, look at the patient file with medical history information, and search the evidence literature using DynaMed, PubMed, or other databases. Based on the information gathered, players answer a series of clinical questions related to the case and then select a treatment plan for the patient. After each question is answered, a message with feedback on the answer is given to reinforce learning. The game is also timed. Players who finish under the bonus time limit earn extra points. Points are also awarded for each clinical question answered correctly. If enough questions are answered incorrectly, a message is displayed indicating that the patient was either admitted to the ER or switched to another physician. The player must then repeat the case. Players compete with one another for points on the game leaderboard. The RELM project was completed in three phases. In phase I, a prototype game was created by the UW team, working with a student game designer. The prototype was presented in May 2013 at the AAMC Western Group on Educational Affairs conference. Players at the conference were very enthusiastic about the idea of a game teaching evidence-based medicine. Playing for points was confirmed as a good motivator for getting people to participate in the game. Conference participants helped identify major enhancements necessary to create a production version of the game. The user interface had to be refined so that it would not distract from game play. Enhancements identified included: . . . .
The PICO matching exercise needed major improvements and each case required an integrated PICO exercise. The cases had to be more engaging so that players would be willing to stick with the game for longer periods. More interactivity needed to be built in without overwhelming players. Most participants at the conference presentation played the game on a tablet computer, so touch functionality had to be improved for mobile optimization.
Clearly, much more work needed to be done to produce a fully functional game. During phase II of the RELM project, in spring 2014, a physician consultant was hired by the UW Health Science Library to work with the librarians to create case-based scenarios for game play. Six scenarios were created for
Downloaded by [Georgian Court University] at 16:56 10 March 2015
RELM: A Serious Game for Evidence-Based Medicine
25
the initial game, focusing on multiple types of cases and with increasing difficulty. All cases involve posing evidence-based questions and deciding on patient treatments after reviewing the evidence presented in the literature. The cases require use of different information resources in order to broaden the physician’s exposure to evidence-based resources. Hiring an experienced physician to create cases for the game proved crucial to making the game more engaging and challenging for players. The most difficult part of phase II was formulating clinical questions from the cases so that the game could proceed through a series of questions and answers to an appropriate diagnosis or treatment plan. Cases presented in the game needed to have a clear and concrete solution in order to not be ambiguous and confusing to players. Translating patient cases into playable game modules proved much more difficult than originally anticipated. In phase III of the RELM project, an experienced game developer was hired to complete needed game enhancements identified in initial testing. The game was developed in JavaScript, leveraging current HTML5 and CSS3 technology. A graphic designer was also hired to make the game more visually appealing. Cases developed in Phase II were segmented into modules with the addition of supporting graphics such as x-rays (see Figure 2). Feedback on answers for each scenario was created for each clinical question posed so that players receive feedback immediately after answering a question. Development is on-going in order to finish six cases so that the game can be tested with volunteer users, both for feedback on game play and to complete usability testing. The most challenging part of phase III was incorporating the case educational material into the game without losing the entertainment factor. Initial testing of the phase III game with physicians and librarians identified several areas for improvement. While game navigation elements seemed intuitive to the game developers, who were very familiar with the game, test subjects requested better navigation cues and clearer instructions for seamless game play. Case modules needed clarification and removal of ambiguous elements. Feedback on each case question in the game modules also needed to be clearly presented before the player could move on in the case.
FUTURE DIRECTIONS Next steps for the RELM game project include: adding more appropriate feedback to case questions; adding more graphics to the game user interface as well as case scenarios; adding more cases; and testing the game with players from the UW community. Future directions for the game include introducing it to UW physicians as well as possibly expanding the program to UW medical students. The effectiveness of the game in teaching evidence-based medicine principles and in improving research literacy will be assessed when the game
A. W. Gleason
Downloaded by [Georgian Court University] at 16:56 10 March 2015
26
FIGURE 2 A sample scenario from one of the RELM cases.
is fully developed and ready for public release. Further study by the game development team on the use of gaming in medical education to improve patient outcomes will be explored as well as the possibility of using
RELM: A Serious Game for Evidence-Based Medicine
27
game-based learning for CME credit. Librarians interested in this project are welcomed to contact the author for more information and to receive notification when the game is publicly available.
ACKNOWLEDGMENTS
Downloaded by [Georgian Court University] at 16:56 10 March 2015
The author would like to acknowledge other key people who have contributed to envisioning and building the RELM game, including Sherry Dodson, Tania Bardyn, Julianne McNalley, Michael Campion, Adam Garrett, Brandon Turner, and Liz Pulanco.
REFERENCES 1. Nicholson, Scott. ‘‘Library Gaming Census Report.’’ American Libraries 40, no. 1=2 (2009): 44. 2. Djaouti, Damien, Julian Alvarez, and Jean-Pierre Jessel. ‘‘Classifying Serious Games: The G=P=S Model.’’ In Handbook of Research on Improving Learning and Motivation Through Educational Games: Multidisciplinary Approaches, edited by Patrick Felicia, 118–136. Waterford, Ireland: Waterford Institute of Technology, 2011. 3. Higher Education Video Game Alliance. ‘‘Higher Ed Games.’’ Accessed October 9, 2014. http://www.higheredgames.org/#news_and_events. 4. Issenberg, S. Barry, William C. McGaghie, Ian R. Hart et al. ‘‘Simulation Technology for Health Care Professional Skills Training and Assessment.’’ JAMA 282, no. 9 (1999): 861–866. 5. Lane, J. Lindsey, Stuart Slavin, and Amitai Ziv. ‘‘Simulation in Medical Education: A Review.’’ Simulation & Gaming 32, no. 3 (2001): 297–314. 6. Wright, Andrew S., Sara Kim, Brian Ross, and Carlos Pellegrini. ‘‘ISIS: The Institute for Simulation and Interprofessional Studies at the University of Washington.’’ Journal of Surgical Education 68, no. 1 (2011): 94–96. 7. Emoshea. ‘‘Second Life and Libraries: What’s the Point?’’ INFOBLOG. Accessed August 24, 2014. http://infoblog.infopeople.org/2007/02/20/second-life-andlibraries-whats-the-point/. 8. EDUCAUSE. ‘‘Seven Things You Should Know About Wii.’’ Accessed August 24, 2014. http://www.educause.edu/library/resources/7-things-you-should-knowabout-wii. 9. Robinson, Richard J., Dominic A. Barron, Andrew J. Grainger, and Ramakrishnan Venkatesh. ‘‘Wii Knee.’’ Emergency Radiology 15, no. 4 (2008): 255–257. 10. Badurdeen, Shiraz, Omar Abdul-Samad, Giles Story, Clare Wilson, Sue Down, and Adrian Harris. ‘‘Nintendo Wii Video-Gaming Ability Predicts Laparoscopic Skill.’’ Surgical Endoscopy 24, no. 8 (2010): 1824–1828. 11. Saposnik, Gustavo, Robert Teasell, Muhammad Mamdani et al. ‘‘Effectiveness of Virtual Reality Using Wii Gaming Technology in Stroke Rehabilitation a Pilot Randomized Clinical Trial and Proof of Principle.’’ Stroke 41, no. 7 (2010): 1477–1484. 12. de Wit-Zuurendonk, Laura D., and S. Guid Oei. ‘‘Serious Gaming in Women’s Health Care.’’ BJOG: An International Journal of Obstetrics & Gynaecology 118, no. s3 (2011): 17–21.
Downloaded by [Georgian Court University] at 16:56 10 March 2015
28
A. W. Gleason
13. Telner, Deanna, Maja Bujas-Bobanovic, David Chan et al. ‘‘Game-Based Versus Traditional Case-Based Learning: Comparing Effectiveness in Stroke Continuing Medical Education.’’ Canadian Family Physician 56, no. 9 (2010): e345–351. 14. Johnson, Steven. Everything Bad Is Good for You: How Today’s Popular Culture is Actually Making Us Smarter. New York: Penguin, 2006. 15. Boeker, Martin, Peter Andel, Werner Vach, and Alexander Frankenschmidt. ‘‘Game-Based E-Learning Is More Effective than a Conventional Instructional Method: A Randomized Controlled Trial with Third-Year Medical Students.’’ PloS one 8, no. 12 (2013): e82328. 16. Straus, Sharon E., W. Scott Richardson, Paul Glasziou, and R. Brian Haynes. Evidence-Based Medicine: How to Practice and Teach EBM. Edinburgh: Elsevier= Churchill Livingstone, 2005. 17. Richardson, W. Scott, Mark C. Wilson, Jim Nishikawa, and Robert S. Hayward. ‘‘The Well-Built Clinical Question: A Key to Evidence-Based Decisions.’’ ACP J Club 123, no. 3 (1995): A12–13.
ABOUT THE AUTHOR Ann Whitney Gleason, MLIS ([email protected]) is Head of the Health Sciences Library, Stony Brook University, Health Sciences Tower, Level 3, Room 142, Stony Brook, NY 11994. At the time this article was written, the author was Associate Director for Resources and Systems, University of Washington Health Sciences Library.
Medical Reference Services Quarterly Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/wmrs20
RELM: Developing a Serious Game to Teach Evidence-Based Medicine in an Academic Health Sciences Setting Ann Whitney Gleason
a
a
Health Sciences Library, Stony Brook University, Stony Brook, New York, USA Published online: 22 Jan 2015.
Click for updates To cite this article: Ann Whitney Gleason (2015) RELM: Developing a Serious Game to Teach EvidenceBased Medicine in an Academic Health Sciences Setting, Medical Reference Services Quarterly, 34:1, 17-28, DOI: 10.1080/02763869.2015.986709 To link to this article: http://dx.doi.org/10.1080/02763869.2015.986709
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Downloaded by [Georgian Court University] at 16:56 10 March 2015
Conditions of access and use can be found at http://www.tandfonline.com/page/termsand-conditions
Medical Reference Services Quarterly, 34(1):17–28, 2015 Published with license by Taylor & Francis ISSN: 0276-3869 print=1540-9597 online DOI: 10.1080/02763869.2015.986709
RELM: Developing a Serious Game to Teach Evidence-Based Medicine in an Academic Health Sciences Setting ANN WHITNEY GLEASON Downloaded by [Georgian Court University] at 16:56 10 March 2015
Health Sciences Library, Stony Brook University, Stony Brook, New York, USA
Gaming as a means of delivering online education continues to gain in popularity. Online games provide an engaging and enjoyable way of learning. Gaming is especially appropriate for case-based teaching, and provides a conducive environment for adult independent learning. With funding from the National Network of Libraries of Medicine, Pacific Northwest Region (NN= LM PNR), the University of Washington (UW) Health Sciences Library, and the UW School of Medicine are collaborating to create an interactive, self-paced online game that teaches players to employ the steps in practicing evidence-based medicine. The game encourages life-long learning and literacy skills and could be used for providing continuing medical education. KEYWORDS Academic health sciences libraries, evidence-based medicine, gaming, library education, medical education, serious games
# Ann Whitney Gleason Received: August 28, 2014; Revised: October 22, 2014; Accepted: October 25, 2014. A poster was also presented at the Medical Library Association Annual Conference in May 20, 2014, entitled ‘‘RELM’’ (Research and Evidence Literacy in Medicine): Exploring Online Gaming in Medical Instruction. Address correspondence to Ann Whitney Gleason, Health Sciences Library, Stony Brook University, Health Sciences Tower, Level 3, Room 142, Stony Brook, NY 11994. E-mail: [email protected] Color versions of one or more of the figures in the article can be found online at www. tandfonline.com/wmrs. 17
18
A. W. Gleason
Downloaded by [Georgian Court University] at 16:56 10 March 2015
INTRODUCTION The use of gaming technology in education has been gaining mainstream popularity over the last few years. In the library arena, public libraries have been the most enthusiastic adopters of this technology, pulling underserved teens into the library through gaming tournaments and other events.1 Studies have shown that these programs are popular. Young people who come into the library for gaming events tend to return and use other library services. Could gaming technology be useful in a higher education academic library setting? This was the question asked at the University of Washington Health Sciences Library. Initially, faculty were skeptical that gaming could offer value in a library serving primarily upper-division students and faculty. It was thought that undergraduates would most likely embrace these kinds of programs, but graduate students and faculty wouldn’t have time for anything beyond serious studies and research. However, after exploring the literature on the applications of gaming in health sciences, it was surprising to find that gaming technology is already being used extensively in medical education and that academic libraries are already using this innovative technology to enhance library programs. The RELM project was launched to introduce this highly interactive mode of learning to the teaching of evidence-based medicine (EBM) in the University of Washington Health Sciences Library.
BACKGROUND AND DEFINITIONS Djaouti and colleagues, in a chapter on classifying educational games published in a 2011 book on improving learning and motivation, define a ‘‘serious game’’ as ‘‘any piece of software that merges a non-entertaining purpose (serious) with a video game structure (game).’’2 This definition distinguishes a serious game from other electronic video games where the purpose is primarily recreational. Top-selling video game consoles include Nintendo, Microsoft Xbox 360, and Sony Playstation 3. Many games are manufactured for these platforms targeted to players of all ages. Some K-12 titles are labeled as ‘‘educational,’’ but the majority of console games are recreational offerings. Computer-based games also offer a wide variety of recreational titles as well as educational games, some of which are commonly available to students in computer labs or libraries at K-12 schools. Gaming in higher education is still at the cutting edge of this technology. The Higher Education Video Game Alliance (HEVGA), a newly formed group of institutions and organizations interested in promoting video game technology in higher education, met for the first time in November 2014 to work on promoting video game technology in learning environments for the 21st century.3
Downloaded by [Georgian Court University] at 16:56 10 March 2015
RELM: A Serious Game for Evidence-Based Medicine
19
Simulations are another category of ‘‘serious games’’ that tend to be more realistic and have uses beyond recreation in business and educational training programs. Simulations range from simple, case study video scenarios to complex computer simulations that interface with lifelike manikins. One example of an online simulation is found in Second Life (SL). SL is an online 3D virtual world where people play games, explore, and learn through their online ‘‘avatars.’’ The Nintendo Wii console is another gaming technology that enhances the typical game experience by using a remote control with accelerometer and motion sensors to make games more realistic. Microsoft’s Kinect is a similar product that uses cameras to simulate movement when playing games on the Xbox 360. Simulation technology has a huge potential for use in graduate-level medical education. Simulations have actually been used for many years for educational purposes including in business, aerospace, manufacturing, and military training. Simulations for medical education have been controversial, with complaints that it is too ‘‘impersonal’’ and not transferrable to real-life situations.4 Despite this controversy, simulations in medical education have become very common in medical schools as well as other health professional programs. Simulations have the capability to provide medical training for many learners at the same time, helping learners to achieve proficiency without being dependent on a clinical setting where real-life human subjects are not always available for study. Simulations enable medical students to practice interviewing and medical skills before encountering real human patients. Studies have shown that repeated simulation practice actually improves medical students’ responses when encountering a real-life situation.5 Simulations in medical education can take many forms. Role-playing involves playing the part of a medical practitioner or patient in a medical situation and acting out the interaction that may occur in practice. These interactions can be recorded, and students can be assessed on their performances. One drawback of this method of simulation is that students may be self-conscious in front of other students and instructors. Another method, computer-based simulation, allows students to practice medical situations as many times as they need at their own pace. Some common medical examples include the basics of history taking and physical examination, listening to heart and lung sounds, and problem-based learning modules where students practice clinical reasoning and problem solving.4 Video-based simulations are also common for teaching physician skills such as interviewing, bedside manner, and counseling techniques. They are also used to model the discussion of familiar medical conditions and situations that physicians typically encounter, again without having to practice on real-life patients. New manikin-based simulators are now being developed that are even more promising for medical education. Early simulators still in use include Resusci Anne for CPR education and Harvey for teaching cardiac examination skills. As the cost for hardware components continues to fall, more
Downloaded by [Georgian Court University] at 16:56 10 March 2015
20
A. W. Gleason
complex and realistic simulators are being developed to teach medical procedures such as IV insertion, endoscopic surgery, arthroscopy, and cardiac surgery.4 The use of simulations for these procedures allows students to practice invasive techniques on a manikin before transferring those skills to real-life patients and enhances the student’s ability to successfully diagnose illnesses with fewer diagnostic tests needed.5 The University of Washington Medical School has a very complex simulator at the Institute for Simulation and Interprofessional Studies (ISIS) where students can practice anesthesiology techniques, administer medications, and monitor vital signs, all on a realistic manikin that simulates body functioning and responses to medication .6 Virtual online simulations such as those found in Second Life may be useful for medical education in the future. Current examples include a heart murmur simulation that enables students to listen to different types of cardiovascular sounds and learn to diagnose conditions; a virtual medical center; the gene pool which has many levels of genetic educational materials; emergency preparedness training simulations; and a medical library where students can search PubMed and browse patient educational materials. Many virtual classes are now also held in Second Life as well. Several library sites are staffed by real librarians who help others to navigate this complex virtual world, providing reference information just like in the real world.7 Very popular several years ago, Second Life continues to have enthusiasts; however, the creation and maintenance of virtual sites in SL is dependent on the continuing interest and time investment of volunteers. Many interesting projects have been initiated only to be abandoned due to lack of maintenance. Another drawback of SL is that it is populated by many commercial and entertainment sites. It is easy for users to waste a lot of time wandering around unless they are guided to specific sites hosted by educational and professional organizations. The University of Pennsylvania has some helpful SL tips on their website at . The development of the Nintendo Wii console in 2006 revolutionized video game technology by introducing motion sensing through the use of an accelerometer and an infrared camera.8 Games played on the Wii are much more interactive and approach the feeling of virtual reality. Unfortunately, the development of games for this new technology is complex and not much has been developed in the educational field. The Wii Sports package is very popular and allows realistic play of sports such as golf, tennis, and bowling. Unfortunately, there is a risk of repetitive motion injuries to video games enthusiasts, who may not usually be very active people, and who may injure themselves through overly enthusiastic movements when playing Wii Sports.9 Interesting medical uses of Wii technology include the improvement of surgeon laparoscopic skills by playing Wii games10 and the rehabilitation of stroke victims with Wii consoles in their own homes, which could
Downloaded by [Georgian Court University] at 16:56 10 March 2015
RELM: A Serious Game for Evidence-Based Medicine
21
improve chances for full recovery of movement, since it can be introduced at an earlier stage than traditional physical therapy.11 In a 2011 systematic review of gaming, several benefits were found for the use of serious games in medical education.12 The entertainment factor of games was found to hold the player’s attention with improved performance noted as a result of sustained attention. Competition was found to improve a player’s engagement with learning. Feedback during game play is also essential to learning and helps make meaningful connections to the real world. Learning theory suggests that independent learning is an important factor for adult learning. Gaming is not only a highly independent form of learning, but it also builds on past experiences, which is another important element of adult learning. Games also provide goal-orientated learning when they introduce case-based, real world scenarios, which must be successfully completed in order to progress in the game. Case-based, serious gaming is now considered a viable addition to educational programs and should be evaluated further for its effectiveness in medical training, and, ultimately, patient outcomes. In a 2010 study comparing traditional case-based learning with game-based learning, no difference was found between the effectiveness of the two methods.13 In fact, participants in the game-based learning cohort reported greater satisfaction with their learning experience. The study concluded that game-based learning was a viable alternative for providing CME opportunities to physicians. In his book Everything Bad is Good for You: How Today’s Popular Culture is Actually Making Us Smarter, Steven Johnson argues that digital gaming actually improves problem-solving ability and even IQ.14 Gaming technology offers new opportunities for motivating learners who are both familiar with and enthusiastic about gaming. It also offers new ways of reaching students with a more interactive learning style. In a 2013 randomized controlled trial with third-year medical students, researchers found that students who participated in a game-based learning activity had better retention of the subject matter than those who learned through traditional methods.15 A widely accepted pedagogical strategy of creative use of gaming for teaching and learning has yet to be developed that would help teachers successfully integrate gaming into higher education classes. Game development requires a collaborative approach among educators, game developers, and subject specialists. Drawbacks to game development are the amount of work that goes into creating game-based learning modules as well as the expense, although inexpensive development frameworks are becoming more available.
DEVELOPING A SERIOUS GAME FOR MEDICAL EDUCATION At the University of Washington Health Sciences Library, librarians are collaborating with School of Medicine education staff to create a web-based,
Downloaded by [Georgian Court University] at 16:56 10 March 2015
22
A. W. Gleason
serious game to teach evidence-based medicine. The University of Washington (UW) School of Medicine is the sole medical school for a fivestate region: Washington, Wyoming, Alaska, Montana, and Idaho (WWAMI). More than 900 students and thousands of faculty work and learn in diverse rural and urban settings across the entire area. The UW Health Sciences Library serves as the medical library, providing access to e-resources at the point of care for clinicians, administrators, instructors, and students, throughout the WWAMI region. At a meeting between administrators from both the library and School of Medicine, the idea of building a game to teach new clinical faculty how to practice EBM emerged. Since librarians at the UW Health Sciences Library are always striving to find new ways of teaching in this challenging, geographically dispersed regional setting, this idea was met with enthusiasm. Online gaming can potentially extend the reach of librarians offering a way to teach faculty as well as students across the WWAMI region in an engaging and interactive way. Two major goals drove the initial idea to create an interactive, online game for teaching EBM. The first goal was to increase the research literacy knowledge, and strengthen the skills of the University of Washington clinical faculty across the five-state region. Stronger skills among practicing physicians who work with students will improve patient care while setting an example for lifelong learning. The second goal was to establish gaming as a successful continuing medical education (CME) mechanism for practicing physicians in distributed areas. EBM integrates the best research evidence with clinical expertise and the patient’s unique circumstances. EBM involves asking clinical questions and finding the best answers from the literature in five steps:16 1. 2. 3. 4. 5.
Asking answerable search questions Performing evidence searches Critically appraising the evidence Integrating clinical expertise Evaluating the effectiveness of the search
EBM uses the PICO (Patient, Intervention, Comparison, Outcome) method as a tool for formulating clearly focused clinical questions before searching the evidence literature.17 The game developed at UW focuses on the first two steps of EBM, asking and acquiring evidence. Research and Evidence Learning in Medicine (RELM) was envisioned as a serious game for teaching and learning EBM. It is an interactive, self-paced, online game which incorporates case-based learning. RELM features elements of social websites. Players must create a login account in order to save their scores in the online leaderboard to compete with other players. Competition plays an important part in motivating learners to continue playing the game. Players who successfully complete all of the game levels have the opportunity to submit new cases for others to play. Funding from the National Network of
Downloaded by [Georgian Court University] at 16:56 10 March 2015
RELM: A Serious Game for Evidence-Based Medicine
23
Libraries of Medicine, Pacific Northwest Region (NN=LM PNR) was received for this project after the initial program specifications were outlined. After reviewing the literature, the reasons for pursuing a gaming project to teach EBM at the University of Washington emerged, stimulating a desire to learn, reinforcing learning through doing, and providing a visual mode of learning. Studies, such as the 2013 randomized controlled trial with third-year medical students, suggest that serious gaming is likely to be an effective training method. Also, gaming is engaging and fun, both attributes that can be lacking in the traditional models employed when teaching library skills; this will hopefully encourage physician participation in the program. Through the development process, the RELM game evolved into a set of medical cases that players engage with, answering clinical questions and finding evidence-based answers that lead to the correct diagnosis and treatment of a patient. The game begins in a fictional clinic (see Figure 1) and players are asked to choose from a selection of rooms, which contain different levels of patient cases from easy to hard. There are three game levels: .
Easy: The case is a commonly encountered scenario. The player is penalized very little for guessing a treatment plan, is given hints, and is prompted with helpful search terms.
FIGURE 1 RELM login screen.
24 .
Downloaded by [Georgian Court University] at 16:56 10 March 2015
.
A. W. Gleason
Standard: The case is a harder scenario. The player is penalized more for guessing, not given many hints, and few search terms are provided. Hard: The case presents a difficult-to-diagnose scenario. The player is heavily penalized for guessing, awarded more points for winning, and offered no hints or search help.
Each RELM case starts with a PICO matching exercise and formulation of a clinical question. Players then read the case overview, look at the patient file with medical history information, and search the evidence literature using DynaMed, PubMed, or other databases. Based on the information gathered, players answer a series of clinical questions related to the case and then select a treatment plan for the patient. After each question is answered, a message with feedback on the answer is given to reinforce learning. The game is also timed. Players who finish under the bonus time limit earn extra points. Points are also awarded for each clinical question answered correctly. If enough questions are answered incorrectly, a message is displayed indicating that the patient was either admitted to the ER or switched to another physician. The player must then repeat the case. Players compete with one another for points on the game leaderboard. The RELM project was completed in three phases. In phase I, a prototype game was created by the UW team, working with a student game designer. The prototype was presented in May 2013 at the AAMC Western Group on Educational Affairs conference. Players at the conference were very enthusiastic about the idea of a game teaching evidence-based medicine. Playing for points was confirmed as a good motivator for getting people to participate in the game. Conference participants helped identify major enhancements necessary to create a production version of the game. The user interface had to be refined so that it would not distract from game play. Enhancements identified included: . . . .
The PICO matching exercise needed major improvements and each case required an integrated PICO exercise. The cases had to be more engaging so that players would be willing to stick with the game for longer periods. More interactivity needed to be built in without overwhelming players. Most participants at the conference presentation played the game on a tablet computer, so touch functionality had to be improved for mobile optimization.
Clearly, much more work needed to be done to produce a fully functional game. During phase II of the RELM project, in spring 2014, a physician consultant was hired by the UW Health Science Library to work with the librarians to create case-based scenarios for game play. Six scenarios were created for
Downloaded by [Georgian Court University] at 16:56 10 March 2015
RELM: A Serious Game for Evidence-Based Medicine
25
the initial game, focusing on multiple types of cases and with increasing difficulty. All cases involve posing evidence-based questions and deciding on patient treatments after reviewing the evidence presented in the literature. The cases require use of different information resources in order to broaden the physician’s exposure to evidence-based resources. Hiring an experienced physician to create cases for the game proved crucial to making the game more engaging and challenging for players. The most difficult part of phase II was formulating clinical questions from the cases so that the game could proceed through a series of questions and answers to an appropriate diagnosis or treatment plan. Cases presented in the game needed to have a clear and concrete solution in order to not be ambiguous and confusing to players. Translating patient cases into playable game modules proved much more difficult than originally anticipated. In phase III of the RELM project, an experienced game developer was hired to complete needed game enhancements identified in initial testing. The game was developed in JavaScript, leveraging current HTML5 and CSS3 technology. A graphic designer was also hired to make the game more visually appealing. Cases developed in Phase II were segmented into modules with the addition of supporting graphics such as x-rays (see Figure 2). Feedback on answers for each scenario was created for each clinical question posed so that players receive feedback immediately after answering a question. Development is on-going in order to finish six cases so that the game can be tested with volunteer users, both for feedback on game play and to complete usability testing. The most challenging part of phase III was incorporating the case educational material into the game without losing the entertainment factor. Initial testing of the phase III game with physicians and librarians identified several areas for improvement. While game navigation elements seemed intuitive to the game developers, who were very familiar with the game, test subjects requested better navigation cues and clearer instructions for seamless game play. Case modules needed clarification and removal of ambiguous elements. Feedback on each case question in the game modules also needed to be clearly presented before the player could move on in the case.
FUTURE DIRECTIONS Next steps for the RELM game project include: adding more appropriate feedback to case questions; adding more graphics to the game user interface as well as case scenarios; adding more cases; and testing the game with players from the UW community. Future directions for the game include introducing it to UW physicians as well as possibly expanding the program to UW medical students. The effectiveness of the game in teaching evidence-based medicine principles and in improving research literacy will be assessed when the game
A. W. Gleason
Downloaded by [Georgian Court University] at 16:56 10 March 2015
26
FIGURE 2 A sample scenario from one of the RELM cases.
is fully developed and ready for public release. Further study by the game development team on the use of gaming in medical education to improve patient outcomes will be explored as well as the possibility of using
RELM: A Serious Game for Evidence-Based Medicine
27
game-based learning for CME credit. Librarians interested in this project are welcomed to contact the author for more information and to receive notification when the game is publicly available.
ACKNOWLEDGMENTS
Downloaded by [Georgian Court University] at 16:56 10 March 2015
The author would like to acknowledge other key people who have contributed to envisioning and building the RELM game, including Sherry Dodson, Tania Bardyn, Julianne McNalley, Michael Campion, Adam Garrett, Brandon Turner, and Liz Pulanco.
REFERENCES 1. Nicholson, Scott. ‘‘Library Gaming Census Report.’’ American Libraries 40, no. 1=2 (2009): 44. 2. Djaouti, Damien, Julian Alvarez, and Jean-Pierre Jessel. ‘‘Classifying Serious Games: The G=P=S Model.’’ In Handbook of Research on Improving Learning and Motivation Through Educational Games: Multidisciplinary Approaches, edited by Patrick Felicia, 118–136. Waterford, Ireland: Waterford Institute of Technology, 2011. 3. Higher Education Video Game Alliance. ‘‘Higher Ed Games.’’ Accessed October 9, 2014. http://www.higheredgames.org/#news_and_events. 4. Issenberg, S. Barry, William C. McGaghie, Ian R. Hart et al. ‘‘Simulation Technology for Health Care Professional Skills Training and Assessment.’’ JAMA 282, no. 9 (1999): 861–866. 5. Lane, J. Lindsey, Stuart Slavin, and Amitai Ziv. ‘‘Simulation in Medical Education: A Review.’’ Simulation & Gaming 32, no. 3 (2001): 297–314. 6. Wright, Andrew S., Sara Kim, Brian Ross, and Carlos Pellegrini. ‘‘ISIS: The Institute for Simulation and Interprofessional Studies at the University of Washington.’’ Journal of Surgical Education 68, no. 1 (2011): 94–96. 7. Emoshea. ‘‘Second Life and Libraries: What’s the Point?’’ INFOBLOG. Accessed August 24, 2014. http://infoblog.infopeople.org/2007/02/20/second-life-andlibraries-whats-the-point/. 8. EDUCAUSE. ‘‘Seven Things You Should Know About Wii.’’ Accessed August 24, 2014. http://www.educause.edu/library/resources/7-things-you-should-knowabout-wii. 9. Robinson, Richard J., Dominic A. Barron, Andrew J. Grainger, and Ramakrishnan Venkatesh. ‘‘Wii Knee.’’ Emergency Radiology 15, no. 4 (2008): 255–257. 10. Badurdeen, Shiraz, Omar Abdul-Samad, Giles Story, Clare Wilson, Sue Down, and Adrian Harris. ‘‘Nintendo Wii Video-Gaming Ability Predicts Laparoscopic Skill.’’ Surgical Endoscopy 24, no. 8 (2010): 1824–1828. 11. Saposnik, Gustavo, Robert Teasell, Muhammad Mamdani et al. ‘‘Effectiveness of Virtual Reality Using Wii Gaming Technology in Stroke Rehabilitation a Pilot Randomized Clinical Trial and Proof of Principle.’’ Stroke 41, no. 7 (2010): 1477–1484. 12. de Wit-Zuurendonk, Laura D., and S. Guid Oei. ‘‘Serious Gaming in Women’s Health Care.’’ BJOG: An International Journal of Obstetrics & Gynaecology 118, no. s3 (2011): 17–21.
Downloaded by [Georgian Court University] at 16:56 10 March 2015
28
A. W. Gleason
13. Telner, Deanna, Maja Bujas-Bobanovic, David Chan et al. ‘‘Game-Based Versus Traditional Case-Based Learning: Comparing Effectiveness in Stroke Continuing Medical Education.’’ Canadian Family Physician 56, no. 9 (2010): e345–351. 14. Johnson, Steven. Everything Bad Is Good for You: How Today’s Popular Culture is Actually Making Us Smarter. New York: Penguin, 2006. 15. Boeker, Martin, Peter Andel, Werner Vach, and Alexander Frankenschmidt. ‘‘Game-Based E-Learning Is More Effective than a Conventional Instructional Method: A Randomized Controlled Trial with Third-Year Medical Students.’’ PloS one 8, no. 12 (2013): e82328. 16. Straus, Sharon E., W. Scott Richardson, Paul Glasziou, and R. Brian Haynes. Evidence-Based Medicine: How to Practice and Teach EBM. Edinburgh: Elsevier= Churchill Livingstone, 2005. 17. Richardson, W. Scott, Mark C. Wilson, Jim Nishikawa, and Robert S. Hayward. ‘‘The Well-Built Clinical Question: A Key to Evidence-Based Decisions.’’ ACP J Club 123, no. 3 (1995): A12–13.
ABOUT THE AUTHOR Ann Whitney Gleason, MLIS ([email protected]) is Head of the Health Sciences Library, Stony Brook University, Health Sciences Tower, Level 3, Room 142, Stony Brook, NY 11994. At the time this article was written, the author was Associate Director for Resources and Systems, University of Washington Health Sciences Library.
