ORIGINAL REPORTS
Design, Realization, and First Validation of an Immersive Web-Based Virtual Patient Simulator for Training Clinical Decisions in Surgery Robert Kleinert, MD,* Nadine Heiermann, MD,* Roger Wahba, MD,* De-Huan Chang, MD,*,† Arnulf H. Hölscher, MD, PhD,* and Dirk L. Stippel, MD, PhD* *
Department of General, Visceral and Cancer Surgery of the University of Cologne, Center of Integrated Oncology of the University of Cologne, Transplant Center Cologne, Cologne, Germany; and †Department of Radiology, University of Cologne, Transplant Center Cologne, Cologne, Germany BACKGROUND: Immersive patient simulators (IPS) allow
an illusionary immersion into a synthetic world where the user can freely navigate through a 3-dimensional environment similar to computer games. Playful learning with IPS allows internalization of medical workflows without harming real patients. Ideally, IPS show high student acceptance and can have positive effect on knowledge gain. Development of IPS with high technical quality is resource intensive. Therefore most of the “high-fidelity” IPS are commercially driven. Usage of IPS in the daily curriculum is still rare. There is no academic-driven simulator that is freely accessible to every student and combines high immersion grade with a profound amount of medical content. AIM: Therefore it was our aim to develop an academic-
driven IPS prototype that is free to use and combines a high immersion grade with profound medical content. In addition, a first validation of the prototype was conducted. METHODS: The conceptual design included definition of
the following parameters: amount of curricular content, grade of technical quality, availability, and level of validation. A preliminary validation was done with 25 students. Studentsʼ opinion about acceptance was evaluated by a Likert-scale questionnaire. Effect on knowledge gain was determined by testing concordance and predictive validity. RESULTS: A custom-made simulator prototype (Artificial learning interface for clinical education [ALICE]) displays a virtual clinic environment that can be explored from a firstperson view similar to a video game. By controlling an
Correspondence: Inquiries to Robert Kleinert, MD, Department of General, Visceral and Cancer Surgery of the University of Cologne, Center of Integrated Oncology of the University of Cologne, Transplant Center Cologne, Kerpener Straße 62, 50937 Cologne, Germany; fax: (221) 478-6258; e-mail: [email protected]
avatar, the user navigates through the environment, is able to treat virtual patients, and faces the consequence of different decisions. ALICE showed high studentsʼ acceptance. There was positive correlation for concordance validity and predictive validity. Simulator usage had positive effect on reproduction of trained content and declarative knowledge. CONCLUSIONS: We successfully developed a university-
based, IPS prototype (ALICE) with profound medical content. ALICE is a nonprofit simulator, easy to use, and showed high studentsʼ acceptance; thus it potentially provides an additional tool for supporting student teaching C 2015 in the daily clinical curriculum. ( J Surg ]:]]]-]]]. J Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: ALICE, immersive patient simulators, web-
based learning, validity, immersion, procedural knowledge COMPETENCIES: Medical
Knowledge, Practice-Based
Learning and Improvement
INTRODUCTION e-Learning is an integrative part of medical education and meets the requirements of the current student generation (“digital natives”). Internet-based e-learning programs enable time-and location-independent learning with individual pace and repetitions and are nowadays mainly used for supportive teaching of theoretical (declarative) knowledge in preclinical and clinical education.1 Clinical education is characterized by the transfer of declarative (“what to do”) into procedural knowledge (“how to do it”). A part of teaching procedural
Journal of Surgical Education & 2015 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2015.05.009
1
FIGURE 1. Impressions from ALICE: upper left with 3D environment that users can explore from a first-person view. Upper right shows patient contact: in the lower bar users can choose diagnostic tests, physical examination, and anamnesis, and have access to the patient record. Lower left represents simple physical examination (text in lower left quadrant is in German language and means “Ouch! This hurts”) and lower left shows the virtual debriefing with explanations. (Here: “In addition you see a coprolith with a sonic shadow.”)
knowledge is learning and mastering algorithms in diagnostic workup and therapy. These workflows are essential not only for handling emergency situations but also they are important parts of professional expertise. Many of todayʼs diagnostic and therapeutic algorithms are defined as standard operation procedures that enable performance uniformity in the clinical daily routine.2 Novel educational concepts like problem-based learning, skills laboratory, and mannequin simulators3,4 are increasingly used for teaching procedural knowledge in diagnosis and treatment pathways. The rapid advance in computer development enables new e-learning technologies for teaching procedural knowledge even on home computers. “Immersive patient simulators (IPS)” are a possible method that enable a representation of a virtual counterpart in a 3-dimensional (3D) virtual environment. Students can freely interact in real time with virtual patients, individually or in teams with other students (Fig. 1). By playful immersion into the digital environment,
students can face the consequences of different decisions (“trial and error”) without putting real patients at risk. Repetitive training allows internalization and consolidation of the scripts that are relevant for the necessary procedure. The immersive aspect plays an important role as immersion grade can promote learning performance.5 The use of such simulators in clinical education is still rare and mostly limited on feasibility studies. The acceptance for the implementation of IPS in the daily curriculum depends on several factors. Clinical teachers may require information about amount of curricular content and effect on learning success (validation). Furthermore, such simulators need a certain level of technical quality (usability, features, and graphical detail level) to ensure a sufficient identification with the avatar (immersion grade) and thus a motivational learning environment. A thematic review of the currently available immersive patient simulator that was recently published by our group revealed that there is
2
Journal of Surgical Education Volume ]/Number ] ] 2015
student had to fill out after finishing the simulator session (Fig. 3). Usage and first validation was approved as a pilot project by the educational committee of the medical faculty at the University of Cologne. The institutional review board was informed, and there were no objections.
worldwide no immersive patient simulator available combining high technical quality with profound amount of medical content and is freely accessible for all students.6 Furthermore, we revealed that immersive simulators are mostly developed by commercial companies and limit the target audience on paying customers. Therefore it was our aim to develop a high-fidelity webbased 3D simulator prototype (Artificial learning interface for clinical education [ALICE]) that specifically addresses medical students and teaches standard diagnostic pattern. In the next step we wanted to perform a first validation of this new educational approach to clarify whether the simulator prototype can have positive effect on knowledge gain and have the potential to support clinical teachers in the daily routine.
Evaluation of simulator acceptance and studentsʼ opinion were analyzed using descriptive methods such as Likert scale. Simulator performance was analyzed using the Cochran, McNemar (concordance validity), and Studentʼs t-test (predictive validity). Data were analyzed using SPSS software package version 15.0 (SPSS Software GmbH, Munich, Germany). A p-value below 0.05 was considered significant.
MATERIAL AND METHODS
RESULTS
Simulator Requirements
Simulator: Medical Content
The conceptual design included the definition of the parameters: amount of curricular content, grade of technical quality, cost-effectiveness, and level of validation. Regarding the content, the prototype should cover the basic surgical declarative and procedural knowledge of 3 representative disease patterns in the field of visceral surgery based on the demands by the German examination regulations. The main requirement on the technical quality is a high immersion grade to obtain a high identification with the learning content. This should be achieved by application of classic concepts of 3D game design7: a free user-simulator interaction supported by an intuitive graphical user interface (GUI) in an avatar-based 3D simulation with up-to-date graphical quality. Long-time motivation should be promoted by the implementation of established game design techniques like for example “reward systems.”8 The simulator should work on any standard operating system and thus on most of the commonly used computers and should be available to all medical students anywhere and at any time. The simulator prototype should be cost-effective to enable a free distribution of this learning tool to other clinical teachers. A first validation of the simulator prototype should point out whether its use has effect on learning success. Simulator performance should be determined using the following parameters: time needed for resolving a case, correct order of the diagnostic pathway, and right or wrong diagnosis. The simulator should continuously record the user behavior by “server side” logging and storing studentsʼ decisions. This ensures further off line analysis. Validation should be assessed according to the consensus guideline for validation of virtual reality surgical simulators.9 Moreover, the acceptance and effect on learnersʼ motivation should be determined. Studentʼs opinion about acceptance, effectivity, and applicability was determined by a questionnaire that every
The simulator prototype (ALICE) displays a small outpatient clinic with a treatment room that can be explored freely from a “first-person view” similar to a video game (Fig. 1). The user can interact with nonplayer characters like nurses, patients, and other doctors. When using the simulator for the first time the users passes a small tutorial that explains the basic controls and functions followed by a short instructional case (case number 1) to learn simulator usage. Educational objective of the introduction case is to become familiar with the simulator usage. ALICE simulates clinical situations that are based on specific educational objectives. Every action causes a reaction by the simulator and the learner faces the consequences of different decisions. Case number 2 starts with a short patient anamnesis and a short multiple-choice test related to the corresponding disease to determine studentsʼ preexisting knowledge. After passing the test, the user is able to interact with the virtual patient and chooses the different options (anamnesis, examination, and diagnostic tests) (Fig. 1). There is neither a restriction in the sequence order nor a restriction on medically indicated examinations. The user is able to freely choose between all options and in case of medically not indicated examinations the test shows a normal finding. Once the student can make a diagnosis, he can initiate the necessary treatments and finish the scene. The simulator prototype included 4 different diseases. Case 1 contained a simple clinical picture (acute appendicitis). Case 2 contains an acute cholecystitis, case 3 contains the clinical picture of diverticulitis of the colon, and case 4 simulates a patient with symptomatic cholecystolithiasis. For each disease, declarative and procedural learning targets were defined. Declarative content was based on the German medical examination regulations and served as template for the procedural content. Moreover, declarative background knowledge was specifically prepared and embedded into the
Journal of Surgical Education Volume ]/Number ] ] 2015
3
Statistics
simulation as multimedia e-learning content. Representative media files like computed tomography scans, X-ray, and ultrasound videos were chosen by an experienced radiologist. The underlying scripts that contain adequate diagnostic and treatment patterns were defined in process records (Fig. 2) by 2 different experienced clinical teachers and served as templates for the analysis of studentsʼ performance. The studentsʼ performance was reviewed immediately after simulation. A virtual instructor summarizes the performance based on a short comparison of studentsʼ choices and stored standard operation procedures. The user gets a detailed review of the chosen procedural pattern, the diagnosis, and chosen treatment. The virtual instructor also summarizes the underlying declarative knowledge and shows the optimal procedural pattern. The debriefing ends with a video presentation of the surgical procedure. Cases were recapitulated and summarized after simulator evaluation by a clinical teacher in small-group face-to-face learning.
Long-time motivation was promoted by storing the studentsʼ performance for future simulator sessions. Every virtual counterpart (“avatar”) has an experience level that is displayed beyond the avatarʼs name. The experience level increases by collecting experience points that are gained by good simulator performance and ranges from “first-year medical student” up to “resident.” The current experience level is saved to the registered user. Once the student returns, the simulation continues at the previously reached level. Moreover, the saved experience level is displayed together with other information (e.g., number of treated patients) on the simulatorʼs webpage so that students can compare their simulator progress. Simulator: Technical Realization The 3D engine was based on a freely available programming environment (“Thinking Worlds,” St. Peterʼs Gate University of Sunderland, United Kingdom; http://www.caspian learning.co.uk/contact). This authoring tool allows easy
FIGURE 2. Template of an “optimal” diagnostic process. Note that the light and dark blue subitems can be freely chosen. 4
Journal of Surgical Education Volume ]/Number ] ] 2015
FIGURE 3. Acceptance, effectivity, and applicability and the preexisting computer affinity was determined by a Likert-scale questionnaire where student had to mark 1 of 6 checkboxes for every question: 1—very reasonable, 2—mainly reasonable, 3—reasonable, 4—partially reasonable, 5—hardly reasonable, and 6—not reasonable. Students appreciated e-learning (question 9) and showed a high simulator acceptance (questions 6 and 10). The prototype was easy to learn (question 2) and use (question 3).
development for different operating systems with adequategraphical-quality runs on all commonly used platforms and devices. The simulation prototype was hosted on a low-cost Standard Personal Computerʼs executable file. Usability of the simulation was ensured by developing a GUI that simplifies human-computer interaction (Fig. 2). The wholesimulator navigation can be controlled by a computer mouse or alternatively directly via touchscreen. The GUI was specifically designed with the purpose to offer an easy navigation combined with a maximum degree of freedom. Motivation was promoted by stylistic elements similar to computer games like small cut scenes and abrupt crossfading of camera views and ambient sound just to name a few. By these manipulations the immersive effect and the motivation should be increased. Cost-effectiveness was ensured by developing on existing standard computers using only free software. The whole project, including programming, 3D graphic design, and authoring, was made by the authors. Hence there were no specific personnel costs except for a time investment of roughly 500 hours development that was invested in by academic surgeons as part of their research commitments. Except for the time used, wholesimulator prototype development cost was less than 1000$. The simulator prototype is available free of charge to interested clinical teachers. Apart from a personal computer there is no additional investment necessary.
The prototype was evaluated with 25 medical students at the University of Cologne. Among them 17 were female and 8 were male. Mean student age was 24 (20-28 years).
Studentʼs opinion about acceptance, effectivity, and applicability was determined by using a 6-point Likert scale. The 6 response categories were chosen as learning performance in Germany is traditionally measured in a 6-point scale and thus the participating students are familiar with a 6-level grading system. By using an even-scale response level the neutral opinion was eliminated (“forced choice”). Likert-scale assessment was averaged and is summarized in Figure 3. Most of the students appreciate e-learning (question 9), showed a high acceptance of the simulator prototype (question 10), and would use such a simulator frequently (question 6). For most of the students the prototype was easy to learn (question 2) and use (question 3). Students had fun while learning with the simulator (question 1) and were in the opinion that ALICE teaches new content (question 4). Most of the students use computers every day (question 7). Concordance validity as parameter for reproduction of trained content was determined by comparison of the performance in case number 2 vs case number 4 as these cases dealt with a similar disease pattern but different diagnoses. According to the parameter “time used for resolving the case” there was a significant difference between case numbers 2 and 4 and thus an increase in simulator performance. For parameters “correct pathway” and “diagnosis” there was no significant improvement either in the McNemar test (p ¼ 0.7) or in the Cochran test (p ¼ 0.135); however, there was a positive trend. Predictive validity is an important parameter for simulator quality10,11 and was tested by measuring the effect of simulator use on declarative knowledge. In all, 11 multiple-choice questions that were asked before and after the simulation were compared and showed a significant improvement in multiple-choice results (Fig. 4).
Journal of Surgical Education Volume ]/Number ] ] 2015
5
Validation
FIGURE 4. The parameter “time used for resolving the case” revealed a significant difference between cases 2 and 4 (t-test) and thus an increase in simulator performance. For parameters “correct pathway” and “diagnosis” the increase was not significant (McNemar test [p ¼ 0.7] and Cochran test [p ¼ 0.135]). Predictive validity showed a significant improvement in answering the multiple-choice questions in the t-test.
Immersive Human Patient Simulators can have positive effect on learning success.12 Current available simulators that combine high immersion grade with simulation of complex procedures like emergency department protocols or clinical pathways are mostly commercially driven as development of such simulators is time and resource consuming. This limits the target audience on paying customers. Therefore we developed a novel cost-effective simulator prototype named “ALICE” for supporting clinical education. The transition from the classroom into clinical teaching can be abstracted in the “Kolb learning cycle”13,14 (Fig. 5). ALICE was designed to support studentsʼ proactive preparation for all steps in the Kolb learning cycle: multimedia summary of theoretical background supports abstract conceptualization (step 1),15 which prepares for the step of active experimentation (step 2). High degree of freedom and immersion enables a playful active experimentation. Students gain experience (step 3) without harming real patients and consolidate medical workflows. These parts of clinical education are also summarized
as “core competencies” of the Accreditation Council for Graduate Medical Education. ALICE can support medical teachers in teaching medical knowledge and enriches problem-based learning sessions. Hence ALICE can potentially support 2 of the 6 core competencies. However, the virtual character of IPS limits the experience on visual and auditive sensory perception. Sufficient simulation of physical examination and professional communication skills is hardly possible. Reflective observation (step 4) is promoted by debriefing with a virtual instructor. Postprocedural review of studentsʼ performance immediately after training is known to be an effective incentive of knowledge gain.16 ALICE offers a debriefing after each case that provides general declarative knowledge but also comments on studentsʼ individual decisions. However, a computer-generated summary is not specifically geared to the needs of the individual student, which is one of the advantages in small-group learning accompanied by an experienced clinical teacher. Indeed, ALICE is neither intended nor able to replace the current clinical curriculum. Clinical education is not limited
6
Journal of Surgical Education Volume ]/Number ] ] 2015
DISCUSSION
FIGURE 5. Modified “Kolb learning cycle”. Simulator training can support the transfer of declarative (“what to do”) knowledge into procedural (“how to do it”) knowledge.
on teaching standard operation procedures but furthermore characterized by weighting clinical findings, evaluation of different hypotheses, and clinical experience. Clinical and especial surgical education is most effective in attendance learning courses (bedside teaching and skills laboratory) accompanied by a medical teacher.3,4 These courses are most effective in small groups17 and moreover when studentsʼ knowledge level is equalized. However, smallgroup learning is impaired by increasing workload of hospital doctors,18 restrictive working time directives, and changes of studentʼs attitude and expectations in the sense of the “Generation Y.”19 Here ALICE can support clinical education by preparing students for the attendance courses as they allow time- and location-independent learning with individual pace and repetitions. When combined with a short performance test, studentsʼ knowledge levels can be equalized and possibly results in a more efficient learning. The effect of simulator learning on students motivation is already described20 and approved by our first validation; however, self-assessment forms are not a sufficient instrument for validation21 and have only informative value. Clinical teachers may ask whether the usage of an immersive patient simulator is beneficial in the daily curriculum. Direct comparison of novel and traditional teaching methods regarding learning success is difficult as it is influenced by many factors. However, the effectiveness of teaching methods can be determined by proving different validity levels.14 Our first validation revealed that ALICE showed positive concordance validity as simulator usage influences future simulator performance. Although student sample size and number of multiple-choice questions were not very high, the prototype showed positive predictive validity as simulator use improved the outcome in the multiple-choice tests. We deliberately limited this evaluation on 11 multiple-choice questions as more might disrupt the flow of the simulation. Immersion plays a fundamental role in avatar-based simulators as identification with the avatar
improves learning success as there is a positive effect of “highfidelity” visual presentation on degree of immersion22 and even learning performance.5 However, immersion grade is influenced by many factors23 and therefore hard to verify. A key factor is technical quality, which is a constant balancing act between 2 constrains: availability and system compatibility. ALICE ensures a ubiquitous availability by providing a platform that is broadcasted directly on the usersʼ device. However, broadcasting speed and technical limitations of the web browsersʼ graphical capabilities impair graphical quality and consequently the immersion grade. Designing ALICE, we developed a university-based, immersive patient simulator with profound medical content. With proven effect on knowledge gain and learnersʼ satisfaction, IPS may be used as an additional tool for teaching procedural knowledge. ALICE was built as a low-budget project and serves as a feasibility study for future improvements. The current prototype is a compromise between feasibility and graphical quality. Future improvement of medical content and graphical quality as well as a more detailed validation study are the next steps in the future development of this simulator.
Journal of Surgical Education Volume ]/Number ] ] 2015
7
CONCLUSION ALICE is a cost-effective IPS that is free to use for every student and can support clinical education ideally in the blended learning context. It allows a time- and locationindependent learning with individual pace. A first evaluation of the prototype revealed a positive effect on knowledge gain and shows a high rate of student acceptance.
REFERENCES 1. Back DA, et al. High efficacy and students’ satisfaction
after voluntary vs mandatory use of an e-learning
program in traumatology and orthopedics—a followup study. J Surg Educ. 2014;71(3):353-359. 2. Papakonstantinou D, et al. Enabling the use of
enhanced medical SOPs by an mLearning training solution. Stud Health Technol Inform. 2013;190:86-88. 3. Zendejas B, et al. Patient outcomes in simulation-
based medical education: a systematic review. J Gen Intern Med. 2013;28(8):1078-1089. 4. Gerdes B, et al. Instituting a surgical skills lab at a
training hospital. Chirurg. 2006;77(11):1033-1039. 5. Gutierrez F, et al. The effect of degree of immersion
ating the clerkship years of medical school: a guide for medical students, implications for residents and faculty. Acad Med. 2010;85(4):706-709. 14. Carter FJ, et al. Consensus guidelines for validation of
virtual reality surgical simulators. Surg Endosc. 2005; 19(12):1523-1532. 15. Chumley-Jones HS, Dobbie A, Alford CL. Web-based
learning: sound educational method or hype? A review of the evaluation literature Acad Med. 2002;77(10): S86-S93.
upon learning performance in virtual reality simulations for medical education. Stud Health Technol Inform. 2007;125:155-160.
16. Welke TM, et al. Personalized oral debriefing versus
6. Kleinert R, et al. 3D immersive patient simulators and
17. Sutkin G, et al. What makes a good clinical teacher in
standardized multimedia instruction after patient crisis simulation. Anesth Analg. 2009;109(1):183-189.
their impact on learning success: a thematic review. J Med Internet Res. 2015;17(4):e91.
medicine? A review of the literature Acad Med. 2008;83(5):452-466.
7. Desurvire H, El-Nasr MS. Methods for game user
18. Degen C, et al. The impact of training and working
research: studying player behavior to enhance game design. IEEE Comput Graph Appl. 2013;33(4):82-87.
conditions on junior doctorsʼ intention to leave clinical practice. BMC Med Educ. 2014;14:119.
8. Katsyri J, et al. Just watching the game ainʼt enough:
19. Schlitzkus LL, Schenarts KD, Schenarts PJ. Is your
striatal fMRI reward responses to successes and failures in a video game during active and vicarious playing. Front Hum Neurosci. 2013;7:278.
20. Knight J. Mentor for success. Nurs Stand. 2010;25(5):
9. Graafland M, Schraagen JM, Schijven MP. Systematic
review of serious games for medical education and surgical skills training. Br J Surg. 2012;99(10):1322-1330. 10. Funke K, et al. Blended learning in surgery using the
Inmedea Simulator. Langenbecks Arch Surg. 2013;398 (2):335-340. 11. Knight JF, et al. Serious gaming technology in major
incident triage training: a pragmatic controlled trial. Resuscitation. 2010;81(9):1175-1179. 12. Cook DA, Erwin PJ, Triola MM. Computerized
virtual patients in health professions education: a systematic review and meta-analysis. Acad Med. 2010;85(10):1589-1602.
8
13. Greenberg L, Blatt B. Perspective: successfully negoti-
residency program ready for Generation Y? J Surg Educ. 2010;67(2):108-111. 18-19. 21. Davis DA, et al. Accuracy of physician self-assessment
compared with observed measures of competence: a systematic review. J Am Med Assoc. 2006;296(9): 1094-1102. 22. Huerta R. Measuring the impact of narrative on
playerʼs presence and immersion in a first person game environment.Disseration Master of Science in Computer Science. Pan American: The University of Texas; 2012. 78. 23. Kuhlen TW, Hentschel B. Quo vadis CAVE: does
immersive visualization still matter? IEEE Comput Graph Appl. 2014;34(5):14-21.
Journal of Surgical Education Volume ]/Number ] ] 2015
Design, Realization, and First Validation of an Immersive Web-Based Virtual Patient Simulator for Training Clinical Decisions in Surgery Robert Kleinert, MD,* Nadine Heiermann, MD,* Roger Wahba, MD,* De-Huan Chang, MD,*,† Arnulf H. Hölscher, MD, PhD,* and Dirk L. Stippel, MD, PhD* *
Department of General, Visceral and Cancer Surgery of the University of Cologne, Center of Integrated Oncology of the University of Cologne, Transplant Center Cologne, Cologne, Germany; and †Department of Radiology, University of Cologne, Transplant Center Cologne, Cologne, Germany BACKGROUND: Immersive patient simulators (IPS) allow
an illusionary immersion into a synthetic world where the user can freely navigate through a 3-dimensional environment similar to computer games. Playful learning with IPS allows internalization of medical workflows without harming real patients. Ideally, IPS show high student acceptance and can have positive effect on knowledge gain. Development of IPS with high technical quality is resource intensive. Therefore most of the “high-fidelity” IPS are commercially driven. Usage of IPS in the daily curriculum is still rare. There is no academic-driven simulator that is freely accessible to every student and combines high immersion grade with a profound amount of medical content. AIM: Therefore it was our aim to develop an academic-
driven IPS prototype that is free to use and combines a high immersion grade with profound medical content. In addition, a first validation of the prototype was conducted. METHODS: The conceptual design included definition of
the following parameters: amount of curricular content, grade of technical quality, availability, and level of validation. A preliminary validation was done with 25 students. Studentsʼ opinion about acceptance was evaluated by a Likert-scale questionnaire. Effect on knowledge gain was determined by testing concordance and predictive validity. RESULTS: A custom-made simulator prototype (Artificial learning interface for clinical education [ALICE]) displays a virtual clinic environment that can be explored from a firstperson view similar to a video game. By controlling an
Correspondence: Inquiries to Robert Kleinert, MD, Department of General, Visceral and Cancer Surgery of the University of Cologne, Center of Integrated Oncology of the University of Cologne, Transplant Center Cologne, Kerpener Straße 62, 50937 Cologne, Germany; fax: (221) 478-6258; e-mail: [email protected]
avatar, the user navigates through the environment, is able to treat virtual patients, and faces the consequence of different decisions. ALICE showed high studentsʼ acceptance. There was positive correlation for concordance validity and predictive validity. Simulator usage had positive effect on reproduction of trained content and declarative knowledge. CONCLUSIONS: We successfully developed a university-
based, IPS prototype (ALICE) with profound medical content. ALICE is a nonprofit simulator, easy to use, and showed high studentsʼ acceptance; thus it potentially provides an additional tool for supporting student teaching C 2015 in the daily clinical curriculum. ( J Surg ]:]]]-]]]. J Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: ALICE, immersive patient simulators, web-
based learning, validity, immersion, procedural knowledge COMPETENCIES: Medical
Knowledge, Practice-Based
Learning and Improvement
INTRODUCTION e-Learning is an integrative part of medical education and meets the requirements of the current student generation (“digital natives”). Internet-based e-learning programs enable time-and location-independent learning with individual pace and repetitions and are nowadays mainly used for supportive teaching of theoretical (declarative) knowledge in preclinical and clinical education.1 Clinical education is characterized by the transfer of declarative (“what to do”) into procedural knowledge (“how to do it”). A part of teaching procedural
Journal of Surgical Education & 2015 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2015.05.009
1
FIGURE 1. Impressions from ALICE: upper left with 3D environment that users can explore from a first-person view. Upper right shows patient contact: in the lower bar users can choose diagnostic tests, physical examination, and anamnesis, and have access to the patient record. Lower left represents simple physical examination (text in lower left quadrant is in German language and means “Ouch! This hurts”) and lower left shows the virtual debriefing with explanations. (Here: “In addition you see a coprolith with a sonic shadow.”)
knowledge is learning and mastering algorithms in diagnostic workup and therapy. These workflows are essential not only for handling emergency situations but also they are important parts of professional expertise. Many of todayʼs diagnostic and therapeutic algorithms are defined as standard operation procedures that enable performance uniformity in the clinical daily routine.2 Novel educational concepts like problem-based learning, skills laboratory, and mannequin simulators3,4 are increasingly used for teaching procedural knowledge in diagnosis and treatment pathways. The rapid advance in computer development enables new e-learning technologies for teaching procedural knowledge even on home computers. “Immersive patient simulators (IPS)” are a possible method that enable a representation of a virtual counterpart in a 3-dimensional (3D) virtual environment. Students can freely interact in real time with virtual patients, individually or in teams with other students (Fig. 1). By playful immersion into the digital environment,
students can face the consequences of different decisions (“trial and error”) without putting real patients at risk. Repetitive training allows internalization and consolidation of the scripts that are relevant for the necessary procedure. The immersive aspect plays an important role as immersion grade can promote learning performance.5 The use of such simulators in clinical education is still rare and mostly limited on feasibility studies. The acceptance for the implementation of IPS in the daily curriculum depends on several factors. Clinical teachers may require information about amount of curricular content and effect on learning success (validation). Furthermore, such simulators need a certain level of technical quality (usability, features, and graphical detail level) to ensure a sufficient identification with the avatar (immersion grade) and thus a motivational learning environment. A thematic review of the currently available immersive patient simulator that was recently published by our group revealed that there is
2
Journal of Surgical Education Volume ]/Number ] ] 2015
student had to fill out after finishing the simulator session (Fig. 3). Usage and first validation was approved as a pilot project by the educational committee of the medical faculty at the University of Cologne. The institutional review board was informed, and there were no objections.
worldwide no immersive patient simulator available combining high technical quality with profound amount of medical content and is freely accessible for all students.6 Furthermore, we revealed that immersive simulators are mostly developed by commercial companies and limit the target audience on paying customers. Therefore it was our aim to develop a high-fidelity webbased 3D simulator prototype (Artificial learning interface for clinical education [ALICE]) that specifically addresses medical students and teaches standard diagnostic pattern. In the next step we wanted to perform a first validation of this new educational approach to clarify whether the simulator prototype can have positive effect on knowledge gain and have the potential to support clinical teachers in the daily routine.
Evaluation of simulator acceptance and studentsʼ opinion were analyzed using descriptive methods such as Likert scale. Simulator performance was analyzed using the Cochran, McNemar (concordance validity), and Studentʼs t-test (predictive validity). Data were analyzed using SPSS software package version 15.0 (SPSS Software GmbH, Munich, Germany). A p-value below 0.05 was considered significant.
MATERIAL AND METHODS
RESULTS
Simulator Requirements
Simulator: Medical Content
The conceptual design included the definition of the parameters: amount of curricular content, grade of technical quality, cost-effectiveness, and level of validation. Regarding the content, the prototype should cover the basic surgical declarative and procedural knowledge of 3 representative disease patterns in the field of visceral surgery based on the demands by the German examination regulations. The main requirement on the technical quality is a high immersion grade to obtain a high identification with the learning content. This should be achieved by application of classic concepts of 3D game design7: a free user-simulator interaction supported by an intuitive graphical user interface (GUI) in an avatar-based 3D simulation with up-to-date graphical quality. Long-time motivation should be promoted by the implementation of established game design techniques like for example “reward systems.”8 The simulator should work on any standard operating system and thus on most of the commonly used computers and should be available to all medical students anywhere and at any time. The simulator prototype should be cost-effective to enable a free distribution of this learning tool to other clinical teachers. A first validation of the simulator prototype should point out whether its use has effect on learning success. Simulator performance should be determined using the following parameters: time needed for resolving a case, correct order of the diagnostic pathway, and right or wrong diagnosis. The simulator should continuously record the user behavior by “server side” logging and storing studentsʼ decisions. This ensures further off line analysis. Validation should be assessed according to the consensus guideline for validation of virtual reality surgical simulators.9 Moreover, the acceptance and effect on learnersʼ motivation should be determined. Studentʼs opinion about acceptance, effectivity, and applicability was determined by a questionnaire that every
The simulator prototype (ALICE) displays a small outpatient clinic with a treatment room that can be explored freely from a “first-person view” similar to a video game (Fig. 1). The user can interact with nonplayer characters like nurses, patients, and other doctors. When using the simulator for the first time the users passes a small tutorial that explains the basic controls and functions followed by a short instructional case (case number 1) to learn simulator usage. Educational objective of the introduction case is to become familiar with the simulator usage. ALICE simulates clinical situations that are based on specific educational objectives. Every action causes a reaction by the simulator and the learner faces the consequences of different decisions. Case number 2 starts with a short patient anamnesis and a short multiple-choice test related to the corresponding disease to determine studentsʼ preexisting knowledge. After passing the test, the user is able to interact with the virtual patient and chooses the different options (anamnesis, examination, and diagnostic tests) (Fig. 1). There is neither a restriction in the sequence order nor a restriction on medically indicated examinations. The user is able to freely choose between all options and in case of medically not indicated examinations the test shows a normal finding. Once the student can make a diagnosis, he can initiate the necessary treatments and finish the scene. The simulator prototype included 4 different diseases. Case 1 contained a simple clinical picture (acute appendicitis). Case 2 contains an acute cholecystitis, case 3 contains the clinical picture of diverticulitis of the colon, and case 4 simulates a patient with symptomatic cholecystolithiasis. For each disease, declarative and procedural learning targets were defined. Declarative content was based on the German medical examination regulations and served as template for the procedural content. Moreover, declarative background knowledge was specifically prepared and embedded into the
Journal of Surgical Education Volume ]/Number ] ] 2015
3
Statistics
simulation as multimedia e-learning content. Representative media files like computed tomography scans, X-ray, and ultrasound videos were chosen by an experienced radiologist. The underlying scripts that contain adequate diagnostic and treatment patterns were defined in process records (Fig. 2) by 2 different experienced clinical teachers and served as templates for the analysis of studentsʼ performance. The studentsʼ performance was reviewed immediately after simulation. A virtual instructor summarizes the performance based on a short comparison of studentsʼ choices and stored standard operation procedures. The user gets a detailed review of the chosen procedural pattern, the diagnosis, and chosen treatment. The virtual instructor also summarizes the underlying declarative knowledge and shows the optimal procedural pattern. The debriefing ends with a video presentation of the surgical procedure. Cases were recapitulated and summarized after simulator evaluation by a clinical teacher in small-group face-to-face learning.
Long-time motivation was promoted by storing the studentsʼ performance for future simulator sessions. Every virtual counterpart (“avatar”) has an experience level that is displayed beyond the avatarʼs name. The experience level increases by collecting experience points that are gained by good simulator performance and ranges from “first-year medical student” up to “resident.” The current experience level is saved to the registered user. Once the student returns, the simulation continues at the previously reached level. Moreover, the saved experience level is displayed together with other information (e.g., number of treated patients) on the simulatorʼs webpage so that students can compare their simulator progress. Simulator: Technical Realization The 3D engine was based on a freely available programming environment (“Thinking Worlds,” St. Peterʼs Gate University of Sunderland, United Kingdom; http://www.caspian learning.co.uk/contact). This authoring tool allows easy
FIGURE 2. Template of an “optimal” diagnostic process. Note that the light and dark blue subitems can be freely chosen. 4
Journal of Surgical Education Volume ]/Number ] ] 2015
FIGURE 3. Acceptance, effectivity, and applicability and the preexisting computer affinity was determined by a Likert-scale questionnaire where student had to mark 1 of 6 checkboxes for every question: 1—very reasonable, 2—mainly reasonable, 3—reasonable, 4—partially reasonable, 5—hardly reasonable, and 6—not reasonable. Students appreciated e-learning (question 9) and showed a high simulator acceptance (questions 6 and 10). The prototype was easy to learn (question 2) and use (question 3).
development for different operating systems with adequategraphical-quality runs on all commonly used platforms and devices. The simulation prototype was hosted on a low-cost Standard Personal Computerʼs executable file. Usability of the simulation was ensured by developing a GUI that simplifies human-computer interaction (Fig. 2). The wholesimulator navigation can be controlled by a computer mouse or alternatively directly via touchscreen. The GUI was specifically designed with the purpose to offer an easy navigation combined with a maximum degree of freedom. Motivation was promoted by stylistic elements similar to computer games like small cut scenes and abrupt crossfading of camera views and ambient sound just to name a few. By these manipulations the immersive effect and the motivation should be increased. Cost-effectiveness was ensured by developing on existing standard computers using only free software. The whole project, including programming, 3D graphic design, and authoring, was made by the authors. Hence there were no specific personnel costs except for a time investment of roughly 500 hours development that was invested in by academic surgeons as part of their research commitments. Except for the time used, wholesimulator prototype development cost was less than 1000$. The simulator prototype is available free of charge to interested clinical teachers. Apart from a personal computer there is no additional investment necessary.
The prototype was evaluated with 25 medical students at the University of Cologne. Among them 17 were female and 8 were male. Mean student age was 24 (20-28 years).
Studentʼs opinion about acceptance, effectivity, and applicability was determined by using a 6-point Likert scale. The 6 response categories were chosen as learning performance in Germany is traditionally measured in a 6-point scale and thus the participating students are familiar with a 6-level grading system. By using an even-scale response level the neutral opinion was eliminated (“forced choice”). Likert-scale assessment was averaged and is summarized in Figure 3. Most of the students appreciate e-learning (question 9), showed a high acceptance of the simulator prototype (question 10), and would use such a simulator frequently (question 6). For most of the students the prototype was easy to learn (question 2) and use (question 3). Students had fun while learning with the simulator (question 1) and were in the opinion that ALICE teaches new content (question 4). Most of the students use computers every day (question 7). Concordance validity as parameter for reproduction of trained content was determined by comparison of the performance in case number 2 vs case number 4 as these cases dealt with a similar disease pattern but different diagnoses. According to the parameter “time used for resolving the case” there was a significant difference between case numbers 2 and 4 and thus an increase in simulator performance. For parameters “correct pathway” and “diagnosis” there was no significant improvement either in the McNemar test (p ¼ 0.7) or in the Cochran test (p ¼ 0.135); however, there was a positive trend. Predictive validity is an important parameter for simulator quality10,11 and was tested by measuring the effect of simulator use on declarative knowledge. In all, 11 multiple-choice questions that were asked before and after the simulation were compared and showed a significant improvement in multiple-choice results (Fig. 4).
Journal of Surgical Education Volume ]/Number ] ] 2015
5
Validation
FIGURE 4. The parameter “time used for resolving the case” revealed a significant difference between cases 2 and 4 (t-test) and thus an increase in simulator performance. For parameters “correct pathway” and “diagnosis” the increase was not significant (McNemar test [p ¼ 0.7] and Cochran test [p ¼ 0.135]). Predictive validity showed a significant improvement in answering the multiple-choice questions in the t-test.
Immersive Human Patient Simulators can have positive effect on learning success.12 Current available simulators that combine high immersion grade with simulation of complex procedures like emergency department protocols or clinical pathways are mostly commercially driven as development of such simulators is time and resource consuming. This limits the target audience on paying customers. Therefore we developed a novel cost-effective simulator prototype named “ALICE” for supporting clinical education. The transition from the classroom into clinical teaching can be abstracted in the “Kolb learning cycle”13,14 (Fig. 5). ALICE was designed to support studentsʼ proactive preparation for all steps in the Kolb learning cycle: multimedia summary of theoretical background supports abstract conceptualization (step 1),15 which prepares for the step of active experimentation (step 2). High degree of freedom and immersion enables a playful active experimentation. Students gain experience (step 3) without harming real patients and consolidate medical workflows. These parts of clinical education are also summarized
as “core competencies” of the Accreditation Council for Graduate Medical Education. ALICE can support medical teachers in teaching medical knowledge and enriches problem-based learning sessions. Hence ALICE can potentially support 2 of the 6 core competencies. However, the virtual character of IPS limits the experience on visual and auditive sensory perception. Sufficient simulation of physical examination and professional communication skills is hardly possible. Reflective observation (step 4) is promoted by debriefing with a virtual instructor. Postprocedural review of studentsʼ performance immediately after training is known to be an effective incentive of knowledge gain.16 ALICE offers a debriefing after each case that provides general declarative knowledge but also comments on studentsʼ individual decisions. However, a computer-generated summary is not specifically geared to the needs of the individual student, which is one of the advantages in small-group learning accompanied by an experienced clinical teacher. Indeed, ALICE is neither intended nor able to replace the current clinical curriculum. Clinical education is not limited
6
Journal of Surgical Education Volume ]/Number ] ] 2015
DISCUSSION
FIGURE 5. Modified “Kolb learning cycle”. Simulator training can support the transfer of declarative (“what to do”) knowledge into procedural (“how to do it”) knowledge.
on teaching standard operation procedures but furthermore characterized by weighting clinical findings, evaluation of different hypotheses, and clinical experience. Clinical and especial surgical education is most effective in attendance learning courses (bedside teaching and skills laboratory) accompanied by a medical teacher.3,4 These courses are most effective in small groups17 and moreover when studentsʼ knowledge level is equalized. However, smallgroup learning is impaired by increasing workload of hospital doctors,18 restrictive working time directives, and changes of studentʼs attitude and expectations in the sense of the “Generation Y.”19 Here ALICE can support clinical education by preparing students for the attendance courses as they allow time- and location-independent learning with individual pace and repetitions. When combined with a short performance test, studentsʼ knowledge levels can be equalized and possibly results in a more efficient learning. The effect of simulator learning on students motivation is already described20 and approved by our first validation; however, self-assessment forms are not a sufficient instrument for validation21 and have only informative value. Clinical teachers may ask whether the usage of an immersive patient simulator is beneficial in the daily curriculum. Direct comparison of novel and traditional teaching methods regarding learning success is difficult as it is influenced by many factors. However, the effectiveness of teaching methods can be determined by proving different validity levels.14 Our first validation revealed that ALICE showed positive concordance validity as simulator usage influences future simulator performance. Although student sample size and number of multiple-choice questions were not very high, the prototype showed positive predictive validity as simulator use improved the outcome in the multiple-choice tests. We deliberately limited this evaluation on 11 multiple-choice questions as more might disrupt the flow of the simulation. Immersion plays a fundamental role in avatar-based simulators as identification with the avatar
improves learning success as there is a positive effect of “highfidelity” visual presentation on degree of immersion22 and even learning performance.5 However, immersion grade is influenced by many factors23 and therefore hard to verify. A key factor is technical quality, which is a constant balancing act between 2 constrains: availability and system compatibility. ALICE ensures a ubiquitous availability by providing a platform that is broadcasted directly on the usersʼ device. However, broadcasting speed and technical limitations of the web browsersʼ graphical capabilities impair graphical quality and consequently the immersion grade. Designing ALICE, we developed a university-based, immersive patient simulator with profound medical content. With proven effect on knowledge gain and learnersʼ satisfaction, IPS may be used as an additional tool for teaching procedural knowledge. ALICE was built as a low-budget project and serves as a feasibility study for future improvements. The current prototype is a compromise between feasibility and graphical quality. Future improvement of medical content and graphical quality as well as a more detailed validation study are the next steps in the future development of this simulator.
Journal of Surgical Education Volume ]/Number ] ] 2015
7
CONCLUSION ALICE is a cost-effective IPS that is free to use for every student and can support clinical education ideally in the blended learning context. It allows a time- and locationindependent learning with individual pace. A first evaluation of the prototype revealed a positive effect on knowledge gain and shows a high rate of student acceptance.
REFERENCES 1. Back DA, et al. High efficacy and students’ satisfaction
after voluntary vs mandatory use of an e-learning
program in traumatology and orthopedics—a followup study. J Surg Educ. 2014;71(3):353-359. 2. Papakonstantinou D, et al. Enabling the use of
enhanced medical SOPs by an mLearning training solution. Stud Health Technol Inform. 2013;190:86-88. 3. Zendejas B, et al. Patient outcomes in simulation-
based medical education: a systematic review. J Gen Intern Med. 2013;28(8):1078-1089. 4. Gerdes B, et al. Instituting a surgical skills lab at a
training hospital. Chirurg. 2006;77(11):1033-1039. 5. Gutierrez F, et al. The effect of degree of immersion
ating the clerkship years of medical school: a guide for medical students, implications for residents and faculty. Acad Med. 2010;85(4):706-709. 14. Carter FJ, et al. Consensus guidelines for validation of
virtual reality surgical simulators. Surg Endosc. 2005; 19(12):1523-1532. 15. Chumley-Jones HS, Dobbie A, Alford CL. Web-based
learning: sound educational method or hype? A review of the evaluation literature Acad Med. 2002;77(10): S86-S93.
upon learning performance in virtual reality simulations for medical education. Stud Health Technol Inform. 2007;125:155-160.
16. Welke TM, et al. Personalized oral debriefing versus
6. Kleinert R, et al. 3D immersive patient simulators and
17. Sutkin G, et al. What makes a good clinical teacher in
standardized multimedia instruction after patient crisis simulation. Anesth Analg. 2009;109(1):183-189.
their impact on learning success: a thematic review. J Med Internet Res. 2015;17(4):e91.
medicine? A review of the literature Acad Med. 2008;83(5):452-466.
7. Desurvire H, El-Nasr MS. Methods for game user
18. Degen C, et al. The impact of training and working
research: studying player behavior to enhance game design. IEEE Comput Graph Appl. 2013;33(4):82-87.
conditions on junior doctorsʼ intention to leave clinical practice. BMC Med Educ. 2014;14:119.
8. Katsyri J, et al. Just watching the game ainʼt enough:
19. Schlitzkus LL, Schenarts KD, Schenarts PJ. Is your
striatal fMRI reward responses to successes and failures in a video game during active and vicarious playing. Front Hum Neurosci. 2013;7:278.
20. Knight J. Mentor for success. Nurs Stand. 2010;25(5):
9. Graafland M, Schraagen JM, Schijven MP. Systematic
review of serious games for medical education and surgical skills training. Br J Surg. 2012;99(10):1322-1330. 10. Funke K, et al. Blended learning in surgery using the
Inmedea Simulator. Langenbecks Arch Surg. 2013;398 (2):335-340. 11. Knight JF, et al. Serious gaming technology in major
incident triage training: a pragmatic controlled trial. Resuscitation. 2010;81(9):1175-1179. 12. Cook DA, Erwin PJ, Triola MM. Computerized
virtual patients in health professions education: a systematic review and meta-analysis. Acad Med. 2010;85(10):1589-1602.
8
13. Greenberg L, Blatt B. Perspective: successfully negoti-
residency program ready for Generation Y? J Surg Educ. 2010;67(2):108-111. 18-19. 21. Davis DA, et al. Accuracy of physician self-assessment
compared with observed measures of competence: a systematic review. J Am Med Assoc. 2006;296(9): 1094-1102. 22. Huerta R. Measuring the impact of narrative on
playerʼs presence and immersion in a first person game environment.Disseration Master of Science in Computer Science. Pan American: The University of Texas; 2012. 78. 23. Kuhlen TW, Hentschel B. Quo vadis CAVE: does
immersive visualization still matter? IEEE Comput Graph Appl. 2014;34(5):14-21.
Journal of Surgical Education Volume ]/Number ] ] 2015
