The American Journal of Surgery (2015) 210, 585-590
Surgical Education
Virtual operating room for team training in surgery Jonathan S. Abelson, M.D.a,*, Elliott Silverman, P.A.a, Jason Banfelderb, Alexandra Naidesb, Ricardo Costaa, Gregory Dakin, M.D.a a
Department of Surgery, New York Presbyterian HospitaldWeill Cornell Medical College, New York, NY 10068, USA; bDepartment of Physiology and Biophysics, Weill Cornell Medical College, New York, NY, USA KEYWORDS: Virtual reality simulation; Team training; Surgery; Operating room
Abstract BACKGROUND: We proposed to develop a novel virtual reality (VR) team training system. The objective of this study was to determine the feasibility of creating a VR operating room to simulate a surgical crisis scenario and evaluate the simulator for construct and face validity. METHODS: We modified ICE STORM (Integrated Clinical Environment; Systems, Training, Operations, Research, Methods), a VR-based system capable of modeling a variety of health care personnel and environments. ICE STORM was used to simulate a standardized surgical crisis scenario, whereby participants needed to correct 4 elements responsible for loss of laparoscopic visualization. The construct and face validity of the environment were measured. RESULTS: Thirty-three participants completed the VR simulation. Attendings completed the simulation in less time than trainees (271 vs 201 seconds, P 5 .032). Participants felt the training environment was realistic and had a favorable impression of the simulation. All participants felt the workload of the simulation was low. CONCLUSIONS: Creation of a VR-based operating room for team training in surgery is feasible and can afford a realistic team training environment. Ó 2015 Elsevier Inc. All rights reserved.
The benefits of simulation in surgery are well documented, allowing trainees to achieve proficiency in shorter times, acquire news skills, and retain skills, all in environments that are inexpensive, reproducible, and safe.1–12 With the abundance of evidence to support simulation, surgical residency programs have rapidly adapted inanimate training
This study was funded in part by Lockheed Martin. * Corresponding author. Tel.: 11-914-980-4530; fax: 11-212-7465236. E-mail address: [email protected] Manuscript received November 26, 2014; revised manuscript January 7, 2015 0002-9610/$ - see front matter Ó 2015 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.amjsurg.2015.01.024
into their curricula, with most programs staffing full-time simulation centers.4,13–15 The American College of Surgeons has recognized the value of simulation in surgery and has developed an extensive accreditation program for simulation centers. Most of the focus in surgical simulation has been on task training of surgical skills, ranging from knot tying to chesttube insertion to flexible endoscopy. Perhaps the most salient example is laparoscopic surgical skill training, whereby a systematic course of box trainer–based laparoscopic tasks has proved so effective, and the course has been mandated for certification by the American Board of Surgery.16 However, technical skill is only one aspect of being an effective
586
The American Journal of Surgery, Vol 210, No 3, September 2015
surgeon. Furthermore, the surgeon is only one part of the team necessary to deliver effective surgical care. Successfully performing an operation requires the coordinated teamwork of the surgeon, anesthesiologist, nurses, hospital staff, and clinical information systems. Recognizing the importance of teamwork in the operating room (OR) has led educators to move from the relatively simplistic training of surgical tasks to the more complicated world of whole team training in the surgical environment.17–19 Team training in surgery involves creation of a simulated OR, in which combinations of real equipment coupled with mannequins and computerized integration allow a human team to recreate real-life operative scenarios. The American College of Surgeons has created a series of standardized OR situations that can be used in training.16 Several groups have reported success with multidisciplinary OR simulation.20–22 Others although have had difficulty executing such simulation secondary to cost and space requirements.13,23 Furthermore, with limited duty hours, it can be difficult to assemble the necessary team members to undergo the training. These factors have limited the widespread use of team training simulations in surgery compared with the relatively simple task training that has been widely adapted. We hypothesized that VR software can offer realistic team training environments that overcome some of the current limitations. Therefore, we proposed to create a multiuser interactive environment in which both team-based skill coupled with surgical decision making can be simulated, critiqued, and evaluated. Lockheed Martin Corporation (Oswego, NY), well known for its military-based simulation and training programs, has developed a VR-based environment for use in medical training called ICE STORM (Integrated Clinical Environment; Systems, Training, Operations, Research, Methods). The objective of this pilot study was to determine the feasibility of modifying the ICE STORM VR OR to simulate a standardized surgical crisis scenario and evaluate the simulator for construct and face validity.24,25
Methods The existing ICE STORM platform contains all the equipment and personnel necessary to simulate a variety of scenarios in a virtual OR. A core team of researchers from both Lockheed Martin and Weill Cornell Medical College (New York, NY) was established to modify ICE STORM to simulate an intraoperative crisis scenario. In addition to modeling the necessary elements of this crisis scenario, additional software was created to allow a human proctor to serve as an interface between study participants and the virtual world. The interface software used a standard iPad (Apple Inc., Cupertino, CA) to allow a human proctor to modify the simulation environment as necessary during the course of the training exercise, depending on the participant’s responses.
The laparoscopic troubleshooting module is a team training scenario published by the American College of Surgeons and has been previously validated.16 The full module is beyond the scope of this pilot project and entails a comprehensive team of surgeon, assistant surgeon, nurses, and anesthesiologist working through several crisis situations while performing a standard laparoscopic operation. In this project, the focus was narrowed to a small portion of the module, called ‘‘loss of laparoscopic visualization.’’ This scenario was then programmed into the modified ICE STORM platform. In the ‘‘loss of laparoscopic visualization’’ scenario, the team was performing a laparoscopic cholecystectomy and the laparoscopic monitor suddenly went dim. It was the job of the operating surgeon to troubleshoot the problem and return the monitor to working form. There were several possible problems that the participant, functioning as the operating surgeon, had to identify and check to restore function to the monitor. Each participant was required to perform 4 mandatory treatments: (1) check camera box and cord; (2) check light-source box and cord; (3) check (clean/inspect) or replace laparoscope (use spare); and (4) exchange and use spare camera. Participants were evaluated by time to completion of the simulation. Participants were given a ‘‘pass’’ if they completed all mandatory treatments in less than 270 seconds. This number was based on preliminary analysis of test subjects. Participants, acting as the surgeon, interacted with the VR world using the Gyration Air Mouse (SMK-Link Electronics, Camarillo, CA; Supplementary Fig. 1) to manipulate the surgeon avatar and interacted with the human proctor administrating the study by simply speaking aloud. The proctor then used the proctor-tool software on a standard iPad (Apple Inc.) to make modifications to the VR world (eg, participant instructs nurse to replace the laparoscope, the proctor enters the appropriate command, and then the nurse avatar replaces the laparoscope). Successful completion of the module was achieved when the participant identified all 4 of the critical elements that could lead to loss of visualization. No matter what order a participant identified the element, the simulation would not end until all 4 components were identified and correctly acted on. This concept, known as the ‘‘full cycle test,’’ ensured that all participants would demonstrate a complete understanding of the most critical elements of the troubleshooting scenario and eliminated the possibility that the participant could end the simulation simply by picking the ‘‘correct’’ cause of failure on the first try. Metric data from the simulation exercise were used to evaluate the construct validity of the virtual system. Three methods were used to determine the face validity: Likert scale questionnaires (Table 1), the Bedford Workload Scale (Supplementary Fig. 2), and the modified NASA-Task Load Index (NASA-TLX) scale (Supplementary Fig. 3). Subjects included attending surgeons (experts) and residents and medical students combined (trainees). Statistical analysis was conducted using SPSS 12.0 statistical software (IBM,
J.S. Abelson et al. Table 1
Virtual OR for team training in surgery
587
Likert scale results
Likert scale question number
Median
P value*
1. 2. 3. 4. 5. 6. 7.
5 4 1 4 3 3 4
.014 .534 .000 .073 .001 .001 .911
5 2 2 4
.000 .000 .000 .607
The training environment was realistic† This course will improve my performance in the OR† How would you rate your prior virtual world?‡ How would you rate your surgical experience level?x How often do you use surgical simulation training modules?‡ How would you rate your experience with non-surgical virtual world?x The VR training module differed greatly from other non–computer-based communication training modules† 8. I liked the VR training environment† 9. I did not like the VR training environment† 10. I found it difficult to communicate using the VR training environment† 11. I would like to do VR training prior to going to the OR†
*One-sample Whitney–Wilcoxon signed rank test value to determine if the median score was different from the midpoint of the scale, which was 4. † Scoring system: 1 5 strongly disagree, 4 5 moderately agree, 7 5 strongly agree. ‡ Scoring system: 1 5 never, 4 5 moderate, 7 5 very often. x Scoring system: 1 5 novice, 4 5 moderate, 7 5 expert.
New York, NY). Specific modalities used were Fisher exact test, ANOVA, and t tests. Metric data were analyzed using the Fisher exact test comparing pass–fail rates and time to completion because the sample size was low. Seven-point Likert scale questionnaires were used to assess a variety of opinions, including participants’ impression of the virtual environment with regard to realism, beliefs on whether the system would improve performance in the OR, and inclination to use the system in the future26 (Table 1). Median scores were used to determine a central tendency. Independent-sample Mann–Whitney test was used to detect any difference in the median scores between trainees and attendings. We also performed a 1-sample Whitney–Wilcoxon signed rank test comparing median scores against the midpoint of the evaluation scale (ie, 4). The aim of this analysis was to reveal whether the trainees’ evaluations were above the midpoint of the scale (ie, positive) or below it (ie, negative).19 The Bedford Workload scale is an unidimensional rating scale designed to identify operator’s spare mental capacity while completing a task.27 The single dimension is assessed using a hierarchical decision tree. Supplementary Fig. 2 explains the scoring system from workload 1 to 10. Independent-sample Mann–Whitney test was used to detect any difference between workload scores of attendings vs trainees. One-sample Wilcoxon signed rank test was calculated to understand any difference between the scores of all participants against the midpoint of the scale, which was 5. The modified NASA-TLX is a subjective, multidimensional, validated assessment tool that rates perceived workload on 6 different subscales: Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration.28 Results are combined but not weighted. Supplementary Fig. 3 reveals each category investigated and the scaling system. An independent-sample Mann– Whitney test was used to detect any difference between workload scores of attendings vs trainees. The 1-sample
Wilcoxon signed rank test was calculated to understand any difference between the scores of all participants against the midpoint of the scale, which was 10.
Results A total of 33 participants, including 26 trainees and 7 attendings, completed the virtual simulation. Attendings completed the simulation is less time than trainees (201 vs 271 seconds, P 5 .032; Fig. 1). A higher percentage of attendings passed the simulation compared with trainees although this was not statistically different (86% vs 58%, P 5 .223). Eleven questions were included in the Likert scale questionnaire. Those questions that pertain directly to opinions on the VR simulation are reviewed here (questions 1, 2, and 7 to 11). Those questions that addressed background information on participants’ surgical experience or previous virtual world experience are not discussed in this section (questions 3 to 6; Table 1). Overall, participants agreed that they liked the simulator (median 5 5, P 5 .000) and disagreed with the statement that they disliked the simulator (median 5 2, P 5 .000). Study participants felt that the training environment was realistic (median 5 5, P 5 .014) and did not feel that communication in the environment was difficult (median 5 2, P 5 .000). However, participants did not feel that the simulation would improve their performance (median 5 4, P 5 .534). Furthermore, they did not agree that they would like to do VR training before going to the OR (median 5 4, P 5 .607). Using the Bedford Workload scale, 82% of all participants felt that the workload was either low or that they had enough spare capacity for desirable additional tasks (Fig. 2). Using the modified NASA-TLX scale, all
588
The American Journal of Surgery, Vol 210, No 3, September 2015
Figure 1 Average simulation completion times, attendings vs trainees. *P 5 .032.
participants were found to have minimal mental, physical, and temporal demand. Likewise, none of the participants reported requiring a high amount of effort to complete the simulation (Fig. 3). Finally, there was no statistically significant difference in responses between attendings and trainees for all responses in the Likert scale, Bedford Workload scale, and Modified NASA-TLX scale.
Comments The OR is a dynamic, high-risk environment where successful delivery of care depends on the co-ordinated action of surgeons, anesthesiologists, nurses, hospital staff, and clinical information. With the increasing complexity of surgical instrumentation and patient disease, efficient OR processes and teamwork will become more important. The value of teamwork among medical professionals is well documented. Failures in co-ordination and communication of information among hospital clinicians have been associated with worse outcomes,29–31 longer lengths of stay, and higher nurse turnover in intensive care units32 and greater postoperative pain with lower functioning levels for
Figure 2
Bedford workload scale: all participants.
patients33–35 across specialties. Surgical teams at the Department of Veterans Affairs hospitals with low mortality rates communicate more effectively and more often than surgical teams associated with high mortality rates.36 In a study of anesthetic-related errors, 80% of occurrences were considered preventable with human error accounting for 75% of them.37 Studies in cardiac surgery show the clear impact of human factors in ‘‘near misses’’ in the high technology OR.38 Primary teamwork competencies, including knowledge, skill, and attitudes, positively correlate to effective teamwork and patient safety.39,40 However, these competencies of effective teamwork are often lacking in the modern OR. Surveys evaluating surgeons working with anesthesiologists have suggested that substandard collaboration occurs 50% of the time.41 Other studies have discovered discrepancy in OR team members’ views on appropriate team structure and importance of effective communication.42,43 The aviation industry noted years ago that 70% of errors were because of preventable human factors, such as failed interpersonal communication, decision making, and leadership.44 This led to the development of crew resource management training programs that use simulator-based training to understand the limitations of human performance and to develop a culture of safety.45 Anesthesiologists have addressed human factors in errors by developing simulators to train staff during anesthetic crises, allowing participants to integrate technical and team training skills with feedback on their performance.46–49 Given the importance of teamwork and communication in the OR, several groups are investigating the role of full-scale OR simulation to both study and improve these skills.17,21,22 The ultimate goals of such simulation include not only training of participants but identification of factors important to overall team performance and improved efficiency of OR systems. In this study, we created an interactive environment to simulate both team-based skills and surgical decision making. This virtual environment might ultimately enable simulation of a variety of OR circumstances, from patient transfer, to hand-off, to intraoperative crisis scenarios. A multiuser virtual world could ultimately be linked to operable surgical procedure simulation platforms to allow full-scale simulation of both surgical procedures and team dynamics. Once a complex OR system is modeled, we
Figure 3
Modified NASA-TLX: all participants.
J.S. Abelson et al.
Virtual OR for team training in surgery
could not only train personnel but would be able to investigate clinical practices and identify inefficiencies with the benefits of speed, safety, measurability, reproducibility, and reduced cost afforded by advanced simulation. This study set out to test such a virtual environment by modifying the ICE STORM technology created by Lockheed Martin. The objective of this pilot study was to determine the feasibility of creating a VR OR to simulate a standardized surgical crisis scenario and evaluate the simulator for construct and face validity. Our results confirm that the VR simulator was capable of simulating the American College of Surgeons ‘‘Loss of laparoscopic visualization’’ scenario. Metric data revealed that attendings completed the simulation in less time, thus confirming construct validity. Attendings and trainees liked the simulator, felt that it was realistic, easy to communicate with, and similar to other non–computer-based, communication-based training modules, thus confirming face validity. However, they did not believe it would improve their performance in the OR, possibly because of the short simulation session. If the simulation incorporated the entire laparoscopic troubleshooting scenario, it is possible that participants would feel that their experience was more useful in their preparation for the OR. Furthermore, participants did not want to complete the VR training before going to the OR. Again this may be because of the limited scope of the pilot simulation. It also may reflect the current lack of experience and acceptance using VR simulation. This will likely require a culture change once VR simulation gains more traction. Using the Bedford Workload Scale, we found that the simulator did not create excessive workload for participants, allowing spare capacity for additional tasks. Finally, using the Modified NASA-TLX scale, we showed that the simulator was neither significantly mentally or physically demanding nor exceedingly frustrating or stressful to use. There are several limitations of this study. First, our sample size is low, and further research is needed with more participants. Second, although the simulator is equipped to accommodate multiple participants simultaneously, this study only investigated one participant at a time. The benefit of having multiple participants present at once is clear as surgeons, anesthesiologists, and nurses would be able to interact with each other in real time to troubleshoot common OR problems. This would force the team to establish a shared goal and shared mental model of the situation, maintain situational awareness as the situation evolves, and communicate with each other professionally so that all members of the team may contribute thoughts in the best interest of the patient. By simplifying the simulation in its current form, we might be overestimating the feasibility and ease of use. Future models will address these limitations by incorporating multiple participants in each scenario. Only then can VR simulation be compared directly with current live simulations to determine if VR is as effective in team training.
589 Eliminating the human proctor tool is also necessary to achieve a program that would meet the demands of today’s training needs by reducing the workload required of those institutions performing the training. The scenario would still need to be programmed to maximize learning by evolving such that the participants would be forced to attempt all possible solutions to successfully complete the simulation. Furthermore, this would contribute to create a fully simulated environment allowing participants to use the system in remote locations, possibly achieving a completely immersive ‘‘holodeck,’’ like that seen in ‘‘Star Trek.’’ As this is a pilot study, we opted to simulate a relatively simple scenario. Certainly future studies would need to simulate more complex and stressful scenarios to fully demonstrate the utility of an integrated VR system as a training instrument. Finally, a key component to team training simulation is automatic debriefing of participants after the simulation. This will culminate in fully functional team training environment with widespread appeal. We conclude that it is feasible to simulate the OR environment using VR and replicate a standardized surgical crisis scenario. This VR simulation serves as a proof-ofconcept for construct and face validity; however, more study is necessary to increase the capability of the simulation and apply it to the entire OR team.
Supplementary data Supplementary data related with this article can be found at http://dx.doi.org/10.1016/j.amjsurg.2015.01.024
References 1. Al Jamal YN, Ali SM, Ruparel RK, et al. The rationale for combining an online audiovisual curriculum with simulation to better educate general surgery trainees. Surgery 2014;156:723–8. 2. Araujo SE, Delaney CP, Seid VE, et al. Short-duration virtual reality simulation training positively impacts performance during laparoscopic colectomy in animal model: results of a single-blinded randomized trial: VR warm-up for laparoscopic colectomy. Surg Endosc 2014; 28:2547–54. 3. Brooks KD, Acton RD, Hemesath K, et al. Surgical skills acquisition: performance of students trained in a rural longitudinal integrated clerkship and those from a traditional block clerkship on a standardized examination using simulated patients. J Surg Educ 2014;71:246–53. 4. Fernandez GL, Page DW, Coe NP, et al. Boot cAMP: educational outcomes after 4 successive years of preparatory simulation-based training at onset of internship. J Surg Educ 2012;69:242–8. 5. Gurusamy KS, Aggarwal R, Palanivelu L, et al. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev; 2009:CD006575. 6. Khatib M, Hald N, Brenton H, et al. Validation of open inguinal hernia repair simulation model: a randomized controlled educational trial. Am J Surg 2014;208:295–301. 7. Nagendran M, Gurusamy KS, Aggarwal R, et al. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev; 2013:CD006575.
590
The American Journal of Surgery, Vol 210, No 3, September 2015
8. Schell SR, Flynn TC. Web-based minimally invasive surgery training: competency assessment in PGY 1-2 surgical residents. Curr Surg 2004;611:120–4. 9. Shetty S, Zevin B, Grantcharov TP, et al. Perceptions, training experiences, and preferences of surgical residents toward laparoscopic simulation training: a resident survey. J Surg Educ 2014;71:727–33. 10. Varas J, Mejı´a R, Riquelme A, et al. Significant transfer of surgical skills obtained with an advanced laparoscopic training program to a laparoscopic jejunojejunostomy in a live porcine model: feasibility of learning advanced laparoscopy in a general surgery residency. Surg Endosc 2012;26:3486–94. 11. Villamaria CY, Eliason JL, Napolitano LM, et al. Endovascular Skills for Trauma and Resuscitative Surgery (ESTARS) course: curriculum development, content validation, and program assessment. J Trauma Acute Care Surg 2014;76:929–35. 12. Zendejas B, Cook DA, Bingener J, et al. Simulation-based mastery learning improves patient outcomes in laparoscopic inguinal hernia repair: a randomized controlled trial. Ann Surg 2011;254:502–9. 13. Henry B, Clark P, Sudan R. Cost and logistics of implementing a tissue-based American College of Surgeons/Association of Program Directors in Surgery surgical skills curriculum for general surgery residents of all clinical years. Am J Surg 2014;207:201–8. 14. Krajewski A, Filippa D, Staff I, et al. Implementation of an intern boot camp curriculum to address clinical competencies under the new Accreditation Council for Graduate Medical Education supervision requirements and duty hour restrictions. JAMA 2013;148: 727–32. 15. Robinson WP, Baril DT, Taha O, et al. Simulation-based training to teach open abdominal aortic aneurysm repair to surgical residents requires dedicated faculty instruction. J Vasc Surg 2013;58:247–53. 16. Dunnington GL, MacRae H, DaRosa DA, et al., ACS/APDS Surgery Resident Skills Curriculum. Available at: https://www.facs.org/ education/program/apds-resident. Accessed October 21, 2014. 17. Aggarwal R, Undre S, Moore K, et al. The simulated operating theatre: comprehensive training for surgical teams. Qual Saf Health Care 2004; 13:i27–32. 18. Moorthy K, Munz Y, Forrest D, et al. Surgical crisis management skills training and assessment: a simulation [corrected]-based approach to enhancing operating room performance. Ann Surg 2006;244:139–47. 19. Undre S, Koutantji M, Sevdalis N, et al. Multidisciplinary crisis simulations: the way forward for training surgical teams. World J Surg 2007;31:1843. 20. Arriaga AF, Gawande AA, Raemer DB, et al. Pilot testing of a model for insurer-driven, large-scale multicenter simulation training for operating room teams. Ann Surg 2014;259:403–10. 21. Paige J, Kozmenko V, Morgan B, et al. From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation. J Surg Educ 2007;64:369–77. 22. Steinemann S, Berg B, Skinner A, et al. In situ, multidisciplinary, simulation-based teamwork training improves early trauma care. J Surg Educ 2011;68:472–7. 23. Phitayakorn R, Minehart R, Pian-Smith MC, et al. Practicality of intraoperative teamwork assessments. J Surg Res 2014;190:22–8. 24. Arain NA, Hogg DC, Gala RB, et al. Construct and face validity of the American College of Surgeons/Association of Program Directors in Surgery laparoscopic troubleshooting team training exercise. Am J Surg 2012;203:54–62. 25. Maithel S, Sierra R, Korndorffer J, et al. Construct and face validity of MIST-VR, Endotower, and CELTS: are we ready for skills assessment using simulators? Surg Endosc 2006;20:104–12. 26. Robbins NB, Heiberger RM. Plotting Likert and Other Rating Scales. Joint Statistical Meetings. Available at: https://www.amstat.org/ sections/SRMS/Proceedings/y2011/Files/300784_64164.pdf; 2011. Accessed October 10, 2014. 27. Roscoe AH, Ellis GA. A Subjective Rating Scale Assessing Pilot Workload in Flight: A Decade of Practical Use. Royal Aerospace Establishment Technical Report 90019. Available at: http://www.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40. 41.
42. 43. 44.
45.
46.
47.
48.
49.
dtic.mil/cgi-bin/GetTRDoc?AD5ADA227864; 1990. Accessed September 20, 2014. Carswell CM, Lio CH, Grant R, et al. Hands-free administration of subjective workload scales: acceptability in a surgical training environment. Appl Ergon 2010;42:138–45. Agarwal HS, Saville BR, Slayton JM, et al. Standardized postoperative handover process improves outcomes in the intensive care unit: a model for operational sustainability and improved team performance. Crit Care Med 2012;40:2109–15. Davenport DL, Henderson WG, Mosca CL, et al. Risk-adjusted morbidity in teaching hospitals correlates with reported levels of communication and collaboration on surgical teams but not with scale measures of teamwork climate, safety climate, or working conditions. J Am Coll Surg 2007;205:778–84. Knaus WA, Draper EA, Wagner DP, et al. An evaluation of outcome from intensive care in major medical centers. Ann Int Med 1986; 104:410–8. Shortell SM, Zimmerman JE, Rousseau DM, et al. The performance of intensive care units: does good management make a difference? Med Care 1994;32:508–25. Gittell JH, Fairfield KM, Bierbaum B, et al. Impact of relational coordination on quality of care, postoperative pain and functioning, and length of stay. Med Care 2000;38:807–19. Pandian V, Miller CR, Mirski MA, et al. Multidisciplinary team approach in the management of tracheostomy patients. Otolaryngol Head Neck Surg 2012;147:684–91. Suarez JI, Zaidat OO, Suri MF, et al. Length of stay and mortality in neurocritically ill patients: impact of a specialized neurocritical care team. Crit Care Med 2004;32:2311–7. Young GJ, Charns MP, Daley J, et al. Best practices for managing surgical services: the role of coordination. Health Care Manage Rev 1997;22:72–81. Chopra V, Bovill JG, Spierdijk J, et al. Reported significant observations during anaesthesia: a prospective analysis over an 18-month period. Br J Anaesth 1992;68:13–7. Carthey J, de Leval MR, Reason JT. The human factor in cardiac surgery: errors and near misses in a high technology medical domain. Ann Thorac Surg 2001;72:300–5. Paige JT, Aaron DL, Yang T, et al. Improved operating room teamwork via SAFETY prep: a rural community hospital’s experience. World J Surg 2009;33:1181–7. Einav Y, Gopher D, Kara I, et al. Preoperative briefing in the operating room: shared cognition, teamwork, and patient safety. Chest 2010;137:443–9. Sexton J, Helmreich R, Thomas E. Error, stress and teamwork in medicine and aviation: cross sectional surveys. BMJ 2000;320: 745–9. Flin R, Yule S, McKenzie L, et al. Attitudes to teamwork and safety in the operating theatre. Surgeon 2006;4:145–51. Undre S, Sevdalis N, Healey AN, et al. Teamwork in the operating theatre: cohesion or confusion? J Eval Clin Pract 2006;12:182–9. Helmreich RL, Merritt AC, Wilhelm JA. The evolution of crew resource management training in commercial aviation. Int J Aviat Psychol 1999;9:19–32. Helmreich RL, Wilhelm J, Gregorich S, et al. Preliminary results from evaluation of cockpit resource management training: performance ratings of flight crews. Aviat Space Environ Med 1990;61:576–9. Gaba DM, Howard SK, Flanagan B, et al. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998;89:8–18. Holzman RS, Cooper JB, Gaba DM, et al. Anesthesia crisis resource management: real-life simulation training in operating room crises. J Clin Anesth 1995;7:675–87. Matveevskii AS, Gravenstein N. Role of simulators, educational programs, and nontechnical skills in anesthesia resident selection, education, and competency assessment. J Crit Care 2008;23:167–72. Yee B, Naik VN, Joo HS, et al. Nontechnical skills in anesthesia crisis management with repeated exposure to simulation-based education. Anesthesiology 2005;103:241–8.
Surgical Education
Virtual operating room for team training in surgery Jonathan S. Abelson, M.D.a,*, Elliott Silverman, P.A.a, Jason Banfelderb, Alexandra Naidesb, Ricardo Costaa, Gregory Dakin, M.D.a a
Department of Surgery, New York Presbyterian HospitaldWeill Cornell Medical College, New York, NY 10068, USA; bDepartment of Physiology and Biophysics, Weill Cornell Medical College, New York, NY, USA KEYWORDS: Virtual reality simulation; Team training; Surgery; Operating room
Abstract BACKGROUND: We proposed to develop a novel virtual reality (VR) team training system. The objective of this study was to determine the feasibility of creating a VR operating room to simulate a surgical crisis scenario and evaluate the simulator for construct and face validity. METHODS: We modified ICE STORM (Integrated Clinical Environment; Systems, Training, Operations, Research, Methods), a VR-based system capable of modeling a variety of health care personnel and environments. ICE STORM was used to simulate a standardized surgical crisis scenario, whereby participants needed to correct 4 elements responsible for loss of laparoscopic visualization. The construct and face validity of the environment were measured. RESULTS: Thirty-three participants completed the VR simulation. Attendings completed the simulation in less time than trainees (271 vs 201 seconds, P 5 .032). Participants felt the training environment was realistic and had a favorable impression of the simulation. All participants felt the workload of the simulation was low. CONCLUSIONS: Creation of a VR-based operating room for team training in surgery is feasible and can afford a realistic team training environment. Ó 2015 Elsevier Inc. All rights reserved.
The benefits of simulation in surgery are well documented, allowing trainees to achieve proficiency in shorter times, acquire news skills, and retain skills, all in environments that are inexpensive, reproducible, and safe.1–12 With the abundance of evidence to support simulation, surgical residency programs have rapidly adapted inanimate training
This study was funded in part by Lockheed Martin. * Corresponding author. Tel.: 11-914-980-4530; fax: 11-212-7465236. E-mail address: [email protected] Manuscript received November 26, 2014; revised manuscript January 7, 2015 0002-9610/$ - see front matter Ó 2015 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.amjsurg.2015.01.024
into their curricula, with most programs staffing full-time simulation centers.4,13–15 The American College of Surgeons has recognized the value of simulation in surgery and has developed an extensive accreditation program for simulation centers. Most of the focus in surgical simulation has been on task training of surgical skills, ranging from knot tying to chesttube insertion to flexible endoscopy. Perhaps the most salient example is laparoscopic surgical skill training, whereby a systematic course of box trainer–based laparoscopic tasks has proved so effective, and the course has been mandated for certification by the American Board of Surgery.16 However, technical skill is only one aspect of being an effective
586
The American Journal of Surgery, Vol 210, No 3, September 2015
surgeon. Furthermore, the surgeon is only one part of the team necessary to deliver effective surgical care. Successfully performing an operation requires the coordinated teamwork of the surgeon, anesthesiologist, nurses, hospital staff, and clinical information systems. Recognizing the importance of teamwork in the operating room (OR) has led educators to move from the relatively simplistic training of surgical tasks to the more complicated world of whole team training in the surgical environment.17–19 Team training in surgery involves creation of a simulated OR, in which combinations of real equipment coupled with mannequins and computerized integration allow a human team to recreate real-life operative scenarios. The American College of Surgeons has created a series of standardized OR situations that can be used in training.16 Several groups have reported success with multidisciplinary OR simulation.20–22 Others although have had difficulty executing such simulation secondary to cost and space requirements.13,23 Furthermore, with limited duty hours, it can be difficult to assemble the necessary team members to undergo the training. These factors have limited the widespread use of team training simulations in surgery compared with the relatively simple task training that has been widely adapted. We hypothesized that VR software can offer realistic team training environments that overcome some of the current limitations. Therefore, we proposed to create a multiuser interactive environment in which both team-based skill coupled with surgical decision making can be simulated, critiqued, and evaluated. Lockheed Martin Corporation (Oswego, NY), well known for its military-based simulation and training programs, has developed a VR-based environment for use in medical training called ICE STORM (Integrated Clinical Environment; Systems, Training, Operations, Research, Methods). The objective of this pilot study was to determine the feasibility of modifying the ICE STORM VR OR to simulate a standardized surgical crisis scenario and evaluate the simulator for construct and face validity.24,25
Methods The existing ICE STORM platform contains all the equipment and personnel necessary to simulate a variety of scenarios in a virtual OR. A core team of researchers from both Lockheed Martin and Weill Cornell Medical College (New York, NY) was established to modify ICE STORM to simulate an intraoperative crisis scenario. In addition to modeling the necessary elements of this crisis scenario, additional software was created to allow a human proctor to serve as an interface between study participants and the virtual world. The interface software used a standard iPad (Apple Inc., Cupertino, CA) to allow a human proctor to modify the simulation environment as necessary during the course of the training exercise, depending on the participant’s responses.
The laparoscopic troubleshooting module is a team training scenario published by the American College of Surgeons and has been previously validated.16 The full module is beyond the scope of this pilot project and entails a comprehensive team of surgeon, assistant surgeon, nurses, and anesthesiologist working through several crisis situations while performing a standard laparoscopic operation. In this project, the focus was narrowed to a small portion of the module, called ‘‘loss of laparoscopic visualization.’’ This scenario was then programmed into the modified ICE STORM platform. In the ‘‘loss of laparoscopic visualization’’ scenario, the team was performing a laparoscopic cholecystectomy and the laparoscopic monitor suddenly went dim. It was the job of the operating surgeon to troubleshoot the problem and return the monitor to working form. There were several possible problems that the participant, functioning as the operating surgeon, had to identify and check to restore function to the monitor. Each participant was required to perform 4 mandatory treatments: (1) check camera box and cord; (2) check light-source box and cord; (3) check (clean/inspect) or replace laparoscope (use spare); and (4) exchange and use spare camera. Participants were evaluated by time to completion of the simulation. Participants were given a ‘‘pass’’ if they completed all mandatory treatments in less than 270 seconds. This number was based on preliminary analysis of test subjects. Participants, acting as the surgeon, interacted with the VR world using the Gyration Air Mouse (SMK-Link Electronics, Camarillo, CA; Supplementary Fig. 1) to manipulate the surgeon avatar and interacted with the human proctor administrating the study by simply speaking aloud. The proctor then used the proctor-tool software on a standard iPad (Apple Inc.) to make modifications to the VR world (eg, participant instructs nurse to replace the laparoscope, the proctor enters the appropriate command, and then the nurse avatar replaces the laparoscope). Successful completion of the module was achieved when the participant identified all 4 of the critical elements that could lead to loss of visualization. No matter what order a participant identified the element, the simulation would not end until all 4 components were identified and correctly acted on. This concept, known as the ‘‘full cycle test,’’ ensured that all participants would demonstrate a complete understanding of the most critical elements of the troubleshooting scenario and eliminated the possibility that the participant could end the simulation simply by picking the ‘‘correct’’ cause of failure on the first try. Metric data from the simulation exercise were used to evaluate the construct validity of the virtual system. Three methods were used to determine the face validity: Likert scale questionnaires (Table 1), the Bedford Workload Scale (Supplementary Fig. 2), and the modified NASA-Task Load Index (NASA-TLX) scale (Supplementary Fig. 3). Subjects included attending surgeons (experts) and residents and medical students combined (trainees). Statistical analysis was conducted using SPSS 12.0 statistical software (IBM,
J.S. Abelson et al. Table 1
Virtual OR for team training in surgery
587
Likert scale results
Likert scale question number
Median
P value*
1. 2. 3. 4. 5. 6. 7.
5 4 1 4 3 3 4
.014 .534 .000 .073 .001 .001 .911
5 2 2 4
.000 .000 .000 .607
The training environment was realistic† This course will improve my performance in the OR† How would you rate your prior virtual world?‡ How would you rate your surgical experience level?x How often do you use surgical simulation training modules?‡ How would you rate your experience with non-surgical virtual world?x The VR training module differed greatly from other non–computer-based communication training modules† 8. I liked the VR training environment† 9. I did not like the VR training environment† 10. I found it difficult to communicate using the VR training environment† 11. I would like to do VR training prior to going to the OR†
*One-sample Whitney–Wilcoxon signed rank test value to determine if the median score was different from the midpoint of the scale, which was 4. † Scoring system: 1 5 strongly disagree, 4 5 moderately agree, 7 5 strongly agree. ‡ Scoring system: 1 5 never, 4 5 moderate, 7 5 very often. x Scoring system: 1 5 novice, 4 5 moderate, 7 5 expert.
New York, NY). Specific modalities used were Fisher exact test, ANOVA, and t tests. Metric data were analyzed using the Fisher exact test comparing pass–fail rates and time to completion because the sample size was low. Seven-point Likert scale questionnaires were used to assess a variety of opinions, including participants’ impression of the virtual environment with regard to realism, beliefs on whether the system would improve performance in the OR, and inclination to use the system in the future26 (Table 1). Median scores were used to determine a central tendency. Independent-sample Mann–Whitney test was used to detect any difference in the median scores between trainees and attendings. We also performed a 1-sample Whitney–Wilcoxon signed rank test comparing median scores against the midpoint of the evaluation scale (ie, 4). The aim of this analysis was to reveal whether the trainees’ evaluations were above the midpoint of the scale (ie, positive) or below it (ie, negative).19 The Bedford Workload scale is an unidimensional rating scale designed to identify operator’s spare mental capacity while completing a task.27 The single dimension is assessed using a hierarchical decision tree. Supplementary Fig. 2 explains the scoring system from workload 1 to 10. Independent-sample Mann–Whitney test was used to detect any difference between workload scores of attendings vs trainees. One-sample Wilcoxon signed rank test was calculated to understand any difference between the scores of all participants against the midpoint of the scale, which was 5. The modified NASA-TLX is a subjective, multidimensional, validated assessment tool that rates perceived workload on 6 different subscales: Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration.28 Results are combined but not weighted. Supplementary Fig. 3 reveals each category investigated and the scaling system. An independent-sample Mann– Whitney test was used to detect any difference between workload scores of attendings vs trainees. The 1-sample
Wilcoxon signed rank test was calculated to understand any difference between the scores of all participants against the midpoint of the scale, which was 10.
Results A total of 33 participants, including 26 trainees and 7 attendings, completed the virtual simulation. Attendings completed the simulation is less time than trainees (201 vs 271 seconds, P 5 .032; Fig. 1). A higher percentage of attendings passed the simulation compared with trainees although this was not statistically different (86% vs 58%, P 5 .223). Eleven questions were included in the Likert scale questionnaire. Those questions that pertain directly to opinions on the VR simulation are reviewed here (questions 1, 2, and 7 to 11). Those questions that addressed background information on participants’ surgical experience or previous virtual world experience are not discussed in this section (questions 3 to 6; Table 1). Overall, participants agreed that they liked the simulator (median 5 5, P 5 .000) and disagreed with the statement that they disliked the simulator (median 5 2, P 5 .000). Study participants felt that the training environment was realistic (median 5 5, P 5 .014) and did not feel that communication in the environment was difficult (median 5 2, P 5 .000). However, participants did not feel that the simulation would improve their performance (median 5 4, P 5 .534). Furthermore, they did not agree that they would like to do VR training before going to the OR (median 5 4, P 5 .607). Using the Bedford Workload scale, 82% of all participants felt that the workload was either low or that they had enough spare capacity for desirable additional tasks (Fig. 2). Using the modified NASA-TLX scale, all
588
The American Journal of Surgery, Vol 210, No 3, September 2015
Figure 1 Average simulation completion times, attendings vs trainees. *P 5 .032.
participants were found to have minimal mental, physical, and temporal demand. Likewise, none of the participants reported requiring a high amount of effort to complete the simulation (Fig. 3). Finally, there was no statistically significant difference in responses between attendings and trainees for all responses in the Likert scale, Bedford Workload scale, and Modified NASA-TLX scale.
Comments The OR is a dynamic, high-risk environment where successful delivery of care depends on the co-ordinated action of surgeons, anesthesiologists, nurses, hospital staff, and clinical information. With the increasing complexity of surgical instrumentation and patient disease, efficient OR processes and teamwork will become more important. The value of teamwork among medical professionals is well documented. Failures in co-ordination and communication of information among hospital clinicians have been associated with worse outcomes,29–31 longer lengths of stay, and higher nurse turnover in intensive care units32 and greater postoperative pain with lower functioning levels for
Figure 2
Bedford workload scale: all participants.
patients33–35 across specialties. Surgical teams at the Department of Veterans Affairs hospitals with low mortality rates communicate more effectively and more often than surgical teams associated with high mortality rates.36 In a study of anesthetic-related errors, 80% of occurrences were considered preventable with human error accounting for 75% of them.37 Studies in cardiac surgery show the clear impact of human factors in ‘‘near misses’’ in the high technology OR.38 Primary teamwork competencies, including knowledge, skill, and attitudes, positively correlate to effective teamwork and patient safety.39,40 However, these competencies of effective teamwork are often lacking in the modern OR. Surveys evaluating surgeons working with anesthesiologists have suggested that substandard collaboration occurs 50% of the time.41 Other studies have discovered discrepancy in OR team members’ views on appropriate team structure and importance of effective communication.42,43 The aviation industry noted years ago that 70% of errors were because of preventable human factors, such as failed interpersonal communication, decision making, and leadership.44 This led to the development of crew resource management training programs that use simulator-based training to understand the limitations of human performance and to develop a culture of safety.45 Anesthesiologists have addressed human factors in errors by developing simulators to train staff during anesthetic crises, allowing participants to integrate technical and team training skills with feedback on their performance.46–49 Given the importance of teamwork and communication in the OR, several groups are investigating the role of full-scale OR simulation to both study and improve these skills.17,21,22 The ultimate goals of such simulation include not only training of participants but identification of factors important to overall team performance and improved efficiency of OR systems. In this study, we created an interactive environment to simulate both team-based skills and surgical decision making. This virtual environment might ultimately enable simulation of a variety of OR circumstances, from patient transfer, to hand-off, to intraoperative crisis scenarios. A multiuser virtual world could ultimately be linked to operable surgical procedure simulation platforms to allow full-scale simulation of both surgical procedures and team dynamics. Once a complex OR system is modeled, we
Figure 3
Modified NASA-TLX: all participants.
J.S. Abelson et al.
Virtual OR for team training in surgery
could not only train personnel but would be able to investigate clinical practices and identify inefficiencies with the benefits of speed, safety, measurability, reproducibility, and reduced cost afforded by advanced simulation. This study set out to test such a virtual environment by modifying the ICE STORM technology created by Lockheed Martin. The objective of this pilot study was to determine the feasibility of creating a VR OR to simulate a standardized surgical crisis scenario and evaluate the simulator for construct and face validity. Our results confirm that the VR simulator was capable of simulating the American College of Surgeons ‘‘Loss of laparoscopic visualization’’ scenario. Metric data revealed that attendings completed the simulation in less time, thus confirming construct validity. Attendings and trainees liked the simulator, felt that it was realistic, easy to communicate with, and similar to other non–computer-based, communication-based training modules, thus confirming face validity. However, they did not believe it would improve their performance in the OR, possibly because of the short simulation session. If the simulation incorporated the entire laparoscopic troubleshooting scenario, it is possible that participants would feel that their experience was more useful in their preparation for the OR. Furthermore, participants did not want to complete the VR training before going to the OR. Again this may be because of the limited scope of the pilot simulation. It also may reflect the current lack of experience and acceptance using VR simulation. This will likely require a culture change once VR simulation gains more traction. Using the Bedford Workload Scale, we found that the simulator did not create excessive workload for participants, allowing spare capacity for additional tasks. Finally, using the Modified NASA-TLX scale, we showed that the simulator was neither significantly mentally or physically demanding nor exceedingly frustrating or stressful to use. There are several limitations of this study. First, our sample size is low, and further research is needed with more participants. Second, although the simulator is equipped to accommodate multiple participants simultaneously, this study only investigated one participant at a time. The benefit of having multiple participants present at once is clear as surgeons, anesthesiologists, and nurses would be able to interact with each other in real time to troubleshoot common OR problems. This would force the team to establish a shared goal and shared mental model of the situation, maintain situational awareness as the situation evolves, and communicate with each other professionally so that all members of the team may contribute thoughts in the best interest of the patient. By simplifying the simulation in its current form, we might be overestimating the feasibility and ease of use. Future models will address these limitations by incorporating multiple participants in each scenario. Only then can VR simulation be compared directly with current live simulations to determine if VR is as effective in team training.
589 Eliminating the human proctor tool is also necessary to achieve a program that would meet the demands of today’s training needs by reducing the workload required of those institutions performing the training. The scenario would still need to be programmed to maximize learning by evolving such that the participants would be forced to attempt all possible solutions to successfully complete the simulation. Furthermore, this would contribute to create a fully simulated environment allowing participants to use the system in remote locations, possibly achieving a completely immersive ‘‘holodeck,’’ like that seen in ‘‘Star Trek.’’ As this is a pilot study, we opted to simulate a relatively simple scenario. Certainly future studies would need to simulate more complex and stressful scenarios to fully demonstrate the utility of an integrated VR system as a training instrument. Finally, a key component to team training simulation is automatic debriefing of participants after the simulation. This will culminate in fully functional team training environment with widespread appeal. We conclude that it is feasible to simulate the OR environment using VR and replicate a standardized surgical crisis scenario. This VR simulation serves as a proof-ofconcept for construct and face validity; however, more study is necessary to increase the capability of the simulation and apply it to the entire OR team.
Supplementary data Supplementary data related with this article can be found at http://dx.doi.org/10.1016/j.amjsurg.2015.01.024
References 1. Al Jamal YN, Ali SM, Ruparel RK, et al. The rationale for combining an online audiovisual curriculum with simulation to better educate general surgery trainees. Surgery 2014;156:723–8. 2. Araujo SE, Delaney CP, Seid VE, et al. Short-duration virtual reality simulation training positively impacts performance during laparoscopic colectomy in animal model: results of a single-blinded randomized trial: VR warm-up for laparoscopic colectomy. Surg Endosc 2014; 28:2547–54. 3. Brooks KD, Acton RD, Hemesath K, et al. Surgical skills acquisition: performance of students trained in a rural longitudinal integrated clerkship and those from a traditional block clerkship on a standardized examination using simulated patients. J Surg Educ 2014;71:246–53. 4. Fernandez GL, Page DW, Coe NP, et al. Boot cAMP: educational outcomes after 4 successive years of preparatory simulation-based training at onset of internship. J Surg Educ 2012;69:242–8. 5. Gurusamy KS, Aggarwal R, Palanivelu L, et al. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev; 2009:CD006575. 6. Khatib M, Hald N, Brenton H, et al. Validation of open inguinal hernia repair simulation model: a randomized controlled educational trial. Am J Surg 2014;208:295–301. 7. Nagendran M, Gurusamy KS, Aggarwal R, et al. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev; 2013:CD006575.
590
The American Journal of Surgery, Vol 210, No 3, September 2015
8. Schell SR, Flynn TC. Web-based minimally invasive surgery training: competency assessment in PGY 1-2 surgical residents. Curr Surg 2004;611:120–4. 9. Shetty S, Zevin B, Grantcharov TP, et al. Perceptions, training experiences, and preferences of surgical residents toward laparoscopic simulation training: a resident survey. J Surg Educ 2014;71:727–33. 10. Varas J, Mejı´a R, Riquelme A, et al. Significant transfer of surgical skills obtained with an advanced laparoscopic training program to a laparoscopic jejunojejunostomy in a live porcine model: feasibility of learning advanced laparoscopy in a general surgery residency. Surg Endosc 2012;26:3486–94. 11. Villamaria CY, Eliason JL, Napolitano LM, et al. Endovascular Skills for Trauma and Resuscitative Surgery (ESTARS) course: curriculum development, content validation, and program assessment. J Trauma Acute Care Surg 2014;76:929–35. 12. Zendejas B, Cook DA, Bingener J, et al. Simulation-based mastery learning improves patient outcomes in laparoscopic inguinal hernia repair: a randomized controlled trial. Ann Surg 2011;254:502–9. 13. Henry B, Clark P, Sudan R. Cost and logistics of implementing a tissue-based American College of Surgeons/Association of Program Directors in Surgery surgical skills curriculum for general surgery residents of all clinical years. Am J Surg 2014;207:201–8. 14. Krajewski A, Filippa D, Staff I, et al. Implementation of an intern boot camp curriculum to address clinical competencies under the new Accreditation Council for Graduate Medical Education supervision requirements and duty hour restrictions. JAMA 2013;148: 727–32. 15. Robinson WP, Baril DT, Taha O, et al. Simulation-based training to teach open abdominal aortic aneurysm repair to surgical residents requires dedicated faculty instruction. J Vasc Surg 2013;58:247–53. 16. Dunnington GL, MacRae H, DaRosa DA, et al., ACS/APDS Surgery Resident Skills Curriculum. Available at: https://www.facs.org/ education/program/apds-resident. Accessed October 21, 2014. 17. Aggarwal R, Undre S, Moore K, et al. The simulated operating theatre: comprehensive training for surgical teams. Qual Saf Health Care 2004; 13:i27–32. 18. Moorthy K, Munz Y, Forrest D, et al. Surgical crisis management skills training and assessment: a simulation [corrected]-based approach to enhancing operating room performance. Ann Surg 2006;244:139–47. 19. Undre S, Koutantji M, Sevdalis N, et al. Multidisciplinary crisis simulations: the way forward for training surgical teams. World J Surg 2007;31:1843. 20. Arriaga AF, Gawande AA, Raemer DB, et al. Pilot testing of a model for insurer-driven, large-scale multicenter simulation training for operating room teams. Ann Surg 2014;259:403–10. 21. Paige J, Kozmenko V, Morgan B, et al. From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation. J Surg Educ 2007;64:369–77. 22. Steinemann S, Berg B, Skinner A, et al. In situ, multidisciplinary, simulation-based teamwork training improves early trauma care. J Surg Educ 2011;68:472–7. 23. Phitayakorn R, Minehart R, Pian-Smith MC, et al. Practicality of intraoperative teamwork assessments. J Surg Res 2014;190:22–8. 24. Arain NA, Hogg DC, Gala RB, et al. Construct and face validity of the American College of Surgeons/Association of Program Directors in Surgery laparoscopic troubleshooting team training exercise. Am J Surg 2012;203:54–62. 25. Maithel S, Sierra R, Korndorffer J, et al. Construct and face validity of MIST-VR, Endotower, and CELTS: are we ready for skills assessment using simulators? Surg Endosc 2006;20:104–12. 26. Robbins NB, Heiberger RM. Plotting Likert and Other Rating Scales. Joint Statistical Meetings. Available at: https://www.amstat.org/ sections/SRMS/Proceedings/y2011/Files/300784_64164.pdf; 2011. Accessed October 10, 2014. 27. Roscoe AH, Ellis GA. A Subjective Rating Scale Assessing Pilot Workload in Flight: A Decade of Practical Use. Royal Aerospace Establishment Technical Report 90019. Available at: http://www.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40. 41.
42. 43. 44.
45.
46.
47.
48.
49.
dtic.mil/cgi-bin/GetTRDoc?AD5ADA227864; 1990. Accessed September 20, 2014. Carswell CM, Lio CH, Grant R, et al. Hands-free administration of subjective workload scales: acceptability in a surgical training environment. Appl Ergon 2010;42:138–45. Agarwal HS, Saville BR, Slayton JM, et al. Standardized postoperative handover process improves outcomes in the intensive care unit: a model for operational sustainability and improved team performance. Crit Care Med 2012;40:2109–15. Davenport DL, Henderson WG, Mosca CL, et al. Risk-adjusted morbidity in teaching hospitals correlates with reported levels of communication and collaboration on surgical teams but not with scale measures of teamwork climate, safety climate, or working conditions. J Am Coll Surg 2007;205:778–84. Knaus WA, Draper EA, Wagner DP, et al. An evaluation of outcome from intensive care in major medical centers. Ann Int Med 1986; 104:410–8. Shortell SM, Zimmerman JE, Rousseau DM, et al. The performance of intensive care units: does good management make a difference? Med Care 1994;32:508–25. Gittell JH, Fairfield KM, Bierbaum B, et al. Impact of relational coordination on quality of care, postoperative pain and functioning, and length of stay. Med Care 2000;38:807–19. Pandian V, Miller CR, Mirski MA, et al. Multidisciplinary team approach in the management of tracheostomy patients. Otolaryngol Head Neck Surg 2012;147:684–91. Suarez JI, Zaidat OO, Suri MF, et al. Length of stay and mortality in neurocritically ill patients: impact of a specialized neurocritical care team. Crit Care Med 2004;32:2311–7. Young GJ, Charns MP, Daley J, et al. Best practices for managing surgical services: the role of coordination. Health Care Manage Rev 1997;22:72–81. Chopra V, Bovill JG, Spierdijk J, et al. Reported significant observations during anaesthesia: a prospective analysis over an 18-month period. Br J Anaesth 1992;68:13–7. Carthey J, de Leval MR, Reason JT. The human factor in cardiac surgery: errors and near misses in a high technology medical domain. Ann Thorac Surg 2001;72:300–5. Paige JT, Aaron DL, Yang T, et al. Improved operating room teamwork via SAFETY prep: a rural community hospital’s experience. World J Surg 2009;33:1181–7. Einav Y, Gopher D, Kara I, et al. Preoperative briefing in the operating room: shared cognition, teamwork, and patient safety. Chest 2010;137:443–9. Sexton J, Helmreich R, Thomas E. Error, stress and teamwork in medicine and aviation: cross sectional surveys. BMJ 2000;320: 745–9. Flin R, Yule S, McKenzie L, et al. Attitudes to teamwork and safety in the operating theatre. Surgeon 2006;4:145–51. Undre S, Sevdalis N, Healey AN, et al. Teamwork in the operating theatre: cohesion or confusion? J Eval Clin Pract 2006;12:182–9. Helmreich RL, Merritt AC, Wilhelm JA. The evolution of crew resource management training in commercial aviation. Int J Aviat Psychol 1999;9:19–32. Helmreich RL, Wilhelm J, Gregorich S, et al. Preliminary results from evaluation of cockpit resource management training: performance ratings of flight crews. Aviat Space Environ Med 1990;61:576–9. Gaba DM, Howard SK, Flanagan B, et al. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998;89:8–18. Holzman RS, Cooper JB, Gaba DM, et al. Anesthesia crisis resource management: real-life simulation training in operating room crises. J Clin Anesth 1995;7:675–87. Matveevskii AS, Gravenstein N. Role of simulators, educational programs, and nontechnical skills in anesthesia resident selection, education, and competency assessment. J Crit Care 2008;23:167–72. Yee B, Naik VN, Joo HS, et al. Nontechnical skills in anesthesia crisis management with repeated exposure to simulation-based education. Anesthesiology 2005;103:241–8.
