International Journal of Surgery 12 (2014) 1266e1272
Contents lists available at ScienceDirect
International Journal of Surgery journal homepage: www.journal-surgery.net
Original research
Medical applications of near-eye display devices: An exploratory study Wolfgang Vorraber a, *, Siegfried Voessner a, Gerhard Stark b, Dietmar Neubacher a, Steven DeMello c, Aaron Bair d a
Graz University of Technology, Department of Engineering- and Business Informatics, Kopernikusgasse 24, 8010 Graz, Austria Hospital of Elisabethinen, Elisabethinergasse 14, 8020 Graz, Austria c University of California Berkeley, Center for Information Technology Research in the Interest of Society, Sutardja Dai Hall, 94720 Berkeley, USA d University of California Davis Medical Center, Department of Emergency Medicine, 2315 Stockton Blvd., 95817 Sacramento, CA, USA b
h i g h l i g h t s We present a framework to identify and categorize use cases for Google Glass. We describe the use of Google Glass during a radiological intervention. An app was developed to project vital physical signs to Google Glass via intranet. Interventionalists reported improved concentration by reduced head movements. However, heat generation by the device and low battery capacity are shortcomings.
a r t i c l e i n f o
a b s t r a c t
Article history: Received 27 June 2014 Received in revised form 16 September 2014 Accepted 29 September 2014 Available online 22 October 2014
Introduction: Near-eye display devices (such as Google Glass) may improve the efficiency and effectiveness of clinical care by giving clinicians information (such as the patient's vital signs) continuously within their field of vision during various procedures. We describe the use of Glass during a radiological intervention in three patients. Other possible applications (including tele-mentoring and the supervision of trainees) are discussed and a classification proposed. Methods: An app was developed to facilitate the use of Glass, so vital physical signs (pulse and blood pressure) could be projected on the near-eye display, via an intranet to protect sensitive data. The device was then used during radiological interventions (percutaneous transluminal angioplasty) in three patients, and assessed by the interventionalists who were interviewed before and after each procedure. Results: The interventionalists reported that Google Glass improved concentration on the task in hand by reducing head and neck movements (which would be needed to view several remote monitors). However, heat generation by the device and low battery capacity are shortcomings for which solutions must be developed, and data protection is mandatory. Conclusion: Google Glass may have a number of clinical applications and can quicken interventions where vital signs or other visual data need to be monitored by the operator. © 2014 Surgical Associates Ltd. Published by Elsevier Ltd. All rights reserved.
Keywords: Process improvement Clinical information service OHMD Near-eye display device Google Glass
1. Introduction A near-eye display device such as Google Glass enables the transmission of information by augmenting visual perception via a projection in the field of vision. This consolidation of information can potentially allow improved situational awareness without distraction from primary tasks. Although near-eye display devices or so-called optical head mounted displays (OHMD) have existed for more than four decades
* Corresponding author. E-mail address: [email protected] (W. Vorraber). http://dx.doi.org/10.1016/j.ijsu.2014.09.014 1743-9191/© 2014 Surgical Associates Ltd. Published by Elsevier Ltd. All rights reserved.
(e.g. [1e3]), the development of Google Glass (from now on we call it Glass) has created excitement about the potential process improvements that might come from using such devices in a clinical setting [4,5]. Some research groups tested video transmission from Glass to a remote audience during surgery [6]. Likewise, some of the first scientific reports on Glass in medical education [7], documentation in forensic medicine [8], videoconferencing and information querying [9] have recently been published. However, there is a potential array of clinical use-case scenarios that have yet to be investigated. Since it seems to be impossible to exhaustively list all specific use cases, we describe process settings where Glass could improve efficiency and effectiveness:
W. Vorraber et al. / International Journal of Surgery 12 (2014) 1266e1272
Surgeons or interventionalists rely on information from various sources (e.g. patients' vital parameters, medical images, patient records, etc.). Due to the fact that almost all data is already in a digital form, it can be easily distributed to various devices. Therefore, if there are at least two, eventually dislocated, information sources needed to perform a task, wearable devices can bridge local gaps and collect data where it is needed. This is especially true in a clinical setting where users frequently need to monitor data (e.g. vital parameters) while performing their core tasks (e.g. interventions or surgeries). Furthermore, Glass can be suitable for information sharing between separated medical staff. This may range from local information sharing (different patient perspectives e e.g. small size of incision) to national and international information sharing (e.g. remote consultations). Our research was guided by the following questions: Where and how might Glass increase efficiency and effectiveness? How can data be protected when using Glass? What are the possible benefits and drawbacks of using Glass in clinical situations? Glass appears to be a suitable platform for user-centered medical information services with rich potential for further development. The publicly available programming interfaces (Glass Development Kit and Mirror API) will likely be fertile ground for further development. Importantly, the Glass Development Kit (GDK) provides developments that allow Glass to work on local networks (rather than the Internet). This is especially important for data privacy, and enables the implementation of secure end-to-end data transfer without third-party access. We suggest the potential uses of Glass include tele-mentoring and training. Of several possible clinical applications we opted to develop a Glass app and a server app to enable Glass to be used during percutaneous transluminal angioplasty (PTA), with feedback from the two interventionalists involved in three such cases. 2. Methods As depicted in Fig. 1, we followed a three-step approach. During the first step, our team identified several potential areas of improvement by the use of near-eye display devices through systematic requirements engineering based on the SOPHIST-Approach [10]. The SOPHIST-Approach is a technique in software engineering, which offers tools to systematically collect requirements that have to be met by a technical system in order to satisfy end users (e.g. physician) needs. For our second step, we developed the classification scheme described in Section 2.1 to identify technical and organizational commonalities and differences of use cases before implementation. Use cases are classified according to the scheme depicted in Table 1. This scheme, which is located at the center of Fig. 1, facilitates abstraction of the identified applications to see that use case stories can be clustered or even be the same from a technical perspective. In a third step, the abstracted use cases are mapped to technical solutions. On the one hand, the abstraction step prevents from reinventing technical solutions for each new user story. On the other hand the abstraction scheme can also be used as a process innovation tool by mapping various property combinations of the scheme to possible new use cases. One especially promising scenario was selected and a technical solution was implemented. After initial lab experiments, this
1267
application framework was tested during three real world interventions (angioplasties) and feedback was collected from the participating interventionalists who wore Glass during the procedure. Similar to [9] we categorize the identified areas of improvements by the use of near-eye display devices into “Virtual Consultations” (bridging of spatial barriers), “Monitoring of real time patient data” (improvement of situational awareness, reduction of change of attention and focus), “Navigation and medical imaging” (reduction of change of attention and focus), “Viewpoint of surgeon for assistance” (improvement of assistance by projection of the viewpoint of surgeons to assistance e.g. in case of small size of incision) and “Teaching” (transmission of the point of view of an experienced expert to students, respectively from an inexperienced student to an expert for remote consultation). In order to facilitate implementation by identifying technical and organizational differences and similarities we propose the categorization scheme described in the following section. 2.1. Scenario categorization Based on process analysis and interviews with medical experts we identified requirements for human-centered information services and formulated use cases. We categorized them according to: Number of users: The number of users per scenario can either be single or multiple. Data dynamics: Data transferred to the near-eye display device can be either static (e.g. electronic health records) or dynamic (e.g. patient vital signs). Collaboration: Determines whether users will or will not collaborate based on the Glass service. Direction of information flow: The direction of the information flow can either be unidirectional or bidirectional. The unidirectional flow can further be refined into information transferred into the near-eye display device (e.g. patient vital signs) and information recorded by the near-eye display device and transferred outbound (in case of a near-eye display device with recording functionality). Mode of operation: The mode of operation can be classified as passive, or active. A passive mode of operation means that the end user does not actively control the near-eye display device during the scenario. Conversely, active mode of operation represents scenarios where users actively control the device (e.g. tapping on an integrated touch pad or using voice commands). Frequency of use: This criterion categorizes the frequency of the selected scenario in the setting investigated. We introduced the following three levels of frequency “High - daily use”, “Medium -weekly use” and “Low e monthly use” Network coverage: Based on the scenario, the network needed for data transmission can be local, regional, national, or international.
2.2. Scenario analysis Detailed process analysis of various interventions revealed that a high degree of multitasking by interventionalists is required. This frequently means having to rely on multiple monitors and displays arranged around the operating theater while concentrating on the fields of operation (see Fig. 2). Such an arrangement requires a change of attention and focus potentially resulting in loss of efficiency. We note that there is room for process improvement by using near-eye display devices to display relevant data directly into the
1268
W. Vorraber et al. / International Journal of Surgery 12 (2014) 1266e1272
Fig. 1. Systematical abstraction of various use cases and required features according to a proposed classification scheme.
Table 1 Criteria for categorizing scenarios of human centered medical information services using a near-eye display device.
line-of-sight resulting in a reduction of the number of head movements and re-localization processes (see Fig. 3). Prior research on near-eye display devices revealed that there was no evidence of negative impact on spatial awareness of the user [11]. We further
postulate that the aggregation or consolidation of data will improve situational awareness of the surgeon. Based on this initial analysis we formulated the following hypothesis that guided our explorative research: Hypothesis: The projection of patient related parameters (e.g. vital signs) and images to surgeons via near-eye display devices consolidates patient data, which leads to increased efficiency of monitoring and improves situational awareness.
Fig. 2. Suboptimally placed monitors (solid line) and fields of operation (dashed line).
In order to explore this hypothesis we selected a case involving percutaneous transluminal angioplasty (PTA). PTA is an intervention that involves placement of an arterial catheter and various wire-guided implements in an effort to open occluded peripheral arteries. The procedure is done using an x-ray monitor to guide the catheter inside the arteries in addition to monitoring of vital signs. Therefore, this scenario requires frequent scanning of multiple monitors and displays. According to our classification scheme defined in Section 2, this scenario can be categorized as shown in Table 1. We implemented a technical solution that projects the patient’s vital signs (oxygen saturation, heart rate, blood pressure and respiratory rate) onto Glass worn by the interventionalist (Figs. 2e6). We gathered feedback for our explorative study from experts using
W. Vorraber et al. / International Journal of Surgery 12 (2014) 1266e1272
1269
Fig. 3. Displaying relevant data directly into the line-of-sight may reduce the number of head movements and re-localization processes (figure from [12]).
Fig. 4. Google Glass specifications and features.
Fig. 6. The original display of the vital sign monitor (in the background) is projected into the interventionalist's near-eye display device. Fig. 5. Technical setup for using Google Glass to display patient vital signs.
Glass in this setting via a qualitative pre- and post-interview. Pre interview questions served to determine the degree of technological affinity of the user. Post interview focused on identifying possible process improvements and drawbacks through the use of
Glass during PTA. The term “process improvement” was narrowed down to economy of time, increase in efficiency (e.g. reduction of the number of re-localization processes), increased focus on core tasks and improved situational awareness. Possible drawbacks through the use of Glass were categorized into negative influence
1270
W. Vorraber et al. / International Journal of Surgery 12 (2014) 1266e1272
on the course of motion, distraction by the service and negative physical influences such as discomfort wearing.
Glass preventing possible synchronization processes with external servers (and potential loss of control of sensitive patient data) if an internet connection were to be inadvertently established.
2.3. Google glass features 3.2. Medical setup The tested version of Glass (Explorer Edition) was introduced in February 2013 and available for selected users at a price of US$ 1500. This version is equipped with an OMAP 4430 SoC CPU, 1 GB RAM, 16 GB storage and weighs approximately 50 g. Fig. 4 illustrates some relevant features provided by the Glass Explorer Edition that open up a broad spectrum of medical applications. For our purposes we focus on the most useful features concerning output, input, sensors and controls. The prism (1) provides a high-resolution display to illustrate images at maximum 640 360 pixel. According to the specifications provided by Google [13], the display “… is the equivalent of a 25 inch high definition screen from eight feet away.” In the used edition the optical unit is coupled with a thin rigid frame. As mentioned in a Glass patent [14], a rotation of the optical unit enables a proper view by the user throughout various positions. Hence, the viewing surface can be brought in the line of sight depending on the users' individual fit. The camera (2) is placed at the front end of Glass and is able to capture videos up to a Quality of 720p and high-resolution pictures of 5 MP. A light sensor (3) is able to react to light changes and determines if the device is currently being worn. Commonly the interaction with Glass is done by voice control via the microphone (3) or by tapping on the device, which could be either a Touch Pad (4) or a button (5). At the back of the ear a speaker (6) is situated. For noisy environments an earplug can be used. 3. Proof of concept As a first proof of concept, we projected the patient's vital signs onto Glass worn by the interventionalist during the procedure. The data was only accessible to the medical interventionalist's team and was transferred solely via a protected local network to ensure data privacy. 3.1. Technical setup Fig. 5 provides an overview of the technical setup. The video feed of the patient's vital signs was transmitted to a standard laptop computer. This video was then streamed to Glass. In order to ensure the patient's safety, a redundant monitor remained next to the surgical field. Pretests revealed that data transmission via video stream results in high power consumption and therefore shortened battery life of Glass. We addressed this by equipping Glass with an external battery pack worn by the interventionalist. The whole technical setup is transportable and was prepared and tested in our laboratory. This setup took an additional five to 10 min to prepare in the operating room. In order to ensure data privacy we implemented our own software solution that limits data transfer to our local intranet. This enables data transfer between Glass and other devices without synchronization with external network resources such as cloud services (i.e. internet connectivity is not required). This solution is based on a client-server concept. The server application manages data transfer from and to Glass and other data sources and sinks (e.g. vital sign display). It is installed on a PC or laptop and is programmed in C# based on.NET 4.5.1. The client application (Glass app) manages data capturing and presentation on Glass. It is installed on Glass (OS version XE 18.11) and is programmed in Java based on Android Development Tools v22.3.0-887826. The Glass app exchanges data exclusively with the server application running in the same secured private network. No data is stored locally on
The patient was a 64-year-old man who presented with chronic claudication (pain in his left leg related to limited arterial blood flow). The lower extremity magnetic resonance angiogram showed a high-grade occlusion of the left distal femoral artery. Prior to the intervention the patient was fully consented for both the procedure and involvement of Glass expanding the monitoring option of the interventionalist. The real world tests were implemented in accordance with Austrian risk, ethics and data privacy guidelines. The percutaneous transluminal angioplasty proceeded via obtaining vascular access by way of the left common femoral artery. A 5 French short sheath (Johnson & Johnson, Miami, FL, USA) was inserted to the left common femoral artery for anterograde approach. Subsequent guidewire passage of the stenosis in the distal femoral artery was achieved and successful guidewire passage dilation was done. Residual stenosis was approximately 30% and good distal flow was observed. 4. Results For our exploratory study, the interventionalists told us what they thought of Glass and its effect on their practice. Importantly, they did not think that the use of Glass had any negative influences on the course of motion, distraction by the device nor any negative physical influences such as discomfort. Feedback from the interventionalists, both of who reported a high degree of technological affinity, suggests the following regarding impact on process improvements: Glass facilitated improved concentration on the main scenario. Notably, the interventionalist fully relied on Glass and did not even look at the backup display during the entire surgery. Glass supported multi-tasking by allowing efficient monitoring of vital parameters during various side tasks at multiple locations. Glass created improved situational awareness. This was borne out when the interventionalist finished his tasks and joined the scientific team in an adjoining room for the interview. During this time the patient remained on the operating table during recovery. The interventionalist still wore Glass and the transfer of patient's vital parameters continued. Suddenly the interventionalist interrupted the interview and returned to the patient, as the patient's vital signs had deteriorated. The substantial experience of the interventionalist made him the only one who recognized the particular issue at hand. He alerted the team, intervened, and the patient went on to recover uneventfully.
5. Discussion Providing vital data on a near-eye display using raw vital signs, icons or simplified visualizations may illustrate significant changes in the patient condition. As we experienced and report here, individual patterns may imply complications. In such a case a warning system could alert the clinical team in a timely fashion. Investigations of such warning systems could be based on existing studies about crash warning systems using head-up displays in cars. Previous work [15] has demonstrated that users perceive
W. Vorraber et al. / International Journal of Surgery 12 (2014) 1266e1272
visual information in the primary range of vision more quickly than distanced sources that could save crucial time. We are planning on investigating additional scenarios including tele-mentoring during surgeries by remote specialists. Various studies describe the possibilities and benefits from distributed collaborations and remote support. Previous research [16] suggests that a device, similar to Glass and capable of synchronous and asynchronous communication, as well as, information sharing is suitable for supporting mobile health workers. Furthermore, integrating the physical environment, capturing gestures or markups from the expert and providing this additional information on the users display led to an additional increase in user performance and may shorten operation time considerably [17]. Additional next steps include an evaluation of Glass on the visualization of the surgeon's field of vision. We anticipate using an external display to facilitate communication with assisting team members in the operation. This scenario is of special interest during surgeries where assisting team members only have limited view on the operating field. During our current investigation we encountered several technical, organizational and user related drawbacks of Glass in its current form: The available Glass Explorer Edition has various shortfalls for daily medical use. In case of computational intensive tasks running on Glass, the battery life is relatively short (i.e. approximately 2 h). Although we overcame this problem by adding an external battery pack, this solution required an element of adaptive engineering. Additionally, there is only a single micro-USB slot. If this is needed for use for the headset then it is otherwise unavailable for other uses. Computationally intensive tasks running on Glass are responsible for excessive heat-buildup and ultimately result in a forced shutdown. Short- or farsighted users can use Glass unrestricted, if they wear contact lenses. Spectacle wearers may not be able to use it properly. Depending on the users face and the form of the spectacle frame, it is possible that both Glass and the spectacle frame won't fit comfortably together. The newer version of Glass can be integrated into prescription spectacle frames [18]. Data privacy is also an important issue that needs to be dealt with from both an organizational as well as technical aspect. As stated by Münsterer [9] general regulations about the use of Glass in a clinical environment have yet to be developed. Specifically developed Glass services (apps) and organizational guidelines need to be developed to ensure data privacy.
6. Preliminary conclusions and limitations This paper reports a proof of concept for using Glass to create user centered medical information services. After a detailed process analysis we identified several areas of improvement by the use of near-eye display devices. We created a framework to structure these areas and selected a scenario where a patient's vital signs were transmitted to an interventionalist wearing Glass via a live video stream. We designed and developed a technical framework (Glass app and server app) that complies with data privacy regulations for this scenario and deployed it in the course of real world angioplasty. Feedback collected from interventionalists provides first evidence for process improvements. However, data collected in this first proof of concept has limited statistical relevance, since it is based on only three interventions by two different interventionalists.
1271
We are encouraged by these preliminary results and are planning to intensify our research in this area based on further quantitative and qualitative studies. We anticipate that a near-eye display device such as Glass can be an enabling technology for process improvements in various medical scenarios. Ethical standards The experiments comply with the current laws of the country in which they were performed. Financial support None Author contribution Wolfgang Vorraber: Conception and design of the study, data collection, data analysis, technical setup, writing of the manuscript. Siegfried Voessner: Conception and design of the study, data collection, data analysis, editing of the manuscript. Gerhard Stark: Conception and design of the study, editing of the manuscript. Dietmar Neubacher: Conception and design of the study, data collection, technical setup, editing of the manuscript. Steven DeMello: Conception and design of the study, editing of the manuscript. Aaron Bair: Conception and design of the study, editing of the manuscript. Conflicts of interest The authors declare that they have no conflict of interest. Acknowledgments We thank Maximilian Sachs (Dept. of Engineering and Business Informatics) for his help in developing software for our proof of concept. References [1] I.E. Sutherland, A head-mounted three dimensional display, in: Proc. December 9-11, 1968, Fall Joint Computer Conference, Part I (AFIPS '68 (Fall, Part I)), ACM, New York, NY, USA, 1968, pp. 757e764. [2] S. Mann, Wearable computing: a first step toward personal imaging, Computer 30 (2) (1997) 25e32. [3] I. Kasai, Y. Tanijiri, T. Endo, H. Ueda, A forgettable near eye display, in: Proc. 4th IEEE International Symposium on Wearable Computers, IEEE Computer Society, Washington DC, 2000, pp. 115e118. [4] H. Whiteman, Google glass 'could transform the way surgery is performed'. Medical News Today. [cited 2014 Feb 26]. Available from: http://www. medicalnewstoday.com/articles/271182.php. [5] A. Gold, Providers navigate potential uses for Google Glass in healthcare. FierceMarkets Questex Media group LLC. [cited 2014 Feb 26]. Available from: http://www.fiercehealthit.com/story/providers-navigate-potential-usesgoogle-glass-healthcare/2013-06-17. [6] R. Mackle, Ohio State Doctor Shows Promise of Google Glass in Live Surgery. Wexner Medical Center. [cited 2014 Apr 11]. Available from: http://www. medicalcenter.osu.edu/mediaroom/releases/Pages/Ohio-State-Doctor-ShowsPromise-of-Google-Glass-in-Live-Surgery.aspx. [7] S. Vallurupalli, H. Paydak, S.K. Agarwal, M. Agrawal, C. Assad-Kottner, Wearable technology to improve education and patient outcomes in a cardiology fellowship program - a feasibility study, Health Technol. 3 (2013) 267e270. [8] U. Albrecht, U. von Jan, J. Kuebler, C. Zoeller, M. Lacher, O.J. Muensterer, M. Ettinger, M. Klintschar, L. Hagemeier, Google glass for documentation of medical findings: evaluation in forensic medicine, J. Med. Internet Res. 16 (2014) 53. [9] O.J. Münsterer, M. Lacher, C. Zoeller, M. Bronstein, J. Kübler, Google glass in pediatric surgery: an exploratory study, Int. J. Surg. 12 (2014) 281e289. [10] C. Rupp, Sophisten, Requirements-Engineering und eManagement 5th edition, München, Carl Hanser Verl. (2009).
1272
W. Vorraber et al. / International Journal of Surgery 12 (2014) 1266e1272
[11] W. Huang, L. Alem, A usability and spatial awareness study of near-eye displays, in: Proc. 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, Seoul, 2012, pp. 906e911. [12] M. Sachs, Requirements Engineering and Design of Human-centered Information Services Utilizing Near-eye Display Devices, Graz University of Technology, 2014. [13] Tech specs Google Glass, Google Inc. [cited 2014 Aug 20]. Available from: https://support.google.com/glass/answer/3064128?hl¼en. [14] M.I. Olsson, M.J. Heinrich, D. Kelly, J. Laptina, Wearable device with input and output structures, U. S. Patent App. 13/212,686, 2013. [15] H. Kim, X. Wu, J.L. Gabbard, N.F. Polys, Exploring head-up augmented reality interfaces for crash warning systems, in: Proc. 5th International Conference
on Automotive User Interfaces and Interactive Vehicular Applications, ACM, New York, 2013, pp. 224e227. [16] J. Li, L. Alem, Supporting distributed collaborations between Mobile health workers and expert clinicians in Home care, in: Proc. CHI '13 Extended Abstracts on Human Factors in Computing Systems, ACM, New York, 2013, pp. 493e498. €llerer, Integrating the physical environment [17] S. Gauglitz, C. Lee, M. Turk, T. Ho into Mobile remote collaboration, in: Proc. 14th International Conference on Human-computer Interaction with Mobile Devices and Services, ACM, New York, 2012, pp. 241e250. [18] X. Miao, Integrated near-to-eye display module, US Patent 8705177 B1, 2014.
Contents lists available at ScienceDirect
International Journal of Surgery journal homepage: www.journal-surgery.net
Original research
Medical applications of near-eye display devices: An exploratory study Wolfgang Vorraber a, *, Siegfried Voessner a, Gerhard Stark b, Dietmar Neubacher a, Steven DeMello c, Aaron Bair d a
Graz University of Technology, Department of Engineering- and Business Informatics, Kopernikusgasse 24, 8010 Graz, Austria Hospital of Elisabethinen, Elisabethinergasse 14, 8020 Graz, Austria c University of California Berkeley, Center for Information Technology Research in the Interest of Society, Sutardja Dai Hall, 94720 Berkeley, USA d University of California Davis Medical Center, Department of Emergency Medicine, 2315 Stockton Blvd., 95817 Sacramento, CA, USA b
h i g h l i g h t s We present a framework to identify and categorize use cases for Google Glass. We describe the use of Google Glass during a radiological intervention. An app was developed to project vital physical signs to Google Glass via intranet. Interventionalists reported improved concentration by reduced head movements. However, heat generation by the device and low battery capacity are shortcomings.
a r t i c l e i n f o
a b s t r a c t
Article history: Received 27 June 2014 Received in revised form 16 September 2014 Accepted 29 September 2014 Available online 22 October 2014
Introduction: Near-eye display devices (such as Google Glass) may improve the efficiency and effectiveness of clinical care by giving clinicians information (such as the patient's vital signs) continuously within their field of vision during various procedures. We describe the use of Glass during a radiological intervention in three patients. Other possible applications (including tele-mentoring and the supervision of trainees) are discussed and a classification proposed. Methods: An app was developed to facilitate the use of Glass, so vital physical signs (pulse and blood pressure) could be projected on the near-eye display, via an intranet to protect sensitive data. The device was then used during radiological interventions (percutaneous transluminal angioplasty) in three patients, and assessed by the interventionalists who were interviewed before and after each procedure. Results: The interventionalists reported that Google Glass improved concentration on the task in hand by reducing head and neck movements (which would be needed to view several remote monitors). However, heat generation by the device and low battery capacity are shortcomings for which solutions must be developed, and data protection is mandatory. Conclusion: Google Glass may have a number of clinical applications and can quicken interventions where vital signs or other visual data need to be monitored by the operator. © 2014 Surgical Associates Ltd. Published by Elsevier Ltd. All rights reserved.
Keywords: Process improvement Clinical information service OHMD Near-eye display device Google Glass
1. Introduction A near-eye display device such as Google Glass enables the transmission of information by augmenting visual perception via a projection in the field of vision. This consolidation of information can potentially allow improved situational awareness without distraction from primary tasks. Although near-eye display devices or so-called optical head mounted displays (OHMD) have existed for more than four decades
* Corresponding author. E-mail address: [email protected] (W. Vorraber). http://dx.doi.org/10.1016/j.ijsu.2014.09.014 1743-9191/© 2014 Surgical Associates Ltd. Published by Elsevier Ltd. All rights reserved.
(e.g. [1e3]), the development of Google Glass (from now on we call it Glass) has created excitement about the potential process improvements that might come from using such devices in a clinical setting [4,5]. Some research groups tested video transmission from Glass to a remote audience during surgery [6]. Likewise, some of the first scientific reports on Glass in medical education [7], documentation in forensic medicine [8], videoconferencing and information querying [9] have recently been published. However, there is a potential array of clinical use-case scenarios that have yet to be investigated. Since it seems to be impossible to exhaustively list all specific use cases, we describe process settings where Glass could improve efficiency and effectiveness:
W. Vorraber et al. / International Journal of Surgery 12 (2014) 1266e1272
Surgeons or interventionalists rely on information from various sources (e.g. patients' vital parameters, medical images, patient records, etc.). Due to the fact that almost all data is already in a digital form, it can be easily distributed to various devices. Therefore, if there are at least two, eventually dislocated, information sources needed to perform a task, wearable devices can bridge local gaps and collect data where it is needed. This is especially true in a clinical setting where users frequently need to monitor data (e.g. vital parameters) while performing their core tasks (e.g. interventions or surgeries). Furthermore, Glass can be suitable for information sharing between separated medical staff. This may range from local information sharing (different patient perspectives e e.g. small size of incision) to national and international information sharing (e.g. remote consultations). Our research was guided by the following questions: Where and how might Glass increase efficiency and effectiveness? How can data be protected when using Glass? What are the possible benefits and drawbacks of using Glass in clinical situations? Glass appears to be a suitable platform for user-centered medical information services with rich potential for further development. The publicly available programming interfaces (Glass Development Kit and Mirror API) will likely be fertile ground for further development. Importantly, the Glass Development Kit (GDK) provides developments that allow Glass to work on local networks (rather than the Internet). This is especially important for data privacy, and enables the implementation of secure end-to-end data transfer without third-party access. We suggest the potential uses of Glass include tele-mentoring and training. Of several possible clinical applications we opted to develop a Glass app and a server app to enable Glass to be used during percutaneous transluminal angioplasty (PTA), with feedback from the two interventionalists involved in three such cases. 2. Methods As depicted in Fig. 1, we followed a three-step approach. During the first step, our team identified several potential areas of improvement by the use of near-eye display devices through systematic requirements engineering based on the SOPHIST-Approach [10]. The SOPHIST-Approach is a technique in software engineering, which offers tools to systematically collect requirements that have to be met by a technical system in order to satisfy end users (e.g. physician) needs. For our second step, we developed the classification scheme described in Section 2.1 to identify technical and organizational commonalities and differences of use cases before implementation. Use cases are classified according to the scheme depicted in Table 1. This scheme, which is located at the center of Fig. 1, facilitates abstraction of the identified applications to see that use case stories can be clustered or even be the same from a technical perspective. In a third step, the abstracted use cases are mapped to technical solutions. On the one hand, the abstraction step prevents from reinventing technical solutions for each new user story. On the other hand the abstraction scheme can also be used as a process innovation tool by mapping various property combinations of the scheme to possible new use cases. One especially promising scenario was selected and a technical solution was implemented. After initial lab experiments, this
1267
application framework was tested during three real world interventions (angioplasties) and feedback was collected from the participating interventionalists who wore Glass during the procedure. Similar to [9] we categorize the identified areas of improvements by the use of near-eye display devices into “Virtual Consultations” (bridging of spatial barriers), “Monitoring of real time patient data” (improvement of situational awareness, reduction of change of attention and focus), “Navigation and medical imaging” (reduction of change of attention and focus), “Viewpoint of surgeon for assistance” (improvement of assistance by projection of the viewpoint of surgeons to assistance e.g. in case of small size of incision) and “Teaching” (transmission of the point of view of an experienced expert to students, respectively from an inexperienced student to an expert for remote consultation). In order to facilitate implementation by identifying technical and organizational differences and similarities we propose the categorization scheme described in the following section. 2.1. Scenario categorization Based on process analysis and interviews with medical experts we identified requirements for human-centered information services and formulated use cases. We categorized them according to: Number of users: The number of users per scenario can either be single or multiple. Data dynamics: Data transferred to the near-eye display device can be either static (e.g. electronic health records) or dynamic (e.g. patient vital signs). Collaboration: Determines whether users will or will not collaborate based on the Glass service. Direction of information flow: The direction of the information flow can either be unidirectional or bidirectional. The unidirectional flow can further be refined into information transferred into the near-eye display device (e.g. patient vital signs) and information recorded by the near-eye display device and transferred outbound (in case of a near-eye display device with recording functionality). Mode of operation: The mode of operation can be classified as passive, or active. A passive mode of operation means that the end user does not actively control the near-eye display device during the scenario. Conversely, active mode of operation represents scenarios where users actively control the device (e.g. tapping on an integrated touch pad or using voice commands). Frequency of use: This criterion categorizes the frequency of the selected scenario in the setting investigated. We introduced the following three levels of frequency “High - daily use”, “Medium -weekly use” and “Low e monthly use” Network coverage: Based on the scenario, the network needed for data transmission can be local, regional, national, or international.
2.2. Scenario analysis Detailed process analysis of various interventions revealed that a high degree of multitasking by interventionalists is required. This frequently means having to rely on multiple monitors and displays arranged around the operating theater while concentrating on the fields of operation (see Fig. 2). Such an arrangement requires a change of attention and focus potentially resulting in loss of efficiency. We note that there is room for process improvement by using near-eye display devices to display relevant data directly into the
1268
W. Vorraber et al. / International Journal of Surgery 12 (2014) 1266e1272
Fig. 1. Systematical abstraction of various use cases and required features according to a proposed classification scheme.
Table 1 Criteria for categorizing scenarios of human centered medical information services using a near-eye display device.
line-of-sight resulting in a reduction of the number of head movements and re-localization processes (see Fig. 3). Prior research on near-eye display devices revealed that there was no evidence of negative impact on spatial awareness of the user [11]. We further
postulate that the aggregation or consolidation of data will improve situational awareness of the surgeon. Based on this initial analysis we formulated the following hypothesis that guided our explorative research: Hypothesis: The projection of patient related parameters (e.g. vital signs) and images to surgeons via near-eye display devices consolidates patient data, which leads to increased efficiency of monitoring and improves situational awareness.
Fig. 2. Suboptimally placed monitors (solid line) and fields of operation (dashed line).
In order to explore this hypothesis we selected a case involving percutaneous transluminal angioplasty (PTA). PTA is an intervention that involves placement of an arterial catheter and various wire-guided implements in an effort to open occluded peripheral arteries. The procedure is done using an x-ray monitor to guide the catheter inside the arteries in addition to monitoring of vital signs. Therefore, this scenario requires frequent scanning of multiple monitors and displays. According to our classification scheme defined in Section 2, this scenario can be categorized as shown in Table 1. We implemented a technical solution that projects the patient’s vital signs (oxygen saturation, heart rate, blood pressure and respiratory rate) onto Glass worn by the interventionalist (Figs. 2e6). We gathered feedback for our explorative study from experts using
W. Vorraber et al. / International Journal of Surgery 12 (2014) 1266e1272
1269
Fig. 3. Displaying relevant data directly into the line-of-sight may reduce the number of head movements and re-localization processes (figure from [12]).
Fig. 4. Google Glass specifications and features.
Fig. 6. The original display of the vital sign monitor (in the background) is projected into the interventionalist's near-eye display device. Fig. 5. Technical setup for using Google Glass to display patient vital signs.
Glass in this setting via a qualitative pre- and post-interview. Pre interview questions served to determine the degree of technological affinity of the user. Post interview focused on identifying possible process improvements and drawbacks through the use of
Glass during PTA. The term “process improvement” was narrowed down to economy of time, increase in efficiency (e.g. reduction of the number of re-localization processes), increased focus on core tasks and improved situational awareness. Possible drawbacks through the use of Glass were categorized into negative influence
1270
W. Vorraber et al. / International Journal of Surgery 12 (2014) 1266e1272
on the course of motion, distraction by the service and negative physical influences such as discomfort wearing.
Glass preventing possible synchronization processes with external servers (and potential loss of control of sensitive patient data) if an internet connection were to be inadvertently established.
2.3. Google glass features 3.2. Medical setup The tested version of Glass (Explorer Edition) was introduced in February 2013 and available for selected users at a price of US$ 1500. This version is equipped with an OMAP 4430 SoC CPU, 1 GB RAM, 16 GB storage and weighs approximately 50 g. Fig. 4 illustrates some relevant features provided by the Glass Explorer Edition that open up a broad spectrum of medical applications. For our purposes we focus on the most useful features concerning output, input, sensors and controls. The prism (1) provides a high-resolution display to illustrate images at maximum 640 360 pixel. According to the specifications provided by Google [13], the display “… is the equivalent of a 25 inch high definition screen from eight feet away.” In the used edition the optical unit is coupled with a thin rigid frame. As mentioned in a Glass patent [14], a rotation of the optical unit enables a proper view by the user throughout various positions. Hence, the viewing surface can be brought in the line of sight depending on the users' individual fit. The camera (2) is placed at the front end of Glass and is able to capture videos up to a Quality of 720p and high-resolution pictures of 5 MP. A light sensor (3) is able to react to light changes and determines if the device is currently being worn. Commonly the interaction with Glass is done by voice control via the microphone (3) or by tapping on the device, which could be either a Touch Pad (4) or a button (5). At the back of the ear a speaker (6) is situated. For noisy environments an earplug can be used. 3. Proof of concept As a first proof of concept, we projected the patient's vital signs onto Glass worn by the interventionalist during the procedure. The data was only accessible to the medical interventionalist's team and was transferred solely via a protected local network to ensure data privacy. 3.1. Technical setup Fig. 5 provides an overview of the technical setup. The video feed of the patient's vital signs was transmitted to a standard laptop computer. This video was then streamed to Glass. In order to ensure the patient's safety, a redundant monitor remained next to the surgical field. Pretests revealed that data transmission via video stream results in high power consumption and therefore shortened battery life of Glass. We addressed this by equipping Glass with an external battery pack worn by the interventionalist. The whole technical setup is transportable and was prepared and tested in our laboratory. This setup took an additional five to 10 min to prepare in the operating room. In order to ensure data privacy we implemented our own software solution that limits data transfer to our local intranet. This enables data transfer between Glass and other devices without synchronization with external network resources such as cloud services (i.e. internet connectivity is not required). This solution is based on a client-server concept. The server application manages data transfer from and to Glass and other data sources and sinks (e.g. vital sign display). It is installed on a PC or laptop and is programmed in C# based on.NET 4.5.1. The client application (Glass app) manages data capturing and presentation on Glass. It is installed on Glass (OS version XE 18.11) and is programmed in Java based on Android Development Tools v22.3.0-887826. The Glass app exchanges data exclusively with the server application running in the same secured private network. No data is stored locally on
The patient was a 64-year-old man who presented with chronic claudication (pain in his left leg related to limited arterial blood flow). The lower extremity magnetic resonance angiogram showed a high-grade occlusion of the left distal femoral artery. Prior to the intervention the patient was fully consented for both the procedure and involvement of Glass expanding the monitoring option of the interventionalist. The real world tests were implemented in accordance with Austrian risk, ethics and data privacy guidelines. The percutaneous transluminal angioplasty proceeded via obtaining vascular access by way of the left common femoral artery. A 5 French short sheath (Johnson & Johnson, Miami, FL, USA) was inserted to the left common femoral artery for anterograde approach. Subsequent guidewire passage of the stenosis in the distal femoral artery was achieved and successful guidewire passage dilation was done. Residual stenosis was approximately 30% and good distal flow was observed. 4. Results For our exploratory study, the interventionalists told us what they thought of Glass and its effect on their practice. Importantly, they did not think that the use of Glass had any negative influences on the course of motion, distraction by the device nor any negative physical influences such as discomfort. Feedback from the interventionalists, both of who reported a high degree of technological affinity, suggests the following regarding impact on process improvements: Glass facilitated improved concentration on the main scenario. Notably, the interventionalist fully relied on Glass and did not even look at the backup display during the entire surgery. Glass supported multi-tasking by allowing efficient monitoring of vital parameters during various side tasks at multiple locations. Glass created improved situational awareness. This was borne out when the interventionalist finished his tasks and joined the scientific team in an adjoining room for the interview. During this time the patient remained on the operating table during recovery. The interventionalist still wore Glass and the transfer of patient's vital parameters continued. Suddenly the interventionalist interrupted the interview and returned to the patient, as the patient's vital signs had deteriorated. The substantial experience of the interventionalist made him the only one who recognized the particular issue at hand. He alerted the team, intervened, and the patient went on to recover uneventfully.
5. Discussion Providing vital data on a near-eye display using raw vital signs, icons or simplified visualizations may illustrate significant changes in the patient condition. As we experienced and report here, individual patterns may imply complications. In such a case a warning system could alert the clinical team in a timely fashion. Investigations of such warning systems could be based on existing studies about crash warning systems using head-up displays in cars. Previous work [15] has demonstrated that users perceive
W. Vorraber et al. / International Journal of Surgery 12 (2014) 1266e1272
visual information in the primary range of vision more quickly than distanced sources that could save crucial time. We are planning on investigating additional scenarios including tele-mentoring during surgeries by remote specialists. Various studies describe the possibilities and benefits from distributed collaborations and remote support. Previous research [16] suggests that a device, similar to Glass and capable of synchronous and asynchronous communication, as well as, information sharing is suitable for supporting mobile health workers. Furthermore, integrating the physical environment, capturing gestures or markups from the expert and providing this additional information on the users display led to an additional increase in user performance and may shorten operation time considerably [17]. Additional next steps include an evaluation of Glass on the visualization of the surgeon's field of vision. We anticipate using an external display to facilitate communication with assisting team members in the operation. This scenario is of special interest during surgeries where assisting team members only have limited view on the operating field. During our current investigation we encountered several technical, organizational and user related drawbacks of Glass in its current form: The available Glass Explorer Edition has various shortfalls for daily medical use. In case of computational intensive tasks running on Glass, the battery life is relatively short (i.e. approximately 2 h). Although we overcame this problem by adding an external battery pack, this solution required an element of adaptive engineering. Additionally, there is only a single micro-USB slot. If this is needed for use for the headset then it is otherwise unavailable for other uses. Computationally intensive tasks running on Glass are responsible for excessive heat-buildup and ultimately result in a forced shutdown. Short- or farsighted users can use Glass unrestricted, if they wear contact lenses. Spectacle wearers may not be able to use it properly. Depending on the users face and the form of the spectacle frame, it is possible that both Glass and the spectacle frame won't fit comfortably together. The newer version of Glass can be integrated into prescription spectacle frames [18]. Data privacy is also an important issue that needs to be dealt with from both an organizational as well as technical aspect. As stated by Münsterer [9] general regulations about the use of Glass in a clinical environment have yet to be developed. Specifically developed Glass services (apps) and organizational guidelines need to be developed to ensure data privacy.
6. Preliminary conclusions and limitations This paper reports a proof of concept for using Glass to create user centered medical information services. After a detailed process analysis we identified several areas of improvement by the use of near-eye display devices. We created a framework to structure these areas and selected a scenario where a patient's vital signs were transmitted to an interventionalist wearing Glass via a live video stream. We designed and developed a technical framework (Glass app and server app) that complies with data privacy regulations for this scenario and deployed it in the course of real world angioplasty. Feedback collected from interventionalists provides first evidence for process improvements. However, data collected in this first proof of concept has limited statistical relevance, since it is based on only three interventions by two different interventionalists.
1271
We are encouraged by these preliminary results and are planning to intensify our research in this area based on further quantitative and qualitative studies. We anticipate that a near-eye display device such as Glass can be an enabling technology for process improvements in various medical scenarios. Ethical standards The experiments comply with the current laws of the country in which they were performed. Financial support None Author contribution Wolfgang Vorraber: Conception and design of the study, data collection, data analysis, technical setup, writing of the manuscript. Siegfried Voessner: Conception and design of the study, data collection, data analysis, editing of the manuscript. Gerhard Stark: Conception and design of the study, editing of the manuscript. Dietmar Neubacher: Conception and design of the study, data collection, technical setup, editing of the manuscript. Steven DeMello: Conception and design of the study, editing of the manuscript. Aaron Bair: Conception and design of the study, editing of the manuscript. Conflicts of interest The authors declare that they have no conflict of interest. Acknowledgments We thank Maximilian Sachs (Dept. of Engineering and Business Informatics) for his help in developing software for our proof of concept. References [1] I.E. Sutherland, A head-mounted three dimensional display, in: Proc. December 9-11, 1968, Fall Joint Computer Conference, Part I (AFIPS '68 (Fall, Part I)), ACM, New York, NY, USA, 1968, pp. 757e764. [2] S. Mann, Wearable computing: a first step toward personal imaging, Computer 30 (2) (1997) 25e32. [3] I. Kasai, Y. Tanijiri, T. Endo, H. Ueda, A forgettable near eye display, in: Proc. 4th IEEE International Symposium on Wearable Computers, IEEE Computer Society, Washington DC, 2000, pp. 115e118. [4] H. Whiteman, Google glass 'could transform the way surgery is performed'. Medical News Today. [cited 2014 Feb 26]. Available from: http://www. medicalnewstoday.com/articles/271182.php. [5] A. Gold, Providers navigate potential uses for Google Glass in healthcare. FierceMarkets Questex Media group LLC. [cited 2014 Feb 26]. Available from: http://www.fiercehealthit.com/story/providers-navigate-potential-usesgoogle-glass-healthcare/2013-06-17. [6] R. Mackle, Ohio State Doctor Shows Promise of Google Glass in Live Surgery. Wexner Medical Center. [cited 2014 Apr 11]. Available from: http://www. medicalcenter.osu.edu/mediaroom/releases/Pages/Ohio-State-Doctor-ShowsPromise-of-Google-Glass-in-Live-Surgery.aspx. [7] S. Vallurupalli, H. Paydak, S.K. Agarwal, M. Agrawal, C. Assad-Kottner, Wearable technology to improve education and patient outcomes in a cardiology fellowship program - a feasibility study, Health Technol. 3 (2013) 267e270. [8] U. Albrecht, U. von Jan, J. Kuebler, C. Zoeller, M. Lacher, O.J. Muensterer, M. Ettinger, M. Klintschar, L. Hagemeier, Google glass for documentation of medical findings: evaluation in forensic medicine, J. Med. Internet Res. 16 (2014) 53. [9] O.J. Münsterer, M. Lacher, C. Zoeller, M. Bronstein, J. Kübler, Google glass in pediatric surgery: an exploratory study, Int. J. Surg. 12 (2014) 281e289. [10] C. Rupp, Sophisten, Requirements-Engineering und eManagement 5th edition, München, Carl Hanser Verl. (2009).
1272
W. Vorraber et al. / International Journal of Surgery 12 (2014) 1266e1272
[11] W. Huang, L. Alem, A usability and spatial awareness study of near-eye displays, in: Proc. 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, Seoul, 2012, pp. 906e911. [12] M. Sachs, Requirements Engineering and Design of Human-centered Information Services Utilizing Near-eye Display Devices, Graz University of Technology, 2014. [13] Tech specs Google Glass, Google Inc. [cited 2014 Aug 20]. Available from: https://support.google.com/glass/answer/3064128?hl¼en. [14] M.I. Olsson, M.J. Heinrich, D. Kelly, J. Laptina, Wearable device with input and output structures, U. S. Patent App. 13/212,686, 2013. [15] H. Kim, X. Wu, J.L. Gabbard, N.F. Polys, Exploring head-up augmented reality interfaces for crash warning systems, in: Proc. 5th International Conference
on Automotive User Interfaces and Interactive Vehicular Applications, ACM, New York, 2013, pp. 224e227. [16] J. Li, L. Alem, Supporting distributed collaborations between Mobile health workers and expert clinicians in Home care, in: Proc. CHI '13 Extended Abstracts on Human Factors in Computing Systems, ACM, New York, 2013, pp. 493e498. €llerer, Integrating the physical environment [17] S. Gauglitz, C. Lee, M. Turk, T. Ho into Mobile remote collaboration, in: Proc. 14th International Conference on Human-computer Interaction with Mobile Devices and Services, ACM, New York, 2012, pp. 241e250. [18] X. Miao, Integrated near-to-eye display module, US Patent 8705177 B1, 2014.
