http://informahealthcare.com/jic ISSN: 1356-1820 (print), 1469-9567 (electronic) J Interprof Care, 2015; 29(1): 13–19 ! 2015 Informa UK Ltd. DOI: 10.3109/13561820.2014.936371

ORIGINAL ARTICLE

Using a situational awareness global assessment technique for interprofessional obstetrical team training with high fidelity simulation Pamela Morgan1, Deborah Tregunno2, Ryan Brydges3, Richard Pittini4, Jordan Tarshis5, Matt Kurrek6, Susan DeSousa5, and Agnes Ryzynski5 1

Department of Anesthesia, Women’s College Hospital, University of Toronto, Toronto, ON, Canada, 2Department of Nursing, York University, Toronto, ON, Canada, 3Wilson Centre, Toronto, ON, Canada, 4Department of Obstetrics & Gynecology, 5Department of Anesthesia, Sunnybrook Health Sciences Centre, Toronto, ON, Canada, and 6Department of Anesthesia, University of Toronto, Toronto, ON, Canada

Abstract

Keywords

Evidence suggests that breakdowns in communication and a lack of situation awareness contribute to poor performance of medical teams. In this pilot study, three interprofessional obstetrical teams determined the feasibility of using the situation awareness global assessment technique (SAGAT) during simulated critical event management of three obstetrical scenarios. After each scenario, teams were asked to complete questionnaires assessing their opinion of how their performance was affected by the introduction of questions during a SAGAT stop. Fifteen obstetrical professionals took part in the study and completed the three scenarios in teams consisting of five members. At nine questions per stop, more participants agreed or strongly agreed that there were too many questions per stop (57.1%) than when we asked six questions per stop (13%) and three questions per stop (0%). A number of interprofessional differences in response to this interprofessional experience were noted. A team SAGAT score was determined by calculating the proportion of correct responses for each individual. Higher scores were associated with better adherence to outcome times, although not statistically significant. A robust study design building on our pilot data is needed to probe the differing interprofessional perceptions of SAGAT and the potential association between its scores and clinical outcome times.

Continuing education, interdisciplinary, interprofessional education, team-based care

Introduction Recent investigations into clinical error demonstrate that breakdown in provider attitudes and behaviors, including poor team communication and clinical decision making, contribute to poor safety outcomes (deLaval, Carthey, Wright, Farewell, & Reason, 2000; Fletcher, McGeorge, Flin, Glavin, & Maran, 2002; Helmreich & Schaefer, 1994). Of particular relevance is the domain of obstetrics, in which communication and teamwork are identified as root causes of sentinel events involving infant death and injury during delivery (Knox, Simpson, & Garite, 1999; The Joint Commission, 2004). Moreover, the Confidential Enquiry into Maternal and Child Health (CEMACH) in the United Kingdom identified the ‘‘lack of communication and teamwork’’ as a leading cause of substandard obstetrical care (Confidential Enquiries, 2004). Such reports suggest that individual competencies in clinical skills are not enough to ensure safe patient outcomes; team communication, collaboration and cooperation skills are essential to effective and safe performance (Rosen et al., 2008). Given the significance of teamwork and communication among caregivers to safe obstetrical outcomes, risk reduction strategies recommended by the Joint Commission include Correspondence: Dr Pamela Morgan, MD, Department of Anesthesia, Women’s College Hospital, University of Toronto, 76 Grenville St., Toronto, Ontario M5S 1B2, Canada. E-mail: [email protected]

History Received 2 July 2013 Revised 25 April 2014 Accepted 16 June 2014 Published online 9 July 2014

perinatal team training focusing on non-technical skills, the use of clinical drills to prepare staff for unusual events, and post-event debriefing to evaluate team performance and identify areas for improvement. As previously reported, the use of high-fidelity simulation can be an effective method of interprofessional education (IPE) for both undergraduate health professional students (Kyrkjebø, Brattenbø, & Smith-Strøm, 2006; Paige et al., 2014) and post-licensure clinicians (Liaw, Zhou, Lau, Siau, & Chan, 2014; Paige et al., 2009). While these data related to IPE simulation are encouraging, Reeves and van Schaik (2012) suggest that few studies of interprofessional simulation have examined interprofessional competence beyond Miller’s lowest level of assessment: knows and knows how. However, a recent systematic review of the literature examining multidisciplinary team training in a simulation setting for acute obstetric emergencies reported demonstrated improvements not just in knowledge, but also in demonstrated skills, communication and team non-technical behaviors in the management of critical obstetric events in seven out of eight studies (Merien, Van de Ven, Mol, Houterman, & Oei, 2010). Hence, additional research is needed to clarify how best to use simulation for IPE training and which outcomes are best improved by such training. Characteristics thought to enhance effective teamwork include a clear understanding of the nature of the emergency, the patient’s needs, the management plan, required tasks and situation awareness (patient focus/involvement) (Bristowe et al., 2012). Situation awareness (SA) is crucial because its absence can result

14

P. Morgan et al.

in poor patient outcomes (Cooper, Endacott, & Cant, 2010). While situation awareness is considered a key factor in the development and maintenance of effective teamwork (ClayWilliams & Braithwaite, 2009; Guise & Segel, 2008; Hicks, Bandiera, & Denny, 2008), it is often used poorly described and not treated as the complex construct that it is (Bristowe et al., 2012). To appropriately represent its complexity, Endsley (1995a) describes three levels of situation awareness: Level 1 – Perception of Elements in current situation; Level 2 – Comprehension of Current situation and Level 3 – Projection of Future Status. In the first level, one must gather information from the surroundings in order to make an assessment. The second level focuses on the need to comprehend the situation based on that information, whereas in the third level there is a need to decide on what might happen next and determine how best to proceed. Failures at each level can negatively affect outcomes (Wright & Endsley, 2008). Situation awareness of a team can be shared which implies that the team is operating with a common set of data, which is crucial since the assessments and actions of one member may have a substantial impact on the actions of others. Teams with high levels of SA are more likely to make effective decisions and take appropriate actions (Wright & Endsley, 2008). Situation awareness global assessment technique Endsley (1995a) presents the Situation Awareness Global Assessment Technique (SAGAT) as a method of objectively and directly measuring situation awareness. During a simulation scenario, the scenario is stopped at various points in time and participants complete written questions as to what is going on around them at that specific time. The scenario is then resumed and stopped again at pre-determined points. Several studies have demonstrated that this temporary ‘‘freeze’’ in the simulation does not impact performance and that subjects are able to resume their management without problems. Evidence supporting the SAGAT model has been established in environments including aviation and commercial transport (Endsley, 1995b, 2000a). SAGAT has been validated and applied across different domains, most recently in medicine (Ha¨nsel et al., 2012; Hogan, Pace, Hapgood, & Boone, 2006; Wright, Taekman, & Endsley, 2004; Zhang et al., 2002). In a study of SAGAT in the training of medical residents using simulation, for example, the authors reported that it demonstrated construct validity in that variation in scores were demonstrated according to the level of training. As well, SAGAT scores were found to be highly correlated with traditional checklist performance measures (Hogan et al., 2006). We were unable to find any articles in the literature that addressed the use of SAGAT for training and assessment of interprofessional teams’ situation awareness. We therefore wished to expand on this line of inquiry linking SAGAT and simulation-based interprofessional education by studying team situation awareness in obstetrical crises.

Methods Study design and framework We designed this pilot study as an evaluation of validity evidence (Cook, Zendejas, Hamstra, Hatala, & Brydges, 2013) related to the development and use of the SAGAT tool in three interprofessional team simulation training scenarios. Specifically, we produced and examined sources of validity evidence to determine if they support or refute use of SAGAT in our study context (Downing, 2003; Messick, 1989). As a secondary focus, we explored participants’ perceptions of the SAGAT (SAGAT Participant Questionnaire) technique using a modified version

J Interprof Care, 2015; 29(1): 13–19

of a previously published and validated questionnaire (Hogan et al., 2006). Participants After research ethics board approval we recruited the teams. Each team included an obstetrician, an anesthesiologist and three registered nurses; all worked with each other on a regular basis in daily practice. Participants volunteered from a pool of approximately 12 obstetricians, 10 anesthesiologists and 40 registered nurses and were enrolled on a first come, first serve basis. Developing the SAGAT tool In order to use SAGAT in this study, it was necessary to develop this tool specifically for the context in which it was to be used. To do so, we employed the ‘‘goal directed task analysis’’ technique (Endsley, 2000b; Wright et al., 2004). In this analysis, the major goals are identified as well as the major sub-goals required to reach these goals. With each sub-goal, there are three major decisions to be identified, each one reflecting one of the three levels of SA (Endsley, 1995a). To determine the major goals, experts are interviewed to identify the goals associated with the specific task and then to further identify the sub-goals. Further interviews are required to address questions focused on the three major decisions (levels of SA) associated with the sub-goals. Once completed, a composite ‘‘tree’’ is constructed for each simulation scenario. For use in a simulated environment, Endsley (2000b) has suggested that four criteria should be met: (1) Timing of SAGAT stops randomly assigned; (2) No SAGAT stop 55 min after the beginning of a scenario; (3) SAGAT stops separated by more than 1 min and (4) at least 30 samplings per SA query collected across subjects and trials for each experimental condition (Endsley, 2000b). To obtain a SAGAT score, Endsley (2000b) has suggested that they be scored as either correct or incorrect, within a tolerance range deemed acceptable by experts in the area. These data can be analyzed as a total SA score (based on the number of correct responses) or on an individual basis. Analysis of individual answers could provide differences that highlight important aspects of team performance (Endsley, 2000b). Once the scenarios and clinically relevant outcomes were finalized, the research team developed SAGAT questions for each scenario. First, for each of the three scenarios, the team established the number of stops and the timing for each stop, based on clinical expertise and experience with each scenario. The first stop was scheduled a minimum of 3 min after scenario commencement and all stops were separated by a period of at least 1 min as recommended by (Endsley, 2000b). Next, for each scenario and for each stop, members of the research team independently developed three questions each to assess all three levels of Endsley’s (1995a) situation awareness approach. A modified Delphi technique was used to achieve consensus on the final three questions for each level of SA as well as the order of clinical importance for the question. Finally, we determined a scoring rubric to delineate acceptable responses for each question and a corresponding dichotomous scoring system to assign one point for a correct answer and 0 for an incorrect/ don’t know answer. For questions requiring a numerical answer (such as heart rate or minutes) an answer was considered correct if ±10% of the actual value. For questions with more than one possible answer (‘‘are there missing resources (equipment/ people), if so, please list’’), the research team determined acceptable responses and appropriate scoring. Research team consensus was obtained for any controversial responses.

DOI: 10.3109/13561820.2014.936371

All scenarios were pre-tested to ensure that the questions and stops were operational. We consider this process as our collection of content validity evidence (i.e. evidence for the relationship between a test’s content and the construct it is intended to measure). Scenario development The following three high-risk simulation scenarios were used: need for cesarean section under general anesthesia, difficult airway, hypoxemia leading to pulseless electrical activity; prolapsed cord, emergency cesarean section, amniotic fluid embolism; and shoulder dystocia with accompanying fetal bradycardia. The first two scenarios were developed and used in previous studies (Morgan, Pittini, Regehr, Marrs, & Haley, 2007; Morgan et al., 2012). For each clinical scenario, the research team determined by consensus a scenario-specific critical clinically relevant outcome measure. For Scenario #1, the recorded time from the absence of SaO2 to the initiation of compressions was chosen as clinically important. Similarly, for the second scenario, the time from the onset of asystole to the initiation of compressions was selected. Finally, for Scenario #3, the time from delivery of the head to delivery of the entire body was chosen. An expert panel were asked to give their opinion on an ‘‘acceptable’’ time from cardiac arrest to the initiation of compressions based on their experience. Similarly, an expert panel of obstetricians were contacted to determine their opinion on an ‘‘acceptable’’ time for the shoulder dystocia outcome. Opinions were aggregated and an average time determined.

Situation awareness training for obstetric teams

15

contents of the simulation scenarios to colleagues, and a release form to record performance. Following a standardized orientation to the simulation equipment and set up as well as a discussion of how the stops would occur, the teams managed Scenario #1. The first responder of the team received the pertinent history, physical findings and laboratory results for the patient. The first responder then entered the labor room and was given the opportunity to interact as usual with the patient mannequin. Pertinent information regarding important history or physical findings was available in the patient’s chart or directly from a member of the research team. Further lab work was supplied as requested. Participants were told that they could call for help as they would normally do, and someone would respond. The scenarios were stopped at 3–4 predetermined times. The vital sign and fetal heart rate monitors were obscured during the stops. During the stop, each team member received a piece of paper with a place for them to write down their answer to each question. A member of the research team read the question aloud to the participants and they were given a few minutes to write down their answers on the sheet of paper. They were advised in advance not to discuss their answers aloud. Teams then managed the second and third scenarios following a similar protocol. Following the final scenario, the teams debriefed one of the recorded scenarios with the facilitator (a member of the research team), specifically reviewing the events that were occurring at the time of the ‘‘stops’’. The facilitator gave feedback as to the team responses to the SAGAT questions at these specific times. The team and facilitator explored approaches to improve situation awareness during the scenarios.

Procedure: simulation sessions Table 1 provides a summary of the clinical scenario, clinical outcomes, the number of stops, the number of questions used to assess the three levels of SA, and examples of the SAGAT questions. Each team was scheduled to attend a three-hour simulation session. Before beginning the first scenario participants were asked to complete a demographics questionnaire, consent form, a confidentiality form stating that they would not reveal the

Study outcomes During each scenario, the computer operator noted the current situation at each stop (e.g. vital signs or fetal heart rate) as a correct reference point to compare the participants’ responses to the SAGAT questions. To determine a score for individual team participants, two members of the research team used the pre-determined rubric and the computer operator recorded numeric information to mark each participant’s score sheet;

Table 1. Scenario description, clinical outcome, number of stops, number of SAGAT questions and sample SAGAT questions.

Scenario and clinical outcome in minutes

Number of stops

Scenario 1 Clinical situation: Morbidly obese parturient with non-reassuring fetal heart rate trace; decision to perform cesarean 4 section: difficult intubation, hypoxemia and cardiac arrest (Pulseless Electrical Activity) Simulated clinical outcome: Time from the absence of SaO2 to the initiation of compressions Sample SAGAT questions: Level 1 Stop 1: What is the patient’s weight/BMI? Level 2 Stop 2: Has the patient’s condition been discussed with the team? Level 3 Stop 3: What is the one key piece of information you want your colleagues to know about the unfolding clinical situation? Level 3 Stop 4: Has the team discussed next steps in this patient’s management? Scenario 2 Clinical situation: Emergency cesarean section in parturient with twin gestation at 34 weeks for cord prolapse: critical event amniotic fluid embolism Simulated clinical outcome: Time from the onset of asystole to the initiation of compressions Sample SAGAT questions: Level 1 Stop 1: What is the presenting twin’s heart rate? Level 2 Stop 2: Has an emergency been declared? Level 3 Stop 3: Are there missing resources (people or equipment)? If yes, list. Level 3 Stop 4: What is the patient’s prognosis? Scenario 3 Clinical situation A G1P0 with NIDDM, induced labour, epidural analgesia: shoulder dystocia and fetal heart rate decelerations, decreased maternal level of consciousness Simulated outcome: Time from delivery of head to delivery of the entire body Sample SAGAT questions: Level 1 Stop 1: Does the patient have any chronic medical condition(s)? If so, what? Level 2 Stop 2: Was the baby delivered in a timely manner? Level 3 Stop 3: Has the team discussed next steps in the patient’s management?

Number of questions/stop 1

4

2

3

3

16

P. Morgan et al.

research team consensus was obtained for any controversial responses. Next, we assigned a SAGAT team score for each scenario by calculating the proportion of correct responses logged by all members of the team (e.g. 0.47 suggests that 47% of the team had a correct response). As a second rubric, we calculated Fleiss’ adjusted kappa (http://justusrandolph.net/kappa) to represent the ‘‘intra-team’’ consistency in the SAGAT responses. One member of the research team reviewed the nine DVDs to derive the clinically relevant outcome times and used an expertderived criterion to judge whether the team’s performance time was clinically acceptable (assigned a ‘‘1’’) or unacceptable (assigned a ‘‘0’’). After each scenario, teams were asked to complete questionnaires assessing their opinion of how their performance was affected by the introduction of questions during a SAGAT stop, the usefulness of the ‘‘stops’’ to improve situation awareness and finally their opinion on how many questions were appropriate per stop. This study focused on the feasibility of the SAGAT technique and on collecting further validity evidence to support its use. Regarding feasibility, we explored participants’ perceptions of this technique including the ‘‘optimum’’ number of questions per stop. Regarding validity evidence, we determined a rubric for aggregating team SAGAT scores and examined associations between these scores and simulated ‘‘clinical outcomes’’ in the form of an acceptable time to complete critical events in the simulation scenario.

Analysis Given the exploratory nature of this pilot project, we did not conduct a power analysis and instead used a convenient sample of three interprofessional obstetric teams. We analyzed the SAGAT team score and Fleiss’ adjusted kappa scores descriptively. Next, we used Student’s t-test to compare the SAGAT team scores for the acceptable versus unacceptable performances (as determined using the clinically relevant outcome). An alpha level of p50.05 indicated a significant difference. Finally, to analyze participants’ responses to the SAGAT feasibility questionnaire, we used chi-square and inspected the data to determine if the responses varied according to the number of questions per stop and by participants’ profession (i.e. anesthesia versus nursing).

Results Three iterations were required to finalize the SAGAT questions. Fifteen obstetrical healthcare personnel took part in the study and completed the three scenarios in teams of five. The average age and number of years in practice for the MD participants was 41 years (range 31–50) and 12 years (range 2–25), respectively, and for the RN participants 46 years (range 31–57) and 23 years (range 8–37), respectively.

J Interprof Care, 2015; 29(1): 13–19

Table 2. SAGAT scores and simulated clinical outcome times. Team 1

Team 2

Team 3

SAGAT proportion score Scenario 1 0.47 0.65 0.52 Scenario 2 0.78 0.58 0.53 Scenario 3 0.60 0.60 0.63 Kappa Scenario 1 0.37 0.57 0.27 Scenario 2 0.32 0.30 0.30 Scenario 3 0.51 0.29 0.54 Clinical time outcome (listed in seconds; 1 ¼ clinically acceptable, 0 ¼ unacceptable) Scenario 1: Time from absences of 26 (1) 106 (0) 365 (0) SaO2 to initiation of compressions (0–30 s) 28 (1) 42 (0) 43 (0) Scenario 2: Time from onset of asystole to initiation of compressions (0–30 s) 195 (0) 103 (1) Scenario 3: Time from delivery of 168 (1) head to delivery of entire body (5180 s)

that most scores were poor, with three scores (one for each team) representing a fair level of consistency. For the three teams performing in three scenarios (nine scores in total), we found that on four scenarios the teams performed within an acceptable clinical time and the average SAGAT score was 0.62 ± 0.13, whereas five scenarios involved unacceptable performance and the average SAGAT score was 0.58 ± 0.05. That difference was not significant (p ¼ 0.52). SAGAT participant questionnaire When analyzed according to the number of questions per stop, there was only one question that had a significant difference in participant response. Specifically, at nine questions per stop, more participants agreed or strongly agreed that there were too many questions per stop (57.1%) than when we asked six questions per stop (13%) and three questions per stop (0%). When analyzed, differences emerged between three groups – anesthesiologists (AN), nurses (RN) and obstetricians (OB). When asked if the stops adversely affected concentration, anesthesiologists were more likely to strongly disagree (62.5% versus 11% OB and 11% RN; Chi ¼ 11.90, p ¼ 0.018) and 0% agreed (versus 22% OB and 33.3% RN). When asked if the freezes adversely affected performance, only nurses agreed (14.8% versus 0% of others; Chi ¼ 11.13, p ¼ 0.025). When asked if freezes decreased anxiety, anesthesiologists were more likely to strongly agree and agree (75% versus 55.6% OB and 40.7% RN; Chi ¼ 22.88, p ¼ 0.001). When asked if freezes made participants more aware, obstetricians disagreed most often (44.4% versus 0% AN and 15% RN; Chi ¼ 17.38, p ¼ 0.008). Finally, when asked if there were too many stops, obstetricians agreed most often (55.6% versus 12.5% AN and 22.2% RN; Chi ¼ 14.64, p ¼ 0.023). All other comparisons yielded no significant differences.

SAGAT team scoring rubric All SAGAT data are reported as mean and standard deviation (SD) unless otherwise specified and are presented in Table 2. When summing the SAGAT score as the proportion of correct responses for all team members, the scores ranged from 0.47 to 0.78 across the scenarios. Hence, the team scores varied widely, though the average score of 0.59 ± 0.09 suggests that the teams collectively responded correctly to approximately 60% of the SAGAT questions. When using the kappa statistic to study the intra-team consistency, it is typical to interpret scores 50.40 as poor and between 0.40 and 0.75 as fair. Our results demonstrated

Discussion A number of studies have demonstrated an improvement in outcomes following interprofessional obstetric team training (Crofts et al., 2006; Draycott & Crofts, 2006; Draycott et al., 2006, 2008; Mann, Marcus, & Sachs, 2006). In a recent study by our research team of 34 interprofessional teams who managed a simulated maternal cardiac arrest, 14 teams either did not recognize that the patient had arrested and/or did not initiate cardiac compressions (Morgan et al., 2010). In the remaining 20 teams, the average time from maternal cardiac arrest until

DOI: 10.3109/13561820.2014.936371

initiation of cardiac compressions was 2 min 55 s (range: 0.4– 6.4 min). In addition, the average time from decision for cesarean section to delivery of the neonate was 9 min (range: 1.3–16.1 min) (Morgan et al., 2010). These findings suggested deficiencies in team situation awareness and supported our development of the SAGAT for interprofessional obstetrical teams’ training in the management of high risk events. We encountered one of the most important issues as identified by Endsley (2000b), which is developing questions directed at each of the three levels of SA (Level 1: Perception of Elements in current situation; Level 2: Comprehension of current situation and; Level 3: Projection of Future Status) that are appropriate to the experimental setting. Our first challenge was to develop the SA questions which would be relevant to all obstetrical team members at each level of SA and at each ‘‘stop’’. Because one of the goals of the study was to determine the ‘‘optimum’’ number of questions per stop (i.e. 3, 6 or 9), we needed to develop nine questions (three questions per level) for each stop for each scenario. We found this extremely difficult because each member of the research team had to overcome their disciplinary focus in order to ensure that the questions were ‘‘generalizable’’ for each member of the interdisciplinary team while at the same time, focusing on events that critical for all to know. In addition, it was difficult to develop questions that were solely applicable to each individual level of SA. Finally, it was challenging at times to ask questions that were not leading or suggestive of what might happen in the next few minutes. We designed SAGAT to extend the conceptualization of SA from the level of the individual to consider the level of SA required for all team members in relation to their respective clinical role and responsibility. We believed that such a manipulation would enhance the ‘‘sociological fidelity’’ of the simulation experience (Sharma, Boet, Kitto, & Reeves, 2011) by making team members aware of their abilities to form a collective, shared knowledge of the clinical situation, including any potential professional boundaries. Yet, our results suggest that it may not be necessary for all members of the team to ‘‘know’’ all the information; in fact each member of the team may only need to know the information that impacts on their role. Given our relatively low kappa score, indicating only fair consistency amongst team members’ responses to the SAGAT tool, we suggest that additional research might focus on whether this inconsistency implies that crucial information is being missed by the teams or if each team member’s awareness is incomplete, yet the team together is highly aware. Indeed, research suggests that there is a subset of information required by all team members (Shulz, Endsley, Kochs, Gelb, & Wagner, 2013) to ensure that team members are interacting in ways that make the whole team greater than the sum of its parts. We would also recommend study of whether participation in simulation educational experiences using SAGAT will make all members of the interdisciplinary team more aware of their surroundings during management of a critical event in an actual clinical setting. This ‘‘awareness’’ may be as simple as glancing at patient monitors displaying vital signs. Our goal in using SAGAT during simulation-based training is to improve clinical outcomes. We found, however, that the SAGAT scores did not differ between teams that performed acceptable versus unacceptable on our proxy clinical outcomes related to their performance. That finding is surprising given that a recent literature review has suggested that there is a relationship between ‘‘team process behaviours’’ and clinical outcome (Schmutz & Manser, 2013). A variety of ‘‘team processes’’ were used in these studies including measures of communication, coordination, non-technical skills and team behavior as some examples. While there were no specific

Situation awareness training for obstetric teams

17

‘‘situation awareness’’ measurements identified in that review, situation awareness is a component of some of the measurement tools (i.e. Anesthesia Non-Technical Skills) (Fletcher et al., 2003). Adding our findings to this review, the implications appear to be that more work is needed to identify team process behaviors and clinical outcomes that are clearly linked in both the simulation and clinical contexts. Our findings determined that nine questions per stop were perceived as excessive and the participants preferred either three or six questions per stop. After examination of the participants’ responses to the SAGAT participant questionnaire, it was obvious that there were professional differences. Interestingly, the anesthesiologists were least disrupted by the SAGAT stops and felt the stops increased their awareness and decreased anxiety in contrast to both obstetricians and nurses. The majority of responses indicated that the SAGAT stops did not negatively affect performance. These findings are consistent with those of other studies (Endsley, 2000b; Hogan et al., 2006). Consistent with our program of research that focuses on interprofessional teams, we chose to develop SAGAT questions and a SAGAT scoring system at a ‘‘team’’ level. Unlike the two published studies using SAGAT in medicine (Ha¨nsel et al., 2012; Hogan et al., 2006) that used individual SAGAT scores for comparison to ‘‘performance’’ outcomes, we chose to evaluate the team’s SA. To our knowledge, this is the first study to use the SAGAT technique to provide a score for team situation awareness. Consequently, we did not have guidance from the literature on how best to pool the SAGAT responses from the different team members in each scenario. We decided therefore to determine a team proportion score, specifically, the proportion of correct responses logged by all members of the team. The proportion scores in the study showed a wide variation with an approximately 60% correct responses to SAGAT questions. This raises the question as to whether the team proportion score is appropriate or whether a more sophisticated analysis of the responses is warranted. Due to the small sample size in this pilot study, we were unable to analyze each discipline’s contribution to score variance. Similarly, we were unable to determine whether a certain Level of SA contributes more than another to the overall SAGAT score. This information may play an important role since without Level 1 SA, there is a higher chance of misinterpreting the situation (Endsley, 2000a). In aviation, for example, Jones and Endsley (1996) found that 76% of SA errors noted in pilots were associated with Level 1 errors. This study has a number of limitations. For example, one limitation is that we enrolled just three teams managing three scenarios and a power analysis was not conducted. Given we explored a new use of SAGAT, for team-based assessment, we chose not to calculate power based on previous studies of individual-based assessment. While this is a limitation, we note that future research may use the data provided in this study as a resource for conducting power analyses and deciding experimental designs when using SAGAT as a teambased outcome measure. As well, the fact that we consolidated the SAGAT scores into a ‘‘team’’ score, is just one approach and other approaches may be entertained. In addition, the process of defining SAGAT questions is idiosyncratic and likely contextspecific.

Concluding comments As presented above, our data suggest a high degree of feasibility of use of SAGAT with interprofessional obstetrical teams and high fidelity simulation. Findings also highlight a number of questions to be explored in future application of the tool.

18

P. Morgan et al.

Specifically, a larger study is required to develop a better understanding of the contribution of each level of SA to clinical outcomes and the degree to which SA is shared across professions. A robust study design building on our pilot data is needed to probe the differing interprofessional perceptions of SAGAT and the potential association between SAGAT scores and clinical outcome times.

Declaration of interest The authors report no conflicts of interest. The authors are responsible for the writing and content of this paper.

References Bristowe, K., Siassakos, D., Hambly, H., Angouri, J., Yelland, A., Draycott, T.J., & Fox, R. (2012). Teamwork for clinical emergencies: Interprofessional focus group analysis and triangulation with simulation. Qualitative Health Research, 22, 1383–1394. Clay-Williams, R., & Braithwaite, J. (2009). Determination of health-care teamwork training competencies: A Delphi study. International Journal of Health Care Quality, 21, 433–440. Confidential Enquiries into Maternal Deaths in the United Kingdom. (2004). Why mothers die, 2000–2002. London, UK: The Centre for Maternal and Child Enquiries (CMACE). Cook, D.A., Zendejas, B., Hamstra, S.J., Hatala, R., & Brydges, R. (2013). What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Advances in Health Sciences Education, 19, 233–250. Cooper, S., Endacott, R., & Cant, R. (2010). Measuring non-technical skills in medical emergency care: a review of assessment measures. Open Access Emergency Medicine, 2, 7–16. Crofts, J.F., Bartlett, C., Ellis, D., Hunt, L.P., Fox, R., & Draycott, T.J. (2006). Training for shoulder dystocia: A trial of simulation using low-fidelity and high-fidelity mannequins. Obstetrics & Gynecology, 108, 1477–1485. de Leval, M.R., Carthey, J., Wright, D.J., Farewell, V.T., & Reason, J.T. (2000). Human factors and cardiac surgery: A multicenter study. The Journal of Thoracic and Cardiovascular Surgery, 119, 661–672. Downing, S.M. (2003). Validity: On the meaningful interpretation of assessment data. Medical Education, 37, 830–837. Draycott, T., & Crofts, J. (2006). Structured team training in obstetrics and its impact on outcome. Fetal and Maternal Medicine Review, 17, 229–237. Draycott, T., Crofts, J., Ash, J.P., Wilson, L.V., Yard, E., Sibanda, T., & Whitelaw, A. (2008). Improving neonatal outcome through practical shoulder dystocia training. Obstetrics & Gynecology, 112, 14–20. Draycott, T., Sibanda, T., Owen, L., Akande, V., Winter, C., Reading, S., & Whitelaw, A. (2006). Does training in obstetric emergencies improve neonatal outcome? British Journal of Obstetrics and Gynaecology, 113, 177–182. Endsley, M.R. (1995a). Measurement of situation awareness in dynamic systems. Human Factors, 37, 65–84. Endsley, M.R. (1995b). Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32–64. Endsley, M. (2000a). Theoretical underpinnings of situation awareness: A critical review. In M.R. Endsley & D.J. Garland (Eds.), Situation awareness analysis and measurement (pp. 3–28). Mahwah, NJ: Lawrence Erlbaum Associates. Endsley, M. (2000b). Direct measurement of situation awareness: Validity and use of SAGAT. In M.R. Endsley & D.J. Garland (Eds.), Situation awareness analysis and measurement (pp. 131–157). Mahwah, NJ: Lawrence Erlbaum Associates. Fletcher, G., Flin, R., McGeorge, P., Glavin, R., Maran, N., & Patey, R. (2003). Anaesthetists’ Non-Technical Skills (ANTS): Evaluation of a behavioural marker system. British Journal of Anaesthesia, 90, 580–588. Fletcher, G., McGeorge, P., Flin, R., Glavin, R., & Maran, N. (2002). The role of non-technical skills in anaesthesia: A review of current literature. British Journal of Anaesthesia, 88, 418–429. Guise, J.M., & Segel, S. (2008). Teamwork in obstetric critical care. Best Practice & Research Clinical Obstetrics & Gynaecology, 22, 937–951. Ha¨nsel, M., Winkelmann, A., Hardt, F., Gijselaers, W., Hacker, W., Stiehl, M., Koch, T.,& Mu¨ller, M.P. (2012). Impact of simulator training and

J Interprof Care, 2015; 29(1): 13–19

crew resource managementtraining on final-year medical students’ performance in sepsis resuscitation: a randomized trial. Minerva Anestesiologica, 78, 901–909. Helmreich, R.L., & Schaefer, H.G. (1994). A model of operating room team performance. In M. Bogner (Ed.), Human error in medicine (pp. 225–253). Hillsdale, NJ: Lawrence Erlbaum and Associates. Hicks, C.M., Bandiera, G.W., & Denny, C.J. (2008). Building a simulation-based crisis resource management course for emergency medicine, phase 1: Results from an interdisciplinary needs assessment survey. Academic Emergency Medicine, 15, 1136–1143. Hogan, M., Pace, D., Hapgood, J., & Boone, D.C. (2006). Use of human patient simulation and the situation awareness global assessment technique in practical trauma skills assessment. Journal of Trauma, 61, 1047–1052. Jones, D., & Endsley, M. (1996). Sources of situation awareness errors in aviation. Aviation Space and Environmental Medicine, 67, 507–512. Knox, G.E., Simpson, K.R., & Garite, T.J. (1999). High reliability perinatal units: an approach to the prevention of patient injury and medical malpractice claims. Journal of Healthcare Risk Management, 19, 24–32. Kyrkjebø, J.M., Brattebø, G., & Smith-Strøm, H. (2006). Improving patient safety by using interprofessional simulation training in health professional education. Journal of Interprofessional Care, 218, 140–149. Liaw, S.Y., Zhou, W.T., Lau, T.C., Siau, C., & Chan, S.W. (2014). An interprofessional communication training using simulation to enhance safe care for a deteriorating patient. Nurse Education Today, 34, 259–264. Mann, S., Marcus, R., & Sachs, B. (2006). Lessons from the cockpit: How team training can reduce errors on L&D. Contemporary OB/GYN, 51, 34–45. Merien, A.E.R., Van de Ven, J., Mol, B.W., Houterman, S., & Oei, S.G. (2010). Multidisciplinary team training in a simulation setting for acute obstetric emergencies. A systematic review. Obstetrics & Gynecology, 115, 1021–1031. Messick, S. (1989). Validity. In R.L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: American Council on Education and Macmillan. Morgan, P.J., Pittini, R., Regehr, G., Marrs, C., & Haley, M. (2007). Evaluating teamwork in a simulated obstetric environment. Anesthesiology, 106, 907–915. Morgan, P., Tregunno, D., Pittini, R., LeBlanc, V., Haley, M.F., DeSousa, S., Milne, K., et al. (2010). Effect of team training on management of obstetrical emergencies. Simulation in Healthcare, 5, 373. Morgan, P., Tregunno, D., Pittini, R., Tarshis, J., Regehr, G., Haley, M.F., DeSousa, S., et al. (2012). Determination of the psychometric properties of a behavioural marking system for obstetrical team training using high-fidelity simulation. BMJ Quality & Safety, 21, 78–82. Paige, J.T., Garbee, D.D., Kozmenko, V., Yu, Q., Kozmenko, L., Yang, T., Bonanno, L., & Swartz, W. (2014). Getting a head start: high-fidelity, simulation-based operating room team training of interprofessional students. Journal of the American College of Surgeons, 218, 140–149. Paige, J.T., Kozmenko, V., Yang, T., Paragi Gururaja, R., Hilton, C.W., Cohn Jr , I., & Chauvin, S.W. (2009). High-fidelity, simulation-based, interdisciplinary operating room team training at the point of care. Surgery, 145, 138–146. Reeves, S., & van Schaik, S. (2012). Simulation: A panacea for interprofessional learning? Journal of Interprofessional Care, 26, 167–169. Rosen, M., Salas, E., Wilson, K.A., King, H.B., Salisbury, M., Augenstein, J.S., Robinson, D.W., et al. (2008). Measuring team performance in simulation-based training: Adopting best practices for healthcare. Simulation in Healthcare, 3, 33–41. Schmutz, J., & Manser, T. (2013). Do team processes really have an effect on clinicalperformance? A systematic literature review. British Journal of Anaesthesia, 11, 529–544. Sharma, S., Boet, S., Kitto, S., & Reeves, S. (2011). Interprofessional simulated learning: The need for ‘sociological fidelity’. Journal of Interprofessional Care, 25, 81–83. Shulz, C., Endsley, M., Kochs, E.F., Gelb, A.W., & Wagner, K.J. (2013). Situation awareness in anesthesia. Concept and research. Anesthesiology, 118, 729–742.

DOI: 10.3109/13561820.2014.936371

The Joint Commission. (2004). Sentinel event alert, Issue 30: Preventing infant death and injury during delivery. Washington, DC: The Joint Commission. Wright, M., & Endsley, M. (2008). Building shared situation awareness in healthcare settings. In C.P. Nemeth (Ed.), Improving Healthcare Team Communication (pp. 97–115). Hampshire, England: Ashgate Publishing Limited.

Situation awareness training for obstetric teams

19

Wright, M., Taekman, J., & Endsley, M.R. (2004). Objective measures of situation awareness in a simulated medical environment. Quality and Safety in Health Care, 13, i65–i71. Zhang, Y., Drews, F., Westenskow, D.R., Foresti, S., Aguter, J., Bermudez, J.C., Blike, G., Loeb, R. (2002). Effects of integrated graphical displays on situation awareness in anaesthesiology. Cognition Technology & Work, 4, 82–90.

Copyright of Journal of Interprofessional Care is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.