This article was downloaded by: [New York University] On: 27 July 2015, At: 18:54 Publisher: Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: 5 Howick Place, London, SW1P 1WG

Ergonomics Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/terg20

An integrated computer-based procedure for teamwork in digital nuclear power plants a

a

a

b

b

a

Qin Gao , Wenzhu Yu , Xiang Jiang , Fei Song , Jiajie Pan & Zhizhong Li a

Department of Industrial Engineering, Tsinghua University, Beijing, P.R. China

b

Shanghai Nuclear Engineering Research & Design Ins, Shanghai, P.R. China Accepted author version posted online: 23 Feb 2015.Published online: 30 Mar 2015.

Click for updates To cite this article: Qin Gao, Wenzhu Yu, Xiang Jiang, Fei Song, Jiajie Pan & Zhizhong Li (2015) An integrated computer-based procedure for teamwork in digital nuclear power plants, Ergonomics, 58:8, 1303-1313, DOI: 10.1080/00140139.2015.1008055 To link to this article: http://dx.doi.org/10.1080/00140139.2015.1008055

PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http:// www.tandfonline.com/page/terms-and-conditions

Ergonomics, 2015 Vol. 58, No. 8, 1303–1313, http://dx.doi.org/10.1080/00140139.2015.1008055

An integrated computer-based procedure for teamwork in digital nuclear power plants Qin Gaoa*, Wenzhu Yua, Xiang Jianga, Fei Songb, Jiajie Panb and Zhizhong Lia a

Department of Industrial Engineering, Tsinghua University, Beijing, P.R. China; bShanghai Nuclear Engineering Research & Design Ins, Shanghai, P.R. China

Downloaded by [New York University] at 18:54 27 July 2015

(Received 3 December 2013; accepted 8 January 2015) Computer-based procedures (CBPs) are expected to improve operator performance in nuclear power plants (NPPs), but they may reduce the openness of interaction between team members and harm teamwork consequently. To support teamwork in the main control room of an NPP, this study proposed a team-level integrated CBP that presents team members’ operation status and execution histories to one another. Through a laboratory experiment, we compared the new integrated design and the existing individual CBP design. Sixty participants, randomly divided into twenty teams of three people each, were assigned to the two conditions to perform simulated emergency operating procedures. The results showed that compared with the existing CBP design, the integrated CBP reduced the effort of team communication and improved team transparency. The results suggest that this novel design is effective to optim team process, but its impact on the behavioural outcomes may be moderated by more factors, such as task duration. Practitioner Summary: The study proposed and evaluated a team-level integrated computer-based procedure, which present team members’ operation status and execution history to one another. The experimental results show that compared with the traditional procedure design, the integrated design reduces the effort of team communication and improves team transparency. Keywords: teamwork; computer-based procedures; shared information about team members; nuclear power plants

1.

Introduction

To accomplish tasks in safety-critical complex systems, such as nuclear power plants (NPPs), operators often use standard procedures – ‘ordered sets of actions which enables a user to achieve a specified objective’ (Niwa and Hollnagel 2002, 289) – to guide their actions, especially under emergency conditions. Computer-based procedures (CBPs) have been widely adopted in modern NPP systems, such as AP1000 (Wen 2011), and are expected to improve operator performance by providing structured, consistent logic procedures and flexible, context-sensitive supporting information (O’Hara et al. 2000). However, without proper design the introduction of CBP might negatively impact operator’s performance and system safety (O’Hara et al. 2000). Previous researches examined a number of interface design issues of CBPs that may influence operator performance, including the information presentation style (Xu et al. 2008), the level of automation (Hwang and Hwang 2003) and the consistency with the operator’s cognitive style (Su, Chen, and Shue 2013). Most of these studies concerned only performance of individual operators. In real NPPs, however, operators work in teams. A team is an organisation including two or more individuals sharing one or more common goals (Kozlowski and Bell 2003). A typical team (also called a crew) in the main control room (MCR) of an NPP includes a shift supervisor (SS, or sometimes a senior technical operator, STO, or technical support, TS) and various board operators (BOs): a reactor operator (RO), a secondary circuit operator (SCO) and, sometimes, auxiliary operators. The SS is the leader of the team and responsible for the whole system. The RO is responsible for the primary coolant circuit – the circuit that changes the nuclear energy to heat energy, while the SCO is responsible for the secondary circuit, which converts the heat energy to electric energy (Vidal et al. 2009). These team members work together to ensure the NPP in a good operational state. Although teamwork is believed to be critical to system performance in NPP, it is often less than satisfactory. According to previous research, poor teamwork is one of the main sources of human errors in NPPs: 17% of the 193 human error incidents for 31 years in NPPs in Japan were caused by poor teamwork (Hirotsu et al. 2001). Also, 10% of the 232 human failures in NPPs in Germany were due to problems in team communication (Stra¨ter 2003). Meanwhile, the introduction of digital technology brings new challenges for teamwork in NPPs. Because the digital system can supply higher- and lower-level information to both the BOs and the SS, there seems no need for them to exchange information as frequently as they used to do in traditional MCRs. Thus the amount and frequency of communication between team members is sharply reduced (Chung, Yoon, and Min 2009; Min, Chung, and Yoon 2004),

*Corresponding author. Email: [email protected] q 2015 Taylor & Francis

Downloaded by [New York University] at 18:54 27 July 2015

1304

Q. Gao et al.

which may lead to a lack of shared knowledge and may negatively influence teamwork. When teams perform procedures under emergency, the existing CBPs provide only the information relevant to individual operators’ responsibilities but no information about the status and progress of other team members. This deficiency may harm team performance and system safety by reducing the ‘openness of interaction’ between team members (Kim et al. 2012; Roth and O’Hara 1999; O’Hara and Higgins 2010; O’Hara et al. 2000). Furthermore, the digitalisation of the system also reduces the possibility that operator obtain information about team members by observing the physical workspace. In traditional MCRs, operators could look around form team members and infer what they are looking at or doing based on other members’ locations and postures. In a digital NPP system, everyone is sitting in front of computers and looking at his/her own screen. Such inference becomes difficult. Shared information regarding team members, such as their location, activities, intentions, characteristics, knowledge and attitudes, is one of the most important factors that influence team performance. This shared information is believed to help team members establish a comprehensive situation awareness of teamwork, to give them hints to understand what’s going on in the team, and to help them anticipate what will happen in the future (Mohammed, Ferzandi, and Hamilton 2010). Shared information improves team transparency (the ‘openness, availability, or disclosure of information’, Palanski, Kahai, and Yammarino 2011, P203). High team transparency further makes communication between team members more effective (Dabbish and Kraut 2008). When team members are knowledgeable about others’ activities, errors and performance, they develop more synchronous activities and volunteer more appropriate backup behaviours between members (LePine et al. 2008; Marks and Panzer 2004). A series of studies found that shared information about team members contributes positively to team performance (Bardram, Hansen, and Soegaard 2006; Cannon-Bowers, Salas, and Converse 1993; Lim and Klein 2006; Roth, Multer, and Raslear 2006). The more similar and more accurate the shared information about the team, the better the resulting teamwork (Lim and Klein 2006; Mathieu et al. 2000). To better support teamwork in MCRs of NPPs, we designed an integrated CBP in this study. To validate this novel design, we conducted a comparative experiment between the novel integrated CBP and the traditional individual CBP. We focused on emergency operating procedures (EOPs), because under emergency condition effective teamwork is particularly critical to system performance (e.g. Lin et al. 2011). Sixty participants, randomly divided into twenty teams (with every three people composing a team), were asked to complete the EOP of the ‘Loss of Coolant Accident’ (LOCA) scenario using one of the two designs. Operation time and error rate were collected as behavioural outcomes. Subjective evaluations of team communication, team transparency and mental workload were also collected as process variables. In the following section, we introduced the design of the integrated CBP in details. 2.

Design of an integrated CBP for teamwork

The integrated CBP in our study was designed based on the existing individual CBP borrowed from Xu et al. study (2008), which has been proved as effective in supporting operator performance: As shown in Figure 1, the individual CBP presents both the procedures and the system information required for operators’ executing procedures. On the left of the interface is a flowchart comprised of all steps in a procedure for an individual operator. The current step being executed is highlighted (in yellow colour), the past steps that have been executed are in deep grey, and the steps need to be executed are in light grey. The detailed logical structure and execution operations of the current step are presented in the middle of the interface. The right side presents system parameters and control buttons related the execution of the current step. To design a team-level integrated CBP, what information should be shared between members is the most important decision. The information related to the status of other team members is enormous, but the screen real estate is limited. So are operators’ attention resources. Based on a synthesis of empirical and experimental studies, Gutwin and Greenberg (2002) strongly argued that Who, What, Where and the relative time cue (now and past) are the most important elements to improve the shared awareness of team members. Based on detailed examination of these factors in the context of EOP tasks in NPP, we chose to present team members’ operational status, execution history and communication history with time stamps on the integrated CBP. Information related to Where is not presented because such information is obvious in a colocated working environment. To represent the information in a way easy to understand and use, we followed the following usability principles in the visual design (Norman 2002; Stone et al. 2005): . Visibility and affordance: Actionable items (e.g. control buttons) were designed with a clear pushing affordance, whereas non-actionable items (e.g. system parameters, procedure steps which were currently not actionable) avoided such an affordance. The current status of the steps, operators and parameters was made clear by colour coding. . Structure: The related content was either grouped together using borders (e.g. procedures for each operator) or placed in close adjacency (e.g. each operator’s execution history was right below his procedure steps; the procedure for the current operator and the detailed content of the current step in that procedure were placed next to each other). The

Downloaded by [New York University] at 18:54 27 July 2015

Ergonomics

Figure 1.

1305

(Colour online) Design of the individual CBP interface.

most important group of information – the detailed view of the current step for the current operator – was emphasised by a different background colour and a large title font. . Consistency: The identical terminology and abbreviations were used throughout the system. We also used consistent font styles, font sizes and colour coding schemes in the system. Figure 2 showed a screenshot of the integrated CBP. On the left of the interface are procedure flowcharts for all team members. The procedure of the current operator is highlighted with a green title bar and located next to the detailed step

Figure 2.

(Colour online) Design of the integrated CBP interface.

1306

Q. Gao et al.

information of the current step in the middle of the interface. The colour coding for indicating the current step (yellow), the completed steps (dark grey) and the steps that need to be accomplished (light grey) is the same as that in the individual CBP. Red dotted lines are used to connect steps in which communication or notification between team members is needed. Oral notifications or confirmations prescribed by the procedure between team members can be sent as electronic messages via the CBP system. The send buttons are in the lower right side of the interface, and the received information is shown in the textbox below the flowchart of the current step in the middle. Each operator’s identification information (name and picture) and his/her execution history information are displayed below the corresponding procedure flowcharts.

Downloaded by [New York University] at 18:54 27 July 2015

3.

Hypotheses

To validate this new design, we conducted a laboratory experiment with the aim to compare team performance under the integrated CBP condition and the individual CBP condition. Because the information about team members such as their operational status and execution history could facilitates awareness of what happened to each other and what is going on around the system and teamwork, we expected that the transparency – the openness and availability of information in teamwork (Frentrup and Theuvsen 2006; Palanski, Kahai, and Yammarino 2011) under the integrated condition would be higher. The supplemental information about other team members’ status gives operators clues to seize the proper chance to communicate with each other (Dabbish and Kraut 2008). Meanwhile, the integrated CBP allows team members to send electronic notifications and messages, as well as to maintain a notification history. Those could lower the communication workload, as we expected. Furthermore, under the integrated CBP condition, operators could generate more synchronous activities and backups to smooth the collaboration. Thus, it was reasonable to expect that when operators used the integrated CBP during their work, they could achieve lower error rate and shorter operation time, and that the mental workload they had during the operation would be lower: Hypothesis 1: Hypothesis 2: Hypothesis 3: Hypothesis 4:

The participants have better performance (a lower error rate and shorter operation time to accomplish tasks) under the integrated CBP condition than under the individual CBP condition. Team communication under the integrated CBP condition is easier than under the individual CBP condition. Team transparency under the integrated CBP condition is higher than under the individual one. Mental workload under the integrated condition is lower than under the individual CBP condition.

4. Methodology 4.1 Participants Eighty-four undergraduate students with an engineering education background in Tsinghua University were recruited in this study. All participants were male because the majority of NPP operators in China are male and because we wanted to reduce unwanted influences of individual differences. Every three students were randomly assigned to a team, acting as the SS, the RO and the SCO. The twenty-eight teams were randomly and equally assigned to the two experimental conditions. All of the students were familiar with computers operation. The ages of these participants ranged from 18 to 27 years, with an average age of 22 years. 4.2

Scenarios and tasks

We chose LOCA as our experimental scenario because the LOCA procedure involves relatively more communication and collaboration among team members. LOCA is a typical emergent accident caused by the loss of coolant in the primary coolant circuit. The experimental procedure was a simplified version of the real LOCA procedure of a 900MW pressurised water reactor from Daya Bay NPP in China. As shown in Table 1, we kept the main structure of the EOP and reduced repetitive manual controls. Communication and other collaborative activities were retained as much as possible. Table 1.

Procedure information before and after the adaption. Before

Logic steps Manual controls

After

RO

EO

SS

RO

EO

SS

10 64

5 23

11 59

10 31

5 20

11 31

Ergonomics

1307

Three operators in one team should complete tasks by following the EOPs step by step. When they encounter a decision point, they need to collect related information from the CBP interface, click the right button to open/close pumps or valves and choose the correct branch of the process to follow. In addition, they need to communicate with each other when necessary. Depending on the size of the leak, accident symptoms are varied. This leads to different procedure routes in the EOP. In the experiment, we simulated six different LOCA conditions and correspondingly six EOP routes. Two were main routes that comprised a complete path from the first to the last step of the procedure – the routes did not terminate or change to other procedures in the middle. One main route featured collaborative operations (e.g. to cool down), and the other mainly involved frequent exchange of notifications and confirmations. The other four bypass routes contained fewer steps.

Downloaded by [New York University] at 18:54 27 July 2015

4.3

Apparatus

The two simulated CBP systems were developed using Microsoft Visual BasicTM. Three computers (corresponding to the three roles of operators) were connected to a server that was responsible for configuring the experimental interface, assigning team roles and randomising task sequences. The experiment data of each participant were collected via the interface and stored in a text file in the corresponding computer. The scenarios and the principles behind the scenarios were all adapted from real NPPs. The layout (the location of the three operators) and the experimental environment were also arranged to resemble the real MCR to let the participants feel like behave in a real MCR. This microworld simulation method has been widely used in studies about the interaction between human and complex systems for decades (e.g. Gao et al. 2013; Pawlak and Vicente 1996; Su, Chen, and Shue 2013; Berggren et al. 2014; Lew, Boring, and Ulrich 2014). For explorative research, testing in a real-world system or a full-scope simulator with qualified operators in these systems requires unnecessarily high cost and is sometimes improper (e.g. there might be too few qualified operators for a statistically meaningful experiment). The core idea of microworld simulation method is to hold the essential characteristics of real process control tasks but in a simple way. The experimental tasks should reflect the nature of the real tasks as much as possible, and meanwhile be easy enough to be understood by novel operators after a quick training. Furthermore, by using microworlds, the researchers can fully control the experiment and trace performance measures easily by instrumenting the simulation environment. For the design of complex systems, it is common that researchers first explore possible design initiatives or scientific principles with microworld simulations, and then validate the findings with full-scope simulators or real-world systems (Berggren et al. 2014; Lew, Boring, and Ulrich 2014; Yin 2012). 4.4

The independent variable

The only independent variable is the design of the CBP. The two levels were the CBP that only contains the procedure information about a single operator (the individual CBP) and the CBP that supplies all three operators’ execution status to each other (the integrated CBP). 4.5

Dependent variables

We collected both behavioural outcomes (operation time and error rate) , and mental workload, team communication and team transparency are measured to reflect team performance in this study. . The operation time was defined as the average operation time of a team to succeed in finishing a procedure. . The error rate was defined as the ratio of the number of trials that a team failed in a procedure to the total number of trials. If any member of the team selects a wrong route at any step in a procedure, announces the wrong message, contacts the wrong person or makes a wrong manual control, the team fails that trial. . Mental workload was measured by a Chinese version of the National Aeronautics and Space Administration task load index (NASA-TLX) (Gao et al. 2013, except the score of ‘performance self-rating’, the higher the score was, the higher the workload was). NASA-TLX reflects workload in six dimensions: (1) mental demands, (2) physical demands, (3) temporal demands, (4) performance self-rating, (5) effort and (6) frustration (Hart and Staveland 1988). In this study, we used the raw TLX, of which the weight of subscales are equal to each other. The raw TLX has been found more convenient and even more sensitive than the weighted TLX (Hart 2006; Hendy, Hamilton, and Landry 1993). Every participant rated the questions with a 10-point scale, and individual ratings from all team members were averaged to represent the workload of the team (Funke and Galster 2009). The internal consistency of the scale was measured by Cronbach’s a, which has been widely used in publications in psychology and other social sciences to reflect the reliability of self-reported data. The Cronbach’s a of the NASA-TLX in this study was calculated as 0.81, indicating a good level of internal consistency.

1308

Q. Gao et al.

Downloaded by [New York University] at 18:54 27 July 2015

. We measured the characteristics of team communication including fluency and accuracy (Lingard 2011) by four selfrating questions: (1) the fluency of communication, (2) the accuracy of communication, (3) communication pressure and (4) efforts to team communication. All questions were rated from 1 to 10. For the first two questions, higher scores indicated higher satisfaction with the team communication; for the latter two questions, higher scores indicated more difficulty in team communication. The Cronbach’s a of this scale was 0.48, lower than the common criterion of 0.6 for acceptable reliability. However, this instrument consisted of only four items, and a small number deflates the value of Cronbach’s a. Bradley (1994) suggested that for a scale of 3 to 4 items, a value of 0.5 is sufficient. If we deleted the item of ‘efforts to team communication’, the Cronbach’s a would reach 0.57. . Transparency is a wildly used term across many fields such as politics, economics and information technology. Its definition varies according to the context. In this study, we defined team transparency as the ease and degree of knowing the team and system-related information. We measured team transparency based on this definition with the following three questions: (1) the ease of knowing a team member’s operational status, (2) the degree of knowing a team member’s operational status and (3) the degree of knowing the team task completion status. All of the questions were rated from 1 to 10. The higher the scores were, the higher the team transparency was. The Cronbach’s a of this survey was 0.59 in this study. 4.6

Experimental procedure

The experiment was conducted in two secluded and quiet rooms in parallel. In real NPPs, the MCR is also isolated from the outside world and is relatively quiet – the background noise is constrained not to exceed 65 db, for the fear of impairing normal verbal communication between any two points in the MCR (O’Hara et al. 2002). Every team completed the experiment on its own, and three participants were assigned to the three roles randomly. The procedure of the experiment is as follows: (1) the participants signed the consent forms and watched to a tutorial of the experimental system and tasks. The tutorial lasted approximately 20 min, followed by a brief session when the participants could raise any question they had about the experiment. (2) Participants practised the procedure. They were encouraged to try their best to avoid any mistakes and to be fast. They could ask questions about the experiment at any time during the practice session. The practice session ended when they consecutively succeeded in completing the main routes three times, or when they completed three long procedures during the first 30 min (based on the experience from the pilot study, 30 min practising is enough for participants to fully understand the team procedure tasks). The practice session lasted approximately 40 min. (3) The participants took a 5-min break. (4) The formal experiment started. Every team was required to complete twenty trials. The two main routes were repeated six times each; the other four bypass routes were repeated twice each. The order of the routes was randomised. Only the performance of the main routes was analysed. Participants were allowed to communicate freely (as much as they liked) in both conditions. However, no more questions could be asked of the experimenter. (5) Upon completion of all tasks, the team was asked to take the NASA-TLX, team communication and team transparency questionnaires. Every participant was asked to answer these questions based on his own experience during the experiment. (6) The participants received their payments and were asked to give suggestions/opinions about the experiment. The entire experiment took approximately 120 min. 5.

Results

Seven teams did not pass the training session: three of them did not attain any long route success within the first half an hour training, which suggested that they may have some problems in understanding our tasks; four groups’ error rates were too high (about 90%) to pass the training, implying poor understanding of and probably inadequate concentration on the tasks. In addition, one group got stuck with a program bug, and their practice session became fallaciously long. By removing data of these teams, we had 20 teams for comparing the differences in team performance between the two treatments. One-way ANOVA with SPSS 16.0 was used. All of the dependent variables passed the normality test by the Kolmogorov-Smirnov method and the homogeneity of variances test by Levene’s test. 5.1

Operation time and error rate

Before testing hypothesis 1 and 2, we ran a two-way ANOVA of between- (the two experiment conditions) and within- (the two main routes) participants mixed design to see if there was a significant effect of the routes on operation time and error rate. The result showed that neither the main effect of the route type (for the operation time: F(1,19) ¼ 0.34, p ¼ 0.565, h 2 ¼ 0.02; for the error rate: F(1,19) ¼ 1.61, p ¼ 0.220, h 2 ¼ 0.07) nor the interaction between the route type and the experiment condition (for the operation time: F(1,19) ¼ 0.07, p ¼ 0.788, h 2 , 0.01; for the error rate: F(1,19) ¼ 0.20,

Ergonomics

1309

Downloaded by [New York University] at 18:54 27 July 2015

p ¼ 0.661, h 2 ¼ 0.01) was significant. Therefore, the error rate and the operation time of the two main routes were respectively averaged. The ANOVA results of the operation time and the error rate of the procedures are shown in Table 2. No statistically significant differences in error rate were found. However, the operation time under the integrated CBP condition was longer than the operation time under the individual CBP condition (F(1,19) ¼ 7.43, p ¼ 0.014, h 2 ¼ 0.29), which was opposite to our expectation. We suspected that the unexpected result might come from the difference in the training time. Thus, we examined the training data for a further explanation and found that the training time of the teams under the integrated CBP condition was significantly shorter than the teams under the individual CBP condition (F(1,19) ¼ 7.36, p ¼ 0.014, h 2 ¼ 0.29, see more information in Table 3). Because the operation time decreases as the training time increased (e.g. Xu et al. 2008), the difference in training time may indicate a systematic error that may confound the results. To eliminate this error, an analysis of covariance (ANCOVA) with training time as the covariate was conducted. Preliminary tests were conducted to ensure that there was no violation of the assumptions of normality, linearity and homogeneity of variances. The results revealed that the operation time under the integrated condition was still longer (F(1,19) ¼ 5.03, p ¼ 0.039, h 2 ¼ 0.23) than under the individual condition. The training level could not explain the difference in the operation time between the two treatments. 5.2 Team communication, mental workload and team transparency Table 4 shows the results of the ANOVA of the ratings on mental workload, team communication and team transparency. No significant difference was found in the overall workload, but teams under the individual CBP condition were more satisfied with their performance (F(1,19) ¼ 4.55, p ¼ 0.047, h 2 ¼ 0.14). This result is partly consistent with the operation time result in 4.1, but did not fit with our expectations. No significant difference was found in the overall ratings on team communication between the two conditions. Further investigation showed that the teams with the integrated CBP made less effort to communicate than the teams with the individual CBP (F(1,19) ¼ 6.44, p ¼ 0.021, h 2 ¼ 0.26). The other items exhibited no significant difference between the two conditions. The total score of team transparency significantly differed in two conditions. The teams under the integrated CBP condition perceived a higher level of team transparency than the teams under the individual CBP condition (F(1,19) ¼ 7.62, p ¼ 0.013, h 2 ¼ 0.30). Further examination showed that the teams under the integrated CBP found that it was easier to obtain information regarding the other team members’ operational status (F(1,19) ¼ 5.74, p ¼ 0.028, h 2 ¼ 0.24). 6. Discussion and conclusions In complex systems, teamwork problems such as inappropriate team structure (e.g. Undre et al. 2006) and communication breakdowns (e.g. Greenberg et al. 2007) are common. Because shared information about team members’ status exhibited a general positive effect in improving team performance across fields (Mathieu et al. 2000; Mesmer-Magnus and DeChurch 2009; Roth, Multer, and Raslear 2006), supplying necessary team member-related information to each other with the information system seems a natural and attractive option to enhance teamwork. Such an intuition leads to a few attempts to develop awareness displays for presenting team members’ mission status, workload and availability (Scott, Sasangohar, and Table 2.

Summary of ANOVA for the behaviour outcomes of team performance.

Operation time (s) Error rate Sample size

Table 3.

Individual CBP M (SD)

Integrated CBP M (SD)

F

p

h2

107.75 (19.69) 0.18 (0.11) 10

138.22 (29.34) 0.29 (0.14) 10

7.43 3.51 –

0.014 0.077 –

0.23 0.07 –

Summary of ANCOVA for the team performance with training data.

Operation time (s) Error rate Training time (s) Sample size

Individual CBP M (SD)

Integrated CBP M (SD)

F

P

h2

107.75 (19.69) 0.18 (0.11) 2532.29 (587.19) 10

138.22 (29.34) 0.29 (0.14) 1827.78 (573.93) 10

5.03 1.19 7.36

0.039 0.291 0.014

0.23 0.07 0.29

1310

Downloaded by [New York University] at 18:54 27 July 2015

Table 4.

Q. Gao et al. Summary of ANOVA for the participative indexes of team performance. Individual CBP M (SD)

Integrated CBP M (SD)

F

p

NASA-TLX (Cronbach’s a ¼ 0.81) 1 Mental demands 2 Physical demands 3 Temporal demands 4 Performance self-rating 5 Effort 6 Frustration

26.13 (7.69) 4.77 (1.37) 3.90 (1.56) 4.37 (1.80) 7.57 (1.83) 5.97 (1.09) 3.70 (1.95)

27.63 (6.08) 5.07 (1.82) 3.27 (1.46) 4.60 (1.38) 6.03 (1.35) 5.83 (2.00) 3.90 (1.02)

0.23 0.17 0.88 0.11 4.55 0.03 0.08

0.634 0.682 0.361 0.749 0.047 0.855 0.777

0.01 0.01 0.05 0.01 0.20 ,0.01 ,0.01

Team communication (Cronbach’s a ¼ 0.48) 1 Fluency 2 Accuracy 3 Communication pressure 4 Effort to communication

30.43 (3.51) 8.40 (1.59) 8.30 (1.05) 2.97 (1.73) 5.30 (1.16)

31.23 (2.89) 8.33 (0.93) 8.27 (0.63) 3.33 (1.49) 4.03 (1.07)

0.31 0.01 0.01 0.30 6.44

0.585 0.910 0.932 0.617 0.021

0.02 ,0.01 ,0.01 0.01 0.26

Team transparency (Cronbach’s a ¼ 0.59) 1 Ease of knowing team members’ operation status 2 Degree of knowing team members’ operation status 3 Degree of knowing team task completion status Sample size

19.63 (2.06) 6.07 (1.43) 6.27 (0.95) 7.30 (0.88) 10

22.57 (2.65) 7.43 (1.10) 7.20 (1.55) 7.93 (0.84) 10

7.62 5.74 2.63 2.70 –

0.013 0.028 0.122 0.118 –

0.30 0.24 0.13 0.13

h2

Cummings 2009; Dabbish and Kraut 2008; Birnholtz, Bi, and Fussell 2012). But the number of such studies is small, and the implementation is limited to distributed collaborative systems. The team process can be very different between distributed teams and the co-located teams, such as operators in MCRs of NPPs. For example, in the latter setting, team members can obtain information about other team members both from the electronic system and from personal interaction with others. How the introduction of computer support can influence the team communication process for co-located groups has rarely been examined. To the best of our knowledge, we are the first to address this void. We designed an integrated CBP supplying team members’ operation status and execution history to each other to support teamwork in modern MCRs in NPPs in this study. The comparative experiments showed that in spite of the unexpected results in behavioural outcomes and mental workload, the teams with the integrated CBP reported a lower level of effort for team communication and a higher level of team transparency. The self-rating scores of team communication showed that the teams using the integrated CBP made less effort to communicate but achieve almost an equivalent level of communication quality as the teams using the individual CBP (indicated by the very close self-rating scores in communication fluency, accuracy and pressure). This is consistent with our expectation that knowing the status of others helps people to select the appropriate time to communicate with each other (Dabbish and Kraut 2008) and reduce non-effective communication. Furthermore, the integrated CBP design improves team transparency. Together the results suggest that under the integrated CBP condition, team members can obtain even better knowledge of each other with less communication effort compared with the individual CBP condition. As mentioned before, team communications are full of chaos in complex systems, and a large number of previous studies suggest that high team transparency is necessary for high-quality teamwork (Frentrup and Theuvsen 2006; Palanski, Kahai, and Yammarino 2011; Rasker, Post, and Schraagen 2000). Palanski and his colleagues (2011) found that high transparency makes team members keeping their promise and improves team trust and team performance. Transparency also helps in reducing free riding, maintaining the justice of teamwork and making team performance more efficient (Mohnen, Pokorny, and Sliwka 2008). However, the behavioural outcomes didn’t improve as expected: the error rates under the two conditions were not different from each other; the operation time under the integrated CBP condition was even longer than under the individual CBP condition. The training level cannot explain the difference between the two conditions. There are a couple of possible explanations: first, the extra information on the integrated CBP interface increases the interface complexity and extends the time for searching the right information (Kang and Seong 1998; Perlman 1987). In the experiment, the participants need to spend a lot of time in locating the information and the control buttons referred in the procedure to execute the steps. Previous research has found a large number of visual elements make stimuli more distracting and lead to longer visual search time (Wickens, Gordon, and Liu 2004). Second, the tasks in this study were relatively easy and the participants might complete the tasks with only a limited knowledge of others’ status. In real NPPs, emergency tasks are more complicated and need deeper collaborations, especially when the situation of the system goes beyond what the procedure has prescript. In these conditions, operators need to gather more information through collaboration to support decision-

Downloaded by [New York University] at 18:54 27 July 2015

Ergonomics

1311

making (Carvalho, Dos Santos, and Vidal 2005) and a higher transparency among team members is important. Third, it takes times for the positive effect of the transparency on the team performance to become visible: Transparency helps to establish trust between team members gradually during their interaction and further improve team performance. The whole process is a function of time (Frentrup and Theuvsen 2006). In our study, the team members only worked together for about 90 min, which was insufficient for reveal changes in team performance. Despite of the improved perception of communication and transparency, however, we observed that there was a trend that the teams using the integrated CBP exchanged a less amount of verbal communication than the teams using the individual CBP. Such a reduction seems natural, as with the integrated design support the exchange of task-related information and notification between team members via the electronic system. But we should be careful with the risk that such a reduction may inhibit the exchange of non-task-related but useful information. For example, we found that some team members using the individual CBP actively talked about the conditions of the team (‘now the difficult part comes. Let’s try our best!’) and of themselves (‘please wait for me . . . need more time . . . ’). But such talking was rare under the integrated condition. These emotional encourages and active self-expressions can foster a positive emotional climate, and may improve the performance of organisations in the long run (Ozcelik, Langton, and Aldrich 2008). More research is needed to examine the impact of showing other members’ activities and status on the development of such climate in colocated teams. There are several limitations of this study. First, the participants were all sampled from university. They had adequate knowledge and skills at solving the problem in the experiment, but their experience level, communication skills and collaboration experiences might be very different from real MCR teams who have been trained for years as team members. Further validation among professional operators is needed to draw conclusion about if the same results exist in a real setting. Another limitation is that the teams in the integrated CBP condition reported that they have a better understanding of other team members’ status, but what information they know and how well they know about it is unclear. In addition, we recorded mainly the quantitative information (e.g. performance time, error rate) but little qualitative information (e.g. verbal communication content) of the operation process. Such qualitative information, if captured, would enrich our understanding of the impact of different designs on team dynamics. Despite of these limitations, the paper is the first initiative to explore the possibility of supporting teamwork in MCRs by supplying team member’s operational status and execution history to each other. Whereas integrating team members’ information into the EOP display does reduce communication effort and improves team transparency, the teams using the integrated EOP design were slower in completing the tasks. The trade-off between ‘providing more information’ and ‘avoid cognitive overload’ needs further examination. There are two possible approaches that future research may explore: (1) identifying and supplying more pertinent information related to the task, the team and the individuals (Belkadi et al. 2013); (2) creating more effective representations with proper abstraction so that people can easily perceive and integrate a variety of information (Dabbish and Kraut 2008). Acknowledgements This study was supported by the National Natural Science Foundation of China (Project no. 70931003).

Disclosure statement No potential conflict of interest was reported by the authors.

References Bardram, J. E., T. R. Hansen, and M. Soegaard. 2006. “AwareMedia: a Shared Interactive Display Supporting Social, Temporal, and Spatial Awareness in Surgery.” In Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work, 109– 118. New York, NY: ACM. doi:10.1145/1180875.1180892. Belkadi, F., E. Bonjour, M. Camargo, N. Troussier, and B. Eynard. 2013. “A Situation Model to Support Awareness in Collaborative Design.” International Journal of Human –Computer Studies 71 (1): 110– 129. doi:10.1016/j.ijhcs.2012.03.002. Berggren, P., B. J. Johansson, N. Baroutsi, and N. Dahlba¨ck. 2014. “The Shared Priorities Measure as a Way of Assessing Team Strategic Awareness: A Bridge Between Self-Assessment and the Deep Blue Sea of Field Recordings.” In ECCE ’14: Proceedings of the 2014 European Conference on Cognitive Ergonomics, 13:1– 13:8. ACM. doi:10.1145/2637248.2637265. Birnholtz, J., N. Bi, and S. Fussell. 2012. “Do You See That I See?: Effects of Perceived Visibility on Awareness Checking Behavior.” In CHI ’12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1765– 1774. ACM. doi:10.1145/ 2207676.2208308. Bradley, C. 1994. “The well-being questionnaire.” In HandBook of Psychology and Diabetes: A Guide to Psychological Measurement in Diabetes Research and Practice, edited by C. Bradley, 89 – 109. London: Harwood Academic. Cannon-Bowers, J. A., E. Salas, and S. A. Converse. 1993. “Shared Mental Models in Expert Team Decision Making.” In Individual and Group Decision Making: Current Issues, edited by N. J. Castellan Jr., 221– 246. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Downloaded by [New York University] at 18:54 27 July 2015

1312

Q. Gao et al.

Carvalho, P. V., I. L. Dos Santos, and M. C. Vidal. 2005. “Nuclear Power Plant Shift Supervisor’s Decision Making During Microincidents.” International Journal of Industrial Ergonomics 35 (7): 619– 644. doi:10.1016/j.ergon.2005.01.010. Chung, Y. H., W. C. Yoon, and D. Min. 2009. “A Model-Based Framework for the Analysis of Team Communication in Nuclear Power Plants.” Reliability Engineering & System Safety 94 (6): 1030– 1040. doi:10.1016/j.ress.2008.11.010. Dabbish, L., and R. Kraut. 2008. “Research Note—Awareness Displays and Social Motivation for Coordinating Communication.” Information Systems Research 19 (2): 221– 238. doi:10.1287/isre.1080.0175. Frentrup, M., and L. Theuvsen. 2006. “Transparency in Supply Chains: Is Trust a Limiting Factor.” In Trust and Risk in Business Networks, edited by M. Fritz, U. Rickert, and G. Schiefer, 65 – 74. Bonn: ILB-Press. Funke, G. J., and S. M. Galster. 2009. “The Effects of Cognitive Processing Load and Collaboration Technology on Team Performance in a Simulated Command and Control Environment.” International Journal of Industrial Ergonomics 39 (3): 541– 547. doi:10.1016/j. ergon.2008.10.007. Gao, Q., Y. Wang, F. Song, Z. Li, and X. Dong. 2013. “Mental Workload Measurement for Emergency Operating Procedures in Digital Nuclear Power Plants.” Ergonomics 56 (7): 1070– 1085. doi:10.1080/00140139.2013.790483. Greenberg, Caprice C., Scott E. Regenbogen, David M. Studdert, Stuart R. Lipsitz, Selwyn O. Rogers, Michael J. Zinner, and Atul A. Gawande. 2007. “Patterns of Communication Breakdowns Resulting in Injury to Surgical Patients.” Journal of the American College of Surgeons 204 (4): 533– 540. doi:10.1016/j.jamcollsurg.2007.01.010. Gutwin, C., and S. Greenberg. 2002. “A Descriptive Framework of Workspace Awareness for Real-Time Groupware.” Computer Supported Cooperative Work (CSCW) 11 (3-4): 411– 446. doi:10.1023/A:1021271517844. Hart, S. G. 2006. “NASA-Task Load Index (NASA-TLX); 20 Years Later.” In Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Vol. 50, No. 9, 904– 908. Santa Monica, CA: Sage Publications. Hart, S. G., and L. E. Staveland. 1988. “Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research.” Human Mental Workload 1 (3): 139– 183. Hendy, K. C., K. M. Hamilton, and L. N. Landry. 1993. “Measuring Participantive Workload: When is One Scale Better Than Many?” Human Factors: The Journal of the Human Factors and Ergonomics Society 35 (4): 579– 601. Hirotsu, Y., K. Suzuki, M. Kojima, and K. Takano. 2001. “Multivariate Analysis of Human Error Incidents Occurring at Nuclear Power Plants: Several Occurrence Patterns of Observed Human Errors.” Cognition, Technology & Work 3 (2): 82 – 91. doi:10.1007/ PL00011526. Hwang, F. -H., and S. -L. Hwang. 2003. “Design and Evaluation of Computerized Operating Procedures in Nuclear Power Plants.” Ergonomics 46 (1– 3): 271– 284. doi:10.1080/00140130303517. Kang, H. G., and P. H. Seong. 1998. “An Information Theory-Based Approach for Quantitative Evaluation of User Interface Complexity.” Nuclear Science, IEEE Transactions On 45 (6): 3165– 3174. doi:10.1109/23.736195. Kim, Y., J. Kim, S. -c. Jang, and W. Jung. 2012. “Empirical Investigation of Communication Characteristics Under a Computer-Based Procedure in an Advanced Control Room.” Journal of Nuclear Science and Technology 49 (10): 988– 998. doi:10.1080/00223131. 2012.723172. Kozlowski, S., and B. S. Bell. 2003. “Work Groups and Teams in Organizations.” In Handbook of Psychology: Industrial and Organizational Psychology, edited by W. C. Borman, D. R. Ilgen, and R. J. Klimoski. Vol. 12, 333– 375. London: Wiley. LePine, J. A., R. F. Piccolo, C. L. Jackson, J. E. Mathieu, and J. R. Saul. 2008. “A Meta-Analysis of Teamwork Processes: Tests of a Multidimensional Model and Relationships With Team Effectiveness Criteria.” Personnel Psychology 61 (2): 273– 307. doi:10.1111/ j.1744-6570.2008.00114.x. Lew, R., R. L. Boring, and T. A. Ulrich. 2014. “A Prototyping Environment for Research on Human-Machine Interfaces in Process Control Use of Microsoft WPF for Microworld and Distributed Control System Development.” In Proceedings of the 7th International Symposium on Resilient Control Systems (ISRCS), 1 – 6. Denver, CO, August 19 – 21. doi:10.1109/ISRCS.2014. 6900111. Lim, B., and K. J. Klein. 2006. “Team Mental Models and Team Performance: A Field Study of the Effects of Team Mental Model Similarity and Accuracy.” Journal of Organizational Behavior 27 (4): 403–418. doi:10.1002/job.387. Lin, C. J., T. Hsieh, P. Tsai, C. Yang, and T. Yenn. 2011. “Development of a Team Workload Assessment Technique for the Main Control Room of Advanced Nuclear Power Plants.” Human Factors and Ergonomics in Manufacturing & Service Industries 21 (4): 397– 411. doi:10.1002/hfm.20247. Lingard, L. 2011. “Beyond “Communication Skills”: Research in Team Communication and Implications for Surgical Education.” In Surgical Education: Theorizing an Emerging Domain, edited by H. Fry and R. Kneebone, 199– 213. New York: Springer. Marks, M. A., and F. J. Panzer. 2004. “The influence of team Monitoring on Team Processes and Performance.” Human Performance 17 (1): 25 – 41. doi:10.1207/S15327043HUP1701_2. Mathieu, J. E., T. S. Heffner, G. F. Goodwin, E. Salas, and J. A. Cannon-Bowers. 2000. “The Influence of Shared Mental Models on Team Process and Performance.” Journal of applied Psychology 85 (2): 273– 283. doi:10.1037/0021-9010.85.2.273. Mesmer-Magnus, J. R., and L. A. DeChurch. 2009. “Information Sharing and Team Performance: A Meta-Analysis.” Journal of Applied Psychology 94 (2): 535– 546. doi:10.1037/a0013773. Min, D. H., Y. H. Chung, and W. C. Yoon. 2004. “Comparative Analysis of Communication at Main Control Rooms of Nuclear Power Plants.” Proceedings of IFAC/IFIP/IFORS/IEA Symposium, Atlanta, GA. Mohammed, S., L. Ferzandi, and K. Hamilton. 2010. “Metaphor No More: A 15-Year Review of the Team Mental Model Construct.” Journal of Management 36 (4): 876– 910. doi:10.1177/0149206309356804. Mohnen, A., K. Pokorny, and D. Sliwka. 2008. “Transparency, Inequity Aversion, and the Dynamics of Peer Pressure in Teams: Theory and Evidence.” Journal of Labor Economics 26 (4): 693– 720. doi:10.1086/591116. Niwa, Y., and E. Hollnagel. 2002. “Integrated Computerisation of Operating Procedures.” Nuclear Engineering and Design 213 (2 – 3): 289– 301. doi:10.1016/S0029-5493(01)00464-2. Norman, D. A. 2002. The Design of Everyday Things. New York: Basic Books.

Downloaded by [New York University] at 18:54 27 July 2015

Ergonomics

1313

O’Hara, J., W. Brown, P. Lewis, and J. Persensky. 2002. Human-System Interface Design Review Guideline. (NUREG-0700, Rev. 2). Washington, D.C: U.S. Nuclear Regulatory Commission. O’Hara, J., and J. Higgins. 2010. “Human-System Interfaces to Automatic Systems: Review Guidance and Technical Basis.” Human Factors of Advanced Reactors (NRC JCN Y-6529) BNL Tech Report No BNL91017. O’Hara, J. M., J. C. Higgins, W. F. Stubler, and J. Kramer. 2000. Computer-Based Procedure Systems: Technical Basis and Human Factors Review Guidance: (NUREG/CR-6634). Washington, D.C: U.S. Nuclear Regulatory Commission. Ozcelik, H., N. Langton, N, and H. Aldrich. 2008. “Doing Well and Doing Good: The Relationship Between Leadership Practices that Facilitate a Positive Emotional Climate and Organizational Performance.” Journal of Managerial Psychology 23 (2): 186– 203. Palanski, M. E., S. S. Kahai, and F. J. Yammarino. 2011. “Team Virtues and Performance: An Examination of Transparency, Behavioral Integrity, and Trust.” Journal of Business Ethics 99 (2): 201– 216. doi:10.1007/s10551-010-0650-7. Pawlak, W. S., and K. J. Vicente. 1996. “Inducing Effective Operator Control Through Ecological Interface Design.” International Journal of Human– Computer Studies 44 (5): 653– 688. doi:10.1006/ijhc.1996.0028. Perlman, G. 1987. “An Axiomatic Model of Information Presentation.” In Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Vol. 31, No. 11, 1229– 1233. Santa Monica, CA: SAGE Publications. Rasker, P. C., W. M. Post, and J. M. C. Schraagen. 2000. “Effects of Two Types of Intra-Team Feedback on Developing a Shared Mental Model in Command & Control Teams.” Ergonomics 43 (8): 1167–1189. doi:10.1080/00140130050084932. Roth, E. M., J. Multer, and T. Raslear. 2006. “Shared Situation Awareness as a Contributor to High Reliability Performance in Railroad Operations.” Organization Studies 27 (7): 967– 987. doi:10.1177/0170840606065705. Roth, E. M., and J. M. O’Hara. 1999. “Exploring the Impact of Advanced Alarms, Displays, and Computerized Procedures on Teams.” In Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Vol. 43, No. 3 158–162. SAGE Publications. Scott, S. D., F. Sasangohar, and M. L. Cummings. 2009. “Investigating Supervisory-Level Activity Awareness Displays for Command and Control Operations.” Proc. of HSIS: Human Systems Integration Symposium, Annapolis, MD. Stone, D., C. Jarrett, M. Woodroffe, and S. Minocha. 2005. User Interface Design and Evaluation. London: Morgan Kaufmann. Stra¨ter, O. 2003. “Investigation of Communication Errors in Nuclear Power Plants.” In Communication in High Risk Environments, edited by R. Dietrich, 155– 179. Hamburg: Helmut Buske Verlag. Su, K. -W., C. -J. Chen, and L. -Y. Shue. 2013. “Implication of Cognitive Style in Designing Computer-Based Procedure Interface.” Human Factors and Ergonomics in Manufacturing & Service Industries 23 (3): 230– 242. doi:10.1002/hfm.20315. Undre, S., N. Sevdalis, A. N. Healey, S. A. Darzi, and C. A. Vincent. 2006. “Teamwork in the Operating Theatre: Cohesion or Confusion?” Journal of Evaluation in Clinical Practice 12 (2): 182– 189. doi:10.1111/j.1365-2753.2006.00614.x. Vidal, M. C. R., P. V. R. Carvalho, M. S. Santos, and I. J. L. d. Santos. 2009. “Collective Work and Resilience of Complex Systems.” Journal of Loss Prevention in the Process Industries 22 (4): 516– 527. doi:10.1016/j.jlp.2009.04.005. Wen, F. 2011. “AP1000 Operation Procedures System and Development.” Journal of Energy and Power Engineering 5 (2011): 711– 717. Wickens, C. D., S. E. Gordon, and Y. Liu. 2004. An Introduction to Human Factors Engineering. Upper Saddle River, NJ: Pearson Prentice Hall. Xu, S., F. Song, Z. Li, Q. Zhao, W. Luo, X. He, and G. Salvendy. 2008. “An Ergonomics Study of Computerized Emergency Operating Procedures: Presentation Style, Task Complexity, and Training Level.” Reliability Engineering & System Safety 93 (10): 1500– 1511. doi:10.1016/j.ress.2007.09.006. Yin, S. 2012. “Proactive Monitoring in Process Control Using Predictive Trend Analysis.” Doctoral dissertation. Retrieved from repository.ntu.edu.sg: http://repository.ntu.edu.sg/handle/10356/49971