Cognitive simulation as a tool for cognitive task analysis.

This article was downloaded by: [University of Cincinnati Libraries] On: 04 January 2015, At: 15:22 Publisher: Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Ergonomics Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/terg20

Cognitive simulation as a tool for cognitive task analysis a

b

EMILIE M. ROTH , DAVID D. WOODS & HARRY E. POPLE Jr.

c

a

Westinghouse Science and Technology Center , 1310 Beulah Road, Pittsburgh, PA, 15235, USA b

Cognitive Systems Engineering Laboratory , The Ohio State University , Columbus, OH, USA

c

University of Pittsburgh and Seer Systems , Pittsburgh, PA, USA Published online: 31 May 2007.

To cite this article: EMILIE M. ROTH , DAVID D. WOODS & HARRY E. POPLE Jr. (1992) Cognitive simulation as a tool for cognitive task analysis, Ergonomics, 35:10, 1163-1198, DOI: 10.1080/00140139208967389 To link to this article: http://dx.doi.org/10.1080/00140139208967389

PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http:// www.tandfonline.com/page/terms-and-conditions

Cognitive simulation as a tool for cognitive task analysis

Downloaded by [University of Cincinnati Libraries] at 15:22 04 January 2015

Westinghouse Science and Technology Center, 1 3 10 Beulah Road, Pittsburgh, PA 1 5235, USA

Cognitive Systems Engineering Laboratory, The Ohio State University, Columbus, OH, USA

HARRY E. POPE, Jr. University of Pittsburgh and Seer Systems, Pittsburgh, PA, USA Keywords: Computer simulation; Fault management; Nuclear power plant; Task analysis;

Artificial intelligence; Cognitive processes. Cognitive simulations are runnable computer programs that represent models of human cognitive activities. We show how one cognitive simulation built as a model of some of the cognitive processes involved in dynamic fault management can be used in conjunction with small-scale empirical data on human performance to uncover the cognitive demands of a task, to identify where intention errors are likely to occur, and to point to improvements in the person-machine system. The simulation, called Cognitive Environment Simulation or CES, has been exercised on several nuclear power plant accident scenarios. Here we report one case to illustrate how a cognitive simulation tool such as CES can be used to clarify the cognitive demands of a problem-solving situation as part of a cognitive task analysis.

1. Introduction Methods for cognitive task analysis occupy a central position in cognitive science and cognitive systems engineering research and application (Hollnagel and Woods 1983, Woods and Rorh 1988a). A variety of methods have been developed and utilized to uncover and describe t h e knowledge and cognitive processes that underlie human performance in particular task domains. These methods are a critical part of the process of developing training and decision aids (e.g., Lesgold 1990, Woods and Roth l988b, 1 989, Redding 1989). There are several basic approaches that have been used to carry out cognitive task analyses. One approach relies on an analysis of the application domain to uncover the cognitive demands inherent in the situation. This is usually based on some variation of goal-means decomposition. Examples include (a) the Bolt Beranek and Newman goal decomposition approach (e-g., Corker et af. 1986); (b) techniques for function based goal-means analysis (e.g., Woods and Hollnagel 1987) that are derived from Jens Rasmussen's abstraction hierarchy (Rasmussen 1986) and Morten Lind's multi-level flow modelling (e.g., Lind 1991); and techniques for analysing operator functions which are based on discrete control models of the task (e.g., Mitchell and Miller 1986). The goal-means decomposition methods focus on building a model of the cognitive demands of the tasks, that is, identifying constraints on task performance described in terms of a language of information processing. The demands identified in these techniques can be such things as


1164

E. M. Rolh et al.

constraints on goal satisfaction (e.g., what goals are relevant during different epochs in a task, what specific criteria must be met to satisfy the goal in different epochs, when are the windows of opportunity), constraints on goal-means relationships (e.g., what resources are available for responding to disturbances including preferred methods, back-ups, staged degrees of response, main-effect-side-effects reiationships, models of how means function to affect goals), and goal-goal constraints (e-g., contexts where competition or interaction between goals arise). Goal-means decomposition models are used to build a problem space representation that describes the kinds of cognitive situations that arise in the course of carrying out domain tasks in different kinds of situations. The resulting problem space has been used successfully to map information requirements linked to different domain contexts and states as the basis for developing new representations that aid practitioner performance (cf., Mitchell and Saki 1987, Woods and Hollnagel 1987 and Easter 1987 for descriptions of specific cases; cf., Woods 199 1 for a general description of representation aiding). The problem space description can be used t o reveal complex situations that will be difficult for any information processing agent, human or machine to deal with (e.g., Roth and Woods 1988). For example, under some domain circumstances, routine plans may be inadequate because multiple but competing constraints need to be satisfied simultaneously. This cognitive situation may force the problem solver to adapt routine plans to negotiate a narrow path that satisfies all of the relevant constraints or to make a judgement about the relative priority of the different constraints in the context. In this way the goal-means decomposition methods are used to specify the cognitive, demands of the task domain, i.e., the requirements for competent performance. The result can be thought of as a model of the problem-solving environment: what kinds of problem-solving situations can arise? What must people know and how must they use that knowledge to handle these problems? What information must be extracted to monitor and assess process state? What knowledge must be activated and utilized to select goals or to monitor and adapt plans? A second approach to cognitive task analysis is to use various empirical techniques for studying practitioner behaviour in situ or in high fidelity environments to uncover how people use information and knowledge to accomplish domain tasks.[ Examples of this 'ethnographic' approach include Pew et al. (1981), Roth and Woods (1 988), Klein et al. (1 9891, Lesgold e/ al. (1 990), Hutchins (I 991), Sarter and Woods (1 991), Cook el a/. ( 1 99 1), and Means et al. (1 988). Specific techniques that are used in these studies include protocol analysis, naturalistic observation, and critical incident analysis. These studies are concerned with understanding the task relevant knowledge and information processing strategies actually used by domain practitioners. Thus, this approach is focused on developing performance models-identifying what practitioners do, both successfully and erroneously, given the demands of the task and the available external resources (pre-planned routines, decision aids, data and displays). The performance model that is built up from these empirical investigations

'There is another approach to CTA that uses empirical techniques such as classificationand sorting tasks to assay the knowledge organization of experts in a specific domain of application (e.g., Cooke and McDonald 1987). Various techniques for psychological scaling are used to extract from the data a model of how experts represent domain knowledge.


Cognitive simulation and rask analysis

1165

identifies ineffective or brittle strategies and the circumstances where they lead to poor performance, identifies adaptive strategies that have been developed by skilled or expert practitioners to meet task demands, and specifies the positive and negative effects of external devices and displays on practitioner information processing (for a positive example cf., Hutchins 1 99 1 , which investigates a case where physical devices are utilized to function as external memories that omoad working memory; for a negative example cf., Cook et al. 1 99 1, where an opaque interface hides system state and increases the potential for error). The thesis of this paper is that cognitive simulation can provide another complementary approach for cognitive task analysis. Cognitive simulations are runnable computer programs that represent models of human cognitive activities (Simon 1 969). We show how one cognitive simulation built as a model of some of the cognitive processes involved in dynamic fauit management can be used in conjunction with other data sources to uncover the cognitive demands of a task, to identify where intention errors are likely to occur, and t o point to improvements in the person-machine system. 2. Cognitive simulation Cognitive simulation is a technique invented by Newel1 and Simon (Simon 1969, Newel1 and Simon 1972) where information processing concepts about human cognitive activities are expressed as a runnable computer program, usually through symbolic processing techniques (e-g., Holland er al. 1986, Laird et al. 1987, Newel1 1990). There is a growing trend to use cognitive simulations to capture the cognitive demands imposed by dynamic fault-management situations (e-g., Thompson el al. 1983, Corker et a!. 1986, Woods et a!. 1987, Amendola et al. 1987, Elkind et al. 1990). The cognitive simulation can be stimulated by inputs from a domain scenario to generate model behaviour which can be compared to observed human behaviour for the same scenario. The advantages of cognitive simulation revolve around the fact that building a runnable computer program forces the modeller to describe mechanisms in great detail. Running the simulation through a scenario produces specific behaviour that can be analysed and compared to other data. As a result it is possible to uncover a variety of consequences of the basic information processing mechanisms that are instantiated in the program. Furthermore, the resulting simulation can be run on a variety of scenarios, including scenarios that were not part of the original design set. Thus, ,the implications of assumptiondconcepts about human cognitive activities captured by the simulation can be explored in a wide range of domain-specific circumstances. As a result, one can see cognitive simulation as a method for more directly linking theory-building and empirical investigations of human problem-solving activities in semantically rich domains.

2.1. Cognirive Environment Simulation (CES) We have been developing a cognitive simulation of the cognitive activities involved in fault management under dynamic conditions in the context of a US Nuclear Regulatory Commission research program to model Nuclear Power Plant (NPP) operator cognitive activities (Woods el al. 1987, 1988, 1990; Roth et al. in preparation). The simulation is called Cognitive Environment Simulation or CES to highlight the cmphasis on capturing the demands imposed by the problem-solving environment. CES is an artificial intelligence (AI) system that simulates the cognitive activities

.


E. M. Rorh et al. involved in dynamic fault management. CES monitors and tracks changes in process state, identifies abnormal and unexpected process behaviours, builds and revises a situation assessment (what influences are currently acting on the monitored process), formulates hypotheses to account for unexplained process behiviour, and formulates intentions to act based on its situation assessment. CES is built on top of an A1 problem-solving system called EAGOL2 that was developed especially for performing diagnostic reasoning in dynamic, data-rich situations. EAGOL incorporates qualitative and abductive reasoning techniques. It has been developed based on detailed studies of cognitive activities in a variety of domains, including internal medicine diagnosis (Gadd and Pople 1990), emergency operations in nuclear power plants (Woods and Roth 1986), NASA shuttle operations, critical care medicine, and intelligence analysis. In the sections that follow we begin by providing a description of the critical features of dynamic fault management situations that distinguish them from other types of domains and pose challenges to cognitive modelling. We then provide a description of CES and how it handles dynamic fault management situations. This is followed by a NPP case study that illustrates how a cognitive simulation such as CES can be used to help elucidate the cognitive demands imposed by a class of domain problems. We end with a discussion of the role a cognitive simulation can play in cognitive task analysis. 2.2. Dynamic fault management Fault management in dynamic applications has a different character from the classic paradigm of troubleshooting a broken device that has been removed from service. In fault management there is some underlying process (an engineered or physiological process which will be referred to as the monitored process) whose state changes over time. Faults disturb the monitored process and diagnosis goes on in parallel with responses to maintain process integrity. These situations frequently involve time pressure, multiple interacting goals, severe consequences of failure, and multiple interleaved tasks. In dynamic process applications, incidents extend, develop and change over time. A fault disturbs the monitored process and triggers influences that produce a time dependent set of disturbances. This cascade of disturbances unfolds over time due to the development of the fault itself (e.g., a leak growing into a break) and due to functional and physical interconnections within the monitored process (Woods 1988). These situations are further complicated by the possibility of multiple faults each producing a temporally evolving set of disturbances which can interact. In dynamic fault management, the monitored process is not and usually cannot be removed from service. This means that the fault manager needs to try to continue to meet some of the goals of the monitored process (e.g., safety) while attempting to uncover the source of the process disturbance. The relative importance of different process goals may change as the incident evolves and some goals may need to be abandoned if they compete with more critical goals. Thus, fault diagnosis occurs as part of a larger context where the expert practitioner must maintain system integrity by coping with the consequences of faults in parallel with untangling the causal chain that underlies these disturbances. 'EAGOL is a proprietary product of Seer Systems.

Cognitive simulation and task analysis

1167


A complicating characteristic of these worlds is that during the course of the event both manual and automatic system actions will be taken to cope with the initiating disturbance that themselves affect the monitored process. The effect of these actions on the process further complicate the diagnostic task. The current and future state of the monitored process is a combined function of control influences and the influences produced by the faults in the monitored process (Woods et al. 1990). Thus there is a need to maintain and constantly update the set of normal and abnormal influences impinging on a process at any given point in time in order to account for plant behaviour and untangle the effects due to the underlying fault from the effects due to subsequent control actions.

2.3. CES as a cognitive simulation of dynamic fault management The Cognitive Environment Simulation is one specific example of a cognitive simulation designed to handle some of the demands of dynamic fault management situations (cf. also, Johnson el al. 1 988). CES contains specific symbolic processing mechanisms designed to:

0

0

build and maintain a coherent situation assessment in a changing environment where multiple influences are at work including faults and operator and automatic system activities; discriminate expected from unexpected events based on qualitative reasoning about influences thought to be acting on the monitored process; engage in diagnostic search to evaluate possible hypotheses that would explain unexpected findings given that multiple influences are acting on the monitored process; generate intentions to take action to respond to diagnosed faults andlor reestablish safety goals.

CES represents one example of a growing class of recognition-drive models of decision making (Klein 1989). Specifically, CES information processing is anomalydriven. Follow-up diagnostic search is based on a layered distributed agent approach to abductive reasoning. The inherent variability of dynamic systems means that there are a large number of changes in the monitored process (and the absence of changes) that could be relevant in principle to situation assessment, diagnosis and action selection. This calls for some mechanism to extract from the continuous flow those changes or absences of change that are significant. To do this cffcctively requires a balance between two kinds of errors: avoid errors of devoting process resources to too many irrelevant changes (data overload) as well as errors of discarding too many potentially relevant changes. For AI systems, if every change invokes the full diagnostic reasoning capabilities of the A1 system, then the system will be unable to build a coherent situation assessment given the variability in dynamic physical systems. The solution for CES as for other A1 reasoners designed to handle dynamic situations (e-g., Abbott 1990) is to create a monitor stage that identifies 'significant' findings from the larger set of changes on the monitored data channels. Only these significant findings trigger additional information processing activities. Within the CES simulation 'significant' findings fall into several classes of anomalies in process behaviour, and an initial stage of analysis is carried out to recognize these anomalies. Abnormal process behaviour is an anomaly where actual state does not match desired state for a given operating context. Unexpected process


E. M. Rorh et al. behaviour is a mismatch between actual process behaviour and the cognitive agent's model of the situation. Recognition of an anomaly triggers follow up lines of reasoning within the system. One line of reasoning is concerned with how to manage abnormalities including selecting control actions from stored doctrine (intentions to act), monitoring for the expected effects of these actions, and detecting failures of these responses to work as expected (e.g., execution errors). Another line of reasoning is concerned with diagnostic search to determine what unknown influences could be acting on the monitored process that would account for unexpected process behaviour. These information processing activities are camed out by a set of software 'agents' each with a distinct responsibility and communications protocol. One kind of software agent is activated when the data changes on an input channel. This type of agent uses information about the set of influences thought to be active currently, and knowledge about influence relationships (an increasing flow will result in an increase tank level) to determine if the change is expected or unexpected given the current situation assessment. Qualitative reasoning techniques are employed to generate expectations about process behaviour based on the set of known influences (Forbus 1988). For example, increasing tank level is expected if an influence has been posted that there is a source of flow into the tank. If the current set of known influences cannot account for the observed process behaviour (an anomaly), an unknown influence is postulated and another software agcnt is invoked to identify the unknown influence. For example, an unexpected change in direction or rate of change invokes an agent that uses knowledge of influence relationships to create a list of hypotheses-influences that could account for the unexpected behaviour. It then engages in knowledge-driven search to evaluate each hypothesis against other data about the state of the monitored process. During an evolving incident, CES creates many software agents. T o integrate the evaluations across the multiple agents, CES creates a special software agent that is responsible for coming up with a coherent explanation for all of the unexplained findings. These two layers in the architecture are needed to enable the A1 system to separate and track the multiple factors affecting process behaviour (automatic system responses, manual responses, and influences created by one o r more faults), especially as they evolve dynamically. Does a new anomaly represent disturbance propagation from the original fault, a sensor failure, or the effects of another breakdown? Control influences inserted to mitigate the consequences of disturbances change the pattern of findings, and expected control influences may fail to materialize due to execution error or additional breakdowns. The multiple layers of analysis allow the system to track multiple influences that may be affecting process state and to investigate different ways to put the pieces together to form a coherent .overall assessment (Woods et a!. 1987). 3. Cognitive simulation in cognitive task analysis: nuclear power plant incidents Applying CES to dynamic fault management in NPPs requires building a knowledge base that contains information about NPP functions, faults, goals, and control actions (the influence relationships needed for the qualitative and abductive reasoning mechanisms in the program). The CES knowledge base includes information on plant parameters available to be monitored, and their normal operating limits; the inter-relationships among plant physical processes; goals for safe plant operation; abnormalities (e.g., power failures; breaks) and the effect they have


Cognirive simulation and task analysis

1169

on plant processes; and what actions can be taken to correct abnormalities. As such it provides a mechanism for modelling the kind of knowledge of NPP that an operator would be presumed to have based on training, procedures, and experience. As described above, the inference engine within CES provides reasoning mechanisms that enable CES to monitor changing plant parameters, to formulate and revise situation assessments, and to generate intentions to act. While the particular reasoning mechanisms utilized by CES d o not mimic in detail the cognitive processes of human operators (e.g., short term memory; detailed monitoring or diagnostic strategies), it performs the major cognitive activities that are required to successfully assess and respond to a NPP emergency event (i.e., the cognitive tasks that human operators would necessarily have to perform to successfully handle the event). By performing the major cognitive activities required to handle a NPP emergency event, it provides a tool for assessing the cognitive challenges imposed by different accident sequences (e.g., what evidence needs to be examined, what knowledge needs to be accessed, what alternative hypotheses arise that need to be discriminated, and what safety goals need to be considered in managing the accident). The CES modelling strategy to date has focused on developing knowledge and reasoning capabilities that enable CES to handle successfully the NPP case of interest and close variants. The premise is that the knowledge and reasoning capabilities that are required for the CES computer simulation to handle successfuliy the case provide a specification of the knowledge and reasoning demands imposed by the domain tasks. I n keeping with the cognitive simulation tradition, the information processing behaviour of the simulation provides a concrete basis for comparison with empirical data on practitioner behaviour. Analysis of similarity and differences in the performance of the computer simulation and that of human practitioners can illuminate the cognitive demands imposed by the domain, the knowledge and reasoning capabilities (cognitive competencies) required for successful performance on the task, and knowledge and information processing limitations of practitioners that restrict performance. In this way a computer simulation can serve to illuminate and amplify available data on human performance as part of a cognitive task analysis. The goal of the simulation exercise is not to produce an exact match between the information processing behaviour of the computer simulation and that of the crew. Because of the breadth of information processing activities involved in these situations, and both pragmatic and theoretical limits on the ability of computer simulations to embody a comprehensive model of human cognition, a computer simulation is necessarily only a partial embodiment of a larger scope conceptual model of human performance (Newell 1990). The goals of the simulation enterprise are to learn from juxtaposing the behaviour of the simulation and the behaviour of human practitioners. By examining the similarities and differences between simulation behaviour and crew behaviour it becomes possible to understand the cognitive demands of the situation, the sources of error, and potential for expertise.

3.1 . NPP accident scenario As part of the simulation development process CES has been exercised on several nuclear power plant accident scenarios (Woods el a/. 1990, Roth et ol. in preparation). Here we report one case to illustrate how a cognitive simulation tool


1170

E. M. Rorh et al.

such as CES can be used to clarify the cognitive demands of a problem-solving situation as part of a cognitive task analysis. We used CES to analyse a type of NPP incident called an interfacing system loss of coolant accident or ISLOCA. This incident involves a failure (e.g., a valve failure) that connects a high pressure system (the primary system-reactor coolant system or RCS, whose safety functions relate to cooling the reactor) to a low pressure system (Residua1 Heat Removal System or RHR system, whose safety functions relate to providing coolant inventory and a heat removal path when the RCS is depressurized). A simplified diagram of the main nuclear power plant system, including the RHR system, is presented in figure 1. There are two RHR isolation valves that allow a flow path between the primary system which is inside the containment building and the R H R system which is outside of containment. These valves are kept closed normally. In the incidents we ran the two valves were failed open. The fault couples the two systems together producing coolant (water) flow from the RCS (the high pressure system; normally at about 2235 psi) to the RHR system (the low pressure system normally at 300 to 400 psi) creating an ISLOCA. The fault is a particular type of primary system break-a primary system break to the RHR systerri as contrasted with a primary system break within containment or a break to some other part of the plant. The ISLOCA has several functional consequences. From the point of view of the RCS system, pressure is decreasing and coolant inventory (measured by the parameter pressurizer level) is decreasing. Looking at these symptoms alone the incident looks like a break in the primary system. Of course, it is a kind of primary system break, but a special kind with special threats and special corrective actions. From the point of view of the RHR system, high energy fluid is flowing inlo a system designed for low pressure operating conditions. There is an RHR relief valve that will open in an attempt to handle the overpressure challenge (see figure I), but eventually there will be a break in the RHR system resulting in primary coolant (which contains low levels of radioactivity) to be spilled into other parts of the plant. It is possible, if several other things go wrong as well, for the incident to develop into a-major threat to plant safety goals: the discharge of radioactive reactor coolant to the environment and loss of emergency core cooling capability because reactor coolant flows to parts of the plant where i t cannot be recirculated. The event is challenging diagnostically due to several factors. If one focuses only on the most salient early symptoms of the fault (i-e., primary system level and prcssure), they suggest a different diagnosis that is more salient to operational personnel: a primary system break inside containment called a LOCA. In addition interconnections among the NPP systems further complicate the diagnostic task. The RHR relief valve vents into the Pressurizer Relief Tank (PRT) inside of containment (see figure 1). The PRT eventually ruptures due to the pressure build-up releasing radioactive cooling water onto the containment floor. The radiation symptoms in containment that result reinforce the primary system break inside containment hypothesis. The complete set of symptoms that would indicate the actual nature of the fault build-up over time so that the primary system break to containment diagnosis becomes increasingly less plausible. This means that the incident has a garden path quality to it (Johnson el al. 1988). If one does identify the complete set of significant findings, putting the findings together into a diagnosis is difficult because the event produces symptoms in multiple regions of the plant that are normally unconnected (i.e., abnormal radiation



1171

in containment and in the RHR system). As a result, the pattern can be interpreted as indicating the presence of multiple independent faults (note that critical incidents in this domain almost always involve more than one failure (Woods and Roth 1986, Woods et al. 1987). ' Two variants of the ISLOCA incident were run. In both cases the R H R isolation valves were failed open. This led to an increase in pressure in the RHR sysrem which resulted in a break approximately three minutes into the event. In case 1 the break was in the R H R piping and led to reactor coolant fluid leaking to the floor of the auxiliary building. In case 2 the break was in a heat exchanger between the R H R and the Component Cooling Water (CCW)System (see figure I). This resulted in reactor coolant fluid leaking into the CCW system. This produces an increase in level and radiation in the CCW Surge Tank. Eventually, the Surge Tank overflows, causing increased radiation in the auxiliary building.

3-2. Cognitive demands of the accident scenarios: dala on crew perfarrnance The analysis involved collection and comparison of two types of 'data': ( 1 ) performance of human crews on the incidents; and (2) performance of CES on the same incidents. Two crews made up of NPP training instructors were observed trying to diagnose and respond to the simulated incidents in a full-scope high fidelity training simulator. One crew was observed in each incident. The instructors who participated were not aware of what emergency event was being simulated. The CES cognitive simulation was run on the same simulated cases (plant parameter values were recorded during the crew runs and used as input to CES). Protocols of the behaviour of the crews were compared with CES behaviour. Each crew (made up of two simulator training instructors) responded to one of the two versions of the ISLOCA incident. Their performance in the simulated events was observed and videotaped. The data were analysed by building behavioural protocols that captured the flow of information processing and knowledge activation in relation to the temporal flow of events in the evolving incident (Woods, in press). The crews eventually were able to diagnose the incident presented to them, but in both cases they experienced difficulty. Both crews found it necessary to step back and attempt to come up with a single explanation to account for what seemed at first blush to be unconnected findings. Both crews pulled out schematic diagrams (in case I-the R H R system, in case 2-both the RHR and the CCW systems) to explore the physical interconnections among the systems. In both cases the available procedural guidance directed the operators to the primary break inside containment procedure (LOCA procedure) instead of the ISLOCA procedure. Later, both crcws explicitly decided to interrupt this procedure and shift to diagnosing and managing the disturbances in the RHR system. Figures 2 and 3 show the behavioural protocols for crew performance during each of the simulated incidents. Crew activities flow top to bottom and left to right. They are classified into three major categories:

0

0

monitoring activities-acknowledging an alarm, reading a parameter value, or otherwise obtaining plant parameter status information; inrerpretation-stating a hypothesis or situation assessment, drawing a conclusion, or forming an intention to act; control activities-taking a control action (e.g., following a set of procedures).

The approximate time of occurrence of each activity, measured from beginning of the run, is noted in parentheses (min: s).


Water Storage Tank

aa'v Surge Tank

I RHR Heat Exchanger \

I

CCW Pump

RHR Punp Component Cooling Water System

Residual Heat Removal System

Figure I .

-

Simplified diagram of nuclear power plant systems.


.

. * Reactor

RHR Pump

Detect

- Discharge Pressurc Alarm (:lo) PRT Temp. High ( :1 5)

'

Trip ( :45)

*

RHR Room

High

Sump Level

Rad. in

High (6 :00)

Containment ( 9 :OO)

PRT Ruptured

High Rad. in Auxiliary Building (1 5 :3O)

b

Told RHR isolation valve position unknown (24 :30)

Low Pressurizer Pressure ( :20) Low Pressurizer l e v e l (:20:1

Interpret

"Got a leakn ( :20)

-

Go to

0

EO Proc. (Reactor Trip Proc. ( :45)

Problem in RHR Room and lost level in Pressurizer

(8:30)

-

Go to E l Proc. (LOC A Proc.) (9 :OO)

8

Leak in containment and leak in RHR Pump room Containment symptwns due to RHR relief valve (1 6 $30) Realize missed kickout to ISLOCA procedure Dcoide to check ECA 1.2 ( ISLOCA proo.) (20 :OO)

-

Examine RHR Schematic Decide unisolsterblc ISLOCA (30 a(3)

-

-

Control

Follow EO Proc.

-

0

Follov El Proc.

I

-

Ask r e m t e operator to check position o f RHR isolation valves (21 :00) 0

Resume E l Proc. (22 :3D)

Continue slow cooldown h safety injection reduction sequence.

-

Start cooldown (22 :30) Value in parentheses is the approximate time when the operator activity occurred (minutes : seconds from start of event). In the case o f detection, the time indicated is the time at which the operators verbally mentioned the plant behaviour (rather than when i t occurred).

Figure 2. Crew decision flow in case 1 ISLOCA: break in residual heat removal ( R H R ) piping.


E. M. Roth et al. 3.2.1. Crew response in case 1: The performance of the crew in the version of the incident where the R W R piping ruptures spilling water into the auxiliary building is presented in figure 2. The first symptom is an RHR discharge pump high pressure alarm. The operators detected this alarm and noted a potential problem in the RHR system. However, almost immediately, symptoms in the primary system began to appear, i.e., low primary system coolant level and low primary system pressure. The crew noted these anomalies in the primary system and they switched their focus of attention and processing resources to follow-up these findings. Next an automatic reactor trip occurred (the safety system automatically discontinues generating electricity and reconfigures plant systems to maintain all safety functions), and the operators followed the standard monitoring activities to verify that this reconfiguration occurred successfully. Current operational philosophy in nuclear power plants dictates that once the reactor trips, operator performance from that point should be directed by a set of emergency operating procedures. These procedures are intended to direct all monitoring and control activities taken by the operators. As a consequence once the reactor tripped the crew accessed a i d followed the emergency procedures. About 6 rnin into the incident, an RHR Room sump level high alarm came on. At around that point the operators entertained the hypothesis that there were two separate leaks: one producing the troubles in the RHR system and another producing the abnormal containment conditions. Shortly after an alarm came on indicating high radiation in containment (because the PRT had ruptured releasing radioactive fluid). When the high radiation in containment alarm occurred the procedures directed the operators to a procedure designed t o cope with primary breaks inside of containment (LOCA procedure). This procedure contains no explicit procedural guidance that can direct the operators to the procedures for handling an ISLOCA fault. At that point, both because of their interpretation of the patterns of symptoms, and because the procedure directed them to d o so, the operators turned to the LOCA procedure. At a later point a high radiation in the auxiliary building alarm came on. This caused the operators to step back and reflect actively on the set of symptoms. They noted that the relief valve on the R H R leads to the PRT and came to the conclusion that high pressure in the R H R produced both the auxiliary building symptoms and the containment symptoms. At that point they diagnosed the problem (approximately 16 rnin into the incident). They decided to begin to follow the procedure for this problem (whiie still 'officially' continuing to carry out the primary system break to containment procedure). The basic strategy in the relevant procedure is to first attempt to terminate the leak (e.g., attempt to close the RHR isolation valves). In the simulated incidents that were run, this could not be done. If attempts to isolate the leak fail, the concern shifts to taking actions to conserve other sources of coolant water (i.e., the refuelling water storage tank or RWST) and to begin a plant cooldown. While this crew diagnosed the problem and carried out the appropriate response strategy, they did have trouble integrating all the evidence to localize the source of the disturbances. Even though the first alarm suggested a problem in the RHR system and they explicitly noted this, the RHR problem was dropped for a period once symptoms in containment arose. In addition, they entertained the hypothesis that two separate faults were present in order to account for the full pattern of findings. It

RHR Pump


Dctea t

Discharge Presr Alarm ( :lo)

0

Symptoms

8

rn PUT Ruptured ( 9 :OO)

to EO Proo. (Reactor Trtp Proc .)

---

Terminrtod simulation at 24 :00

Go to CCV & RHR Schematics (1 0 :3O)

O o t o E l Proc. (LOCA Proc.) (3 :30)

( :30)

High RHR pump Disehrrge pressure (1 9:Oo)

CCV Surge Tank lnc. (10:00)

+ Rcrctor Trip ( :30)

8, GO

Containment (17:m)

Alarm (3 :45)

(3 :30)

Low Pressurizer Pressure ( :20) Low Pressurizer Level ( :20:1

Interpret

- 8 R a d . in

CCW High Rad.

Containment

Rwlizo that

8

RHR relief valve goes to PRT (1 3 :45) 8

Control

8 Follow

EO Proo.

Follow ElProc.

-

Decide leak i n RHR heat exchanger with CCW (1 4 :00)

a Ask remote operator t o close CCW isolation valve to RHR heat

-

-

exchanger terminating leak into CCV (1 6:15)

* Value in parentheses is the approximate t h e when the operator activity occurred (minutes: seconds from start In the case o f detection, the time indicated i s the time at which the oporrtors of verbally mentioned the plant bchaviour (rather than when i t occurred).

Figure 3. Crew decision flow in case 2 ISLOCA:break in component cooling water heat exchanger.

1176

E. M.Roth et al.


was approximately 16 minutes into the event, well past the point when the RHR pipe broke, that they fully diagnosed the situation.

3.2.2. Crew response in case 2: The performance of the crew in case 2 is shown in figure 3. This crew was presented with the same incident except that the RHR break occurred in the component cooling water (CCW) heat exchanger. As in case 1, they noted a problem in the RHR system when the RHR pump discharge pressure high alarm first came on. However they soon shifted their attention to containment symptoms. As in the case of the first crew, their assessment and the procedure led to them to the specific procedure for a primary break to containment when indications of radiation in containment appeared. Approximately 4 min after the start of the event, a high radiation in CCW alarm came on, followed by indications that the CCW surge tank level was increasing rapidly. The crew noted these findings and focused their attention on identifying from where contaminated water was entering the CCW. While still technically in the LOCA procedure, they stepped back to investigate the CCW problem and examined schematics of the CCW and RHR systems. In this case the challenge is to understand how contaminated fluid from the reactor coolant system could reach the CCW, and how the problem in the CCW could be related to the PRT rupturing, and the radiation inside containment. In particular, diagnosis requires activating knowledge that a primary system break to the RHR could produce a break in the CCW heat exchanger, creating a flow path for reactor coolant fluid through the R H R to the CCW system, and activating knowledge that the same fault could result in release of reactor coolant fluid to the PRT through the RHR relief valve. They postulated correctly that the contaminated water was entering the CCW through a leak in the RHR heat exchanger. They also correctly concluded that the PRT rupture and containment symptoms were due to the relief valve in the RHR venting to the PRT. They then decided to take an action to isolate the RHR heat exchanger. They closed an isolation valve that effectively terminated the leak into the CCW. This was a successful recovery action that went beyond the response plan contained in the available procedures. While the crew terminated the leak from the RHR into the CCW successfully, they never explicitly dealt with the source of this problem-the primary system leak into the R H R system. While they determined correctly that water was coming into the CCW from the RHR, they never addressed the question of how the contaminated water entered the RHR system in the first place. They did nor consider explicitly the possibility of an R H R ISLOCA. They did not attempt to check whether the isolation valves between the reactor coolant system and the RHR were open. The second crew was presented with a diagnostically more challenging event that required them to consider possible flow paths connecting three separate systems. They were not entirely successful in putting the whole picture together. In particular they never explicitly considered a leak into the RHR from the RCS or tried to check on the status of the isolation valves. While the action they took to isolate the RHR heat exchanger temporarily terminated the break into the CCW, it did not deal with the main problem of reactor coolant entering the RHR system. As a result, the incident could have continued to evolve through continuing consequences of the basic fault. 3.3. Cognitive demands of the accident scenarios: data on CES perfirmance

CES was exercised on the same two scenarios used with the instructor crews. At the


Cognitive simulation and lark analysis

.

1177

time when the instructor crews were run on the incidents, the plant parameter values were recorded from the NPP simulator onto a data file. This data file was used as input ito CESa3Thus the behaviour of CES could be examined on exactly the same events that were used with the human crews. In both cases CES diagnosed both the primary coolant leak from the RCS into the RHR and the break in the R H R successfully. It followed the same line of reasoning that the instructor crews were observed to follow. However, it diagnosed the problems much earlier in the evolution of the events than the instructor crews. The process of building the CES knowledge base to the point where it could handle the events, and then analysing the factors that led CES to diagnose the events more readily than the operators were able to, provided insight into the knowledge and processing capabilities required to handle the events successfully. Before CES could be exercised on the ISLOCA events, it was necessary to expand the CES knowledge base to include NPP knowledge relevant to the ISLOCA class of incidents. This included knowledge about the residual heat removal and component cooling water systems, knowledge of the physical interconnections among the primary system (RCS), RHR, and CCW systems, and knowledge of the symptoms and possible consequences of a primary coolant leak from the primary system to the R H R system. CES was then exercised on each of the two cases. Below we provide a summary description of CES performance in each of the two cases followed b y the actual ouput of CES for the case. A comparison of the performance of CES with the human crews, and conclusions drawn from the exercise, are provided following the presentation of the CES outputs. 3.3.1. CES response in case I : In case 1 the first anomaly that CES notes is the RHR pump discharge pressure alarm. Based on this CES hypothesizes that there is an active flow path between the primary system (RCS)and the RHR-that is an ISLOCA.The ISLOCA hypothesis causes it to expect a number of additional plant symptoms that would result from the fault; specifically that primary system (pressurizer) level and pressure should decrease. Consequently when these symptoms appear they are expected and do not generate any further diagnostic activity. This is contrasted with the behaviour of the instructor crews who did not seem to connect the primary system symptoms with the RHR alarm. Based on the ISLOCA hypothesis, CES goes on to project other potential future effects of the increased pressure buildup in the RHR. These include the possibility of the pressurizer relief tank rupturing due to the increased flow from the RHR relief valve, the possibility of an RHR piping break that spills water into the auxiliary building, and the possibility of a break in the heat exchanger between the R H R and the component cooling water system (CCW). These projections enable CES to explain additional abnormal plant symptoms as they arise. The pressurizer relief tank symptoms are explained by the inflow from the RHR. When containment radiation symptoms appear CES briefly considers a number of possible explanations (e.g., a primary break into containment), but given knowledge of the pressurizer relief tank activity, it concludes that containment radiation is due to the pressurizer relief tank rupturing. When symptoms appear in the auxiliary building (i.e., high sump

31n these incidents 232 plant parameters were' used as input to CES.Values for these were input to CES every 10 s.

E. M. Roth et al.


1 178

level in the RHR room and auxiliary building radiation), CES is poised to explain them as the result of an RHR piping break because of the projected influences and potential outcomes it posted earlier. As a result CES was able to diagnose correctly the fault and account for all the symptoms manifest. The output protocol of CES for Case 1 follow^.^ The protocol is generated by giving voice to the individual software agents responsible for monitoring plant parameters, pursuing unexpected findings, and integrating the set of findings into a coherent explanation. The phrases in boldface in the CES output protocols indicate when CES notes a plant parameter behaviour that is anomalous or a change in plant state (e.g., the activation of a safety system such as safety injection; a plant trip). The time indicated is in seconds from the start of the event. The indented text underneath the boldface text indicates the alternatives CES considers in searching for an explanation for an unexpected plant behaviour, or the consequences it expects as a result of a change in plant state (e-g., the consequences expected as a result of a safety injection). The phrases in capital letters (e-g., PRT-TEMP;INCREASINGPROGRESSION) are the labels for NPP parameters, states, processes, faults, and behaviours, that are encoded in the CES knowledge-base. Commentary explaining the CES behaviour is interweaved with the CES output. The commentary appears in italics and is not part of the CES output.

CES output protocol for case I : In this run, the leak in the RHR isolarion valves begins at rime 5 and is annunciated with an alarm signal as thefirst indication of trouble. The program picks up on this alarm and finds confirmarion /or the hypothesis of an aciive transport bet ween the RCS and he RHR. Observations at time 10 concerning RHR-DISCHARGE-PRESSURE-HI-A ...

We note that the state of RHR-DISCHARGE-PRESSURE-HI-A is now YES. There is only one possible explanation for this:

RCSIRHK-A-TRANSPORT-ACT1 VE is strongly suggested by the observed behaviour. This could also explain the following changes:

PRZR-LEVEL DECREASING-PROGRESSION ...strongly suggestive RHR-PUMP-DISCHARGE-PRESSURE-A INCREASING-PROGRESSION ... strongly suggestive The following behaviour is consistent with this hypothesis, but could alternatively be explained by other known influences: PRZR-PRESSU RE DECREASING-PROGRESSION .. .strongly suggestive

'The protocols presented are the outputs of the CES cognitive simulation. In a few cases the variable names have been changed to facilitate comprehension. For the same reason some formatting changes have also been made.

Cogniiive simulaiion and task analysis

1179

At time step 10 (10s into theevent) CES has concluded that rhereis activejlowfrom the RCS inlo rrain A of the RHR (RCS/RHR-A-TRANSPORT-ACTIVE) based on the High RHR discharge pump pressure alarm (RHR-DISCHARGE-PRESSURE-HI-A).


CES then goes on to mention other (otherwise unexplained) planr parameter behaviours that could be explained by the inregace breach (e.g., pressurizer level beha viour). In the case of pressurizer pressure, the rare of decrease is not yet suflcienrly great to warrant search for an explanation. (It can be explained by known influences on pressurizer pressure.) However, CES posts an expectation that pressurizer pressure should start to decrease based on the influences generated by the hypothesized interface breach. This expectation is conJirmed on lime step 20. With rhe conclusion of an active rransport between the RCS and RHR, the program causes activation $additional expectation based on knowledge of likely consequences o f t his event: namely, the prospect that high pressure in rhe RHR can lead to a break somewhere in the R H R piping or in the RHR heat exchanger. For this purpose, it employs a type of knowledge structure called a 'scenario: that lays out likely sequences of events that might unfold as a consequence of this initial fault state. ~cena'riosare one of the knowledge representarion formats in the EAGOL AIsystem that underlies the CESsimulation. Scenarios aresimilar to transition network representaliom, and are used to represent the characteristic sequences of evenrs thar unfold over time as o consequence of an initial disrurbance (e.g., the predictable sequence of plan1 disturbances ihat occur a/ter a plant trip). It is used by CES to project into the future the consequences of distirrbances.

Observations at time 20 concerning PRT-PRESSURE

...

A H IGH-VALU E suggests an abnormal INCREASING-PROGRESSION influence. There is only one possible explanation for this:

RHR-AIPRT-TRANSPORT-ACTIVE is strongly suggested by the observed behaviour. This could also explain the following change:

PRT-TEMPINCREASING-PROGRESSION. ..strongly suggestive However, the following expected change has not been observed: ,

PRT-LEVEL INCREASING-PROGRESSION... strongly indicative, fairly strongly indicated

CES nores an abnormal increase in PRT pressure. In the context of the leak into the RHR and the increased RHR pressure. CES concludes that the increase in PRT pressure is due to venting of the RHR safety valves into the PRT.

CES nores that an increase in PR T level would also be expected but has not been observed. In due course, the missing evidence concerning PR T level will become man vest. Once again, the program uses the current conclusion of RHR-PRT transport lo anticipate fulure occurrences-this time, invoking a scenario that 'envisions' rupture of the PR T rupture disc, which will happen when the PRT overfills.

E. M . Roth et al.

1180

..

Observation at time 20 concerning PRZR-PRESSURE.

There is now unequivocal evidence of an unexplained DECREASING influence on PRZRPRESSURE.

This confirms rhe expecfation of a decrease in pressurizer pressure that had been posted earlier in connection with the RCSIRHR fransport.


We note the occurrence of a REACTOR-TRIP at time 4laS

Consequences of this include: ELEC-GENERATOR-MW DECREASING-PROGRESSION AUCTIONEERED-NUCLEAR-POWER DECREASING-PROGRESSION PRZR-PROGRAM-LEVEL DECREASING-PROGRESS1ON

TREF DECREASING-PROGRESSION A reacror trip is an automatic shut-down ofthe plant. From the perspective of CES this is an

aulomatic control action that produces a series of expected influences on plan1 state. This is a partial listing oJthe sorts ofinfluences on the plant that are invoked with the occurrence ofa reacior trip. As a consequence when changes in plant state arise that are consistent with these influences ihey will not be marked as unexpecred. We note the occurrence of a SI-ACTUATION at time 62.

Consequences of this include: CHARGING-FLOW DECREASING-PROGRESSION SI-FLOW-C INCREASING-PROGRESSION

CTMT-ISOLOA-TRAIN-A YES

this point an emergency core cooling system called SaMy Injection (SI) is activated because of the decrease in primary system pressure. CES notes this automaficsystem control action and posts a series of expected influences on plant srate that are expected as a consequence.

A1

Observations at time 72 concerning PRTYLEVEL.. .

There is now unequivocal evidence of an unexplained INCREASING influence on PRT-

LEVEL. -

-

W u e to constraints of the program generating the time-stamped plant parameter input data to CES, the interval between time-stamped snapshots, is 10 s plus or minus 2 s. Thus in this case the snapshot was taken at 41 s.

Cognitive simularion and task analysis

1181

At time 72, the program notices the anricipated rise in PRT level, and thirry seconds later, discovers a rise in conrainment radiation that signals the rupture of lhe PR T.


On the next rime step rhe program follows a pattern in its commentary that will reoccur rhroughout borh runs. There are two separate commentaries on conrainment radiation for time step 102. The jrst comment is generated by the sojlware agenr /hat has specific responsibiliryfor finding the cause of unexplained radiarion in containmen1 (reJerred to as a 'decisionanalyst 7. 11concludes that 'there are several possible explanariok for rhis . . . ' This is because the knowledge base encornparses a number of faults that can c a u e elevation of radiation Ievels inside containment. The second commenl for time step 102 concerning containment-activiry is made by the sojiware agent who has responsibility /or synthesizing the overall pattern ofjndings into a un$ed explanation (referred to as rhe 'decisionmaster 7, who has re-inrerpreted the situation in light of scenariobased expectations. The decision masrer is able ro narrow down the set o/ possible explanation for the conrainment radiation. Thus rhe comment: 'In this setting.. .'the set of possibilities enurneraled is constrained to those events that meet prior scenario-based expe~rations.~ Observations at time 102 concerning CONTAINMENT-AClWlTY.. . A HIGH-VALUE suggests an abnormal INCREASING-PROGRESSION influences.

There are several possible explanations for this: SECONDARY-LEAK-TO-CONTAINMENT is moderately suggested by the observed behaviour. the following expected changes have not been observed: CTMT-PRESSURE INCREASING-PROGRESSION. . . moderately indicative, fairly weakly indicated.

CTMT-RECIRCULATION-SUMPT-AINCREASING-PROGRESSION . .. moderately indicative, fairly strongly indicated CTMT-ATMOSPHERIC-TEMP INCREASING-PROGRESSION .. .moderately indicative, fairly strongly indicated

PRT-RUPTURE-DISC-RUPTUREDis moderately suggested by the observed behavio~r.~ The following behaviour is consistent with this hypothesis, but could alternatively be explained by other known influences: 6'Decision analyst' and 'decision master' refer to A1 software agents that are pan of the EAGOL problem-solving architecture. Decision analysts are charged with explaining the behaviour of a singIe plant parameter. At any given time there may be several decision analysts working in parallel on different aspects of the problem. The decision master is responsible for coming up with a coherent explanation for all of the unexplained findings. 7PRT-RUPTURE-DISC-RUPTURED is the label encoded in the CES knowledge-base for a PRT rupture.

E. M. Roth et al. PRT-PRESSURE DECREASING-PROGRESSION. .. moderately suggestive However, the following expected changes have not been observed:

CTMT-PRESSURE INCREASING-PROGRESSION... moderately indicative, fairly weakly indicated

CTMT-RECIRCULATION-SUMP-A INCREASING-PROGRESSION ..


moderately indicative, fairly strongly indicated

CTMT-ATMOSPHERIC-TEMP INCREASING-PROGRESSION.. .moderately indicative, fairly strongly indicated PRIMARY-LEAK-TO-CONTAINMENT is moderately suggested by the observed behaviour. the following expected changes have not been observed:

CTMT-PRESSURE INCREASING-PROGRESSION...moderately indicative, fairly weakly indicated CTMT-RECIRCULATION-SUMP-A INCREASING-PROGRESSION . .. moderately indicative, fairly strongly indicated CTMT-ATMOSPHERIC-TEMP INCREASING-PROGRESSION . .. moderately , indicative, fairly strongly indicated INTERFACE-SYSTEM-BREAK-TO-CONTAINMENT is weakly suggested by the observed behaviour. the following expected changes have not been observed:

CTMT-PRESSURE INCREASING-PROGRESSION.. . weakly indicative, fairly weakly indicated

CTMT-RECIRCULATION-SUMP-AINCREASING PROGRESSION.. . weakly indicative, fairly strongly indicated

-

CTMT- ATMOSPHERIC -TEMP INCREASING PROGRESSION . . . weakly indicative, fairly strongly indicated

The cornmentary above was generated by the A / sofrware agent responsiblefor explaining containmenl activity (i.e., a 'decision analystj). It is entertaining a number oajpossiblefaults that could account for the unexpected containment radialion including a primary break in containment (PRIMAR Y-LEAK-TO-CONTAINMENT)and that the pressurizer relief tank hm rupt wed (PR T-RUPTURE-DISC-R UPTURED). In each case it lists out additional symptoms it would expect given zhat fault that have not ye1 been observed. The commentary below was generated by the A1 sofrware agent responsiblefor coming up with a coherent explanation for all the unexplainedfindings (i.e., the 'decision rnasrerj).

Cognitive simulation and task analysis Observations at time 102.. . As for the CONTAINMENT-ACTIV1TY...

In this setting, there is only one possible explanation for this: Onset of PRT-RU PTURE-DISC-RUPTURED


Observations at time 142 concerning CTMT-PRESSURE ...

There is now unequivocal evidence of an unexplained INCREASING influence on CTMTPRESSURE. This commenr simply signals rhe realization of another experration, namely the rise in conrainmcnr pressure following fiom rhe ruptured PRT. In the following, rhe program norices a rise in the sump level in the auxiliary building pump room. As before, there are two comments: the first by a decision analyst that finds only one diagnostic possibility. This is accepted by the decision master as consistent with the scenario expectations. Observations at time 356 concerning RNR-PUMP-RM-SUMP-LEVELA ...

A H IGH-VAW E suggests an abnormal INCREASING-PROGRESSION influence.

There is only one possible explanation for this: R HR-TRAIN-A-PIPING-RUPTURED is moderately suggested by the observed behaviour.

the following expected change has not been observed: AUX-BLDG-RADIATION INCREASING-PROGRESSION. . .moderately indicative, fairly weakly indicated. Observations at time 356.. . As for the

RHR-PUMP-RM-SUMP-LEVEL-A .. .

In this setting, there is only one possible explanation for this: Onsel of RHR-TRAIN-A-PIPING-RUPTURED. Observations at time 1 190 concerning AUX-BLDGRADIATION ...

There is now unequivocal evidence of an unexplained INCREASING influence on AUXBLDG-RADIATION. If we make rhe assumption that AUX-BLDG-RADIATIONand RHR-PUMP-RM-SUMP-

E. M. Roth et al.

1184

LEVEL-A have a common cause, there would be only one possible explanation for this: RHR-TRAIN-A-PIPING-RUPTURED.


Finally, the program notices a rise in the radialion levels in the Aux Building, as expected on the basis of a piping break in the RHR system.

3.3.2. CES response in case 2: As in case 1 CES correctly diagnoses the initial ISLOCA and correctly interprets the additional plant disturbances as repercussions of this initial fault. CES handles case 2 much the way it handled case 1. As in case'l, the first anomaly CES notices is the RHR pump discharge pressure alarm. This evokes the ISLOCA hypothesis which in turn generates a number of expectations and projections of possible future outcomes. As in case 1, the disturbances in the primary system (i.e., pressurizer level and pressure) are explained as the result of the ISLOCA and are not pursued further (because they were expected). Similarly the pressurizer relief tank symptoms are explained by the increased flow from the RHR relief valve. The increased radiation in containment is explained in turn by the rupturing of the pressurizer relief tank. As in case 1, CES envisions a number of potential future outcomes of the RHR pressure buildup, including a break in the RHR piping, and a break in the RHR heat exchanger that interfaces with the component cooling water system (CCW). CES notices disturbances in the CCW system (i-e., radiation and an increase in CCW surge tank level). These are both explained by postulating a break in the RHR heat exchanger due to the increased R H R pressure buildup. Note that while CES projected the same set of possible future disturbances in cases 1 and 2, it comes to a different diagnosis in case 2 because of differences in the pattern of symptoms that arises. Thus, while the envisioning mechanism in CES make it poised to explain potential disturbances that might arise in the future, it does not predetermine its diagnoses. When CES notices radiation in the auxiliary building it entertains two hypotheses as plausible explanations including ( 1 ) that the CCW surge tank has overflowed releasing radioactive coolant into the auxiliary building; and (2) that there is a second break in the RHR piping. When it gets definitive evidence that the CCW surge tank has overflowed, it correctly concludes that it is this that accounts for the auxiliary building radiation symptoms.

CES output protocol for case 2: The course of CESs analysis closely follows that of case I up to tjme step 49.

Observations at time 9 concerning RHR-DISCHARGE-PRESSURE-HI-A

...

W; note that the state of RHR-DISCHARGE-PRESSURE-HI-A is now YES.

There is only one possible explanation for this:

RCSIRHR-A-TRANSPORT-ACTIVEis strongly suggested by the observed behaviour. This could also explain the following change:


1185

RHR-PUMP-DISCHARGE-PRESSURE-A INCREASING-PROGRESSION . . . strongly suggestive The following behaviours are consistent with this hypothesis, but could alternatively be explained by other known influences: PRZR-LEVEL DECREASING-PROGRESSION.. . strongly suggestive


PRZR-PRESSURE DECREASING-PROGRESSION. ..strongly suggestive

With this evidence, the conclusion of RCSIRHR-A-TRANSPORT can be made with some confidence. Observations at time 19 concerning PRZR-LEVEL...

There is now unequivocal evidence of an unexplained DECREASING influence on PRZRLEVEL. Observations at time 19 concerning PRZR-PRESSURE...

There is now unequivocal evidence of a n unexplained DECREASING influence on PRZRPRESSURE. Obsewations at time 29 concerning PRT-PRESSURE.., A HIGH-VALUE suggests an abnormal INCREASING-PROGRESSION influence.

There is only one possible explanation for this.

RHR-A/PRT-TRANSPORT-ACTWE is strongly suggested by the observed behaviour. This could also explain the following change:

PRT-TEMP INCREASING-PROGRESSION . ..strongly suggestive However, the following expected change has not been observed:

PRT-LEVEL INCREASING-PROGRESSION. . .strongly strongly indicated

indicative,

fairly

With this evidence, the conclusion of RHR-AIPRT-TRANSPORT can be made with some confidence.

We note the occurrence of a REAnOR-TRIP at time 39..

.

Consequences of this include: ELEC-GENERATOR-MW DECREASING-PROGRESSION

E. M. Roth et al.

1186

AUCTIONEERED-NUCLEAR-POWER DECREASING-PROGRESSION PRZR-PROGRAM-LEVEL DECREASINGPROGRESSION TREF DECREASING-PROGRESSION Observations at time 49 concerning CCW-RAD-MONITOR-TRAIN-A ...


A HIGH-VALUE suggests an abnormal INCREASING-PROGRESSION influence.

There is only one possible explanation for this:

RHR-TRAIN-A-HX-BREAK-TO-CCW is moderately suggested by the observed behaviour. the following expected change has not been observed:

CCW-SURGE-TANK-LEVELA INCREASING-PROGRESSION ...moderately indicative, fairly strongly indicated

When CES norices radiation in CCW it concludes that there must be a break in rho RHR hear exchanger with the CC W system. This conclusion is drawn because a porential break in the RHR hear exchanger ro the CCW sysrem war onricipated on the baris of the scenario that war acrivared when the leak fiom the RCS into the RHR was diagnosed. Note that the possibility of a break in the RHR/CC W hear exchanger was also projected as a possibility in care 1 but was never observed. An additional scenario is invoked on the basis of the decision that there is a leak into the CCW: namely, rhat in due course, the level in the CCW surge tank isgoing ro reach 100% and begin to over-ow onto rhejloor offheaux. building. The comment above was made by a decision analyst. The comment below was made by the decision master.

Observations at time 49..

.

As for the CCW-RAD-MONITOR-TRAIN-A .. .

In this setting, there is only one possible explanation for this: Onset of RHR-TRAIN-A-HX-BREAK-TGCCW.

We note the occurrence of a SI-ACTUATION at time 59..

Consequences of this include:

CHARGING-FLOW DECREASING-PROGRESSION SI-FLOW-C INCREASING-PROGRESSION

.


CTMT-ISOL-OA-TRAIN-A YES

The following provides additional reinforcement for the decision made earlier.

...

Observations at time 129 concerning CCW-SURGE-TANK-LEVEL-A


There is now unequivocal evidence of an unexplained INCREASING influence on CCWSURGE-TANK-LEVEL-A. If we make the assumption that CCW-SURGE-TANK-LEVEL-A and CCW-RADMONITOR-TRAIN-A have a common cause, there would be only one possible explanation for this: RHR-TRAIN-A-HX-BREAK-TO-CC W.

The following two episodes involving PR T level and containment activity are identical to those found in the previous run.

Observations at time 149 concerning PRT-LEVEL

...

There is now unequivocal evidence of an unexplained INCREASING influence on PRT-

LEVEL. Observations at time 200 concerning CONTAINMENT-ACTIVITY...

A HIGH-VALUE suggests an abnormal INCREASING-PROGRESSION influence. There are several possible explanations for this:

SECONDARY-LEAK-TO-CONTAINMENT is moderately suggested by the observed behaviour. the following expected changes have not been observed:

CTMT-PRESSURE INCREASING-PROGRESSION...moderately indicative, fairly weakly indicated CTMT-RECIRCULATION-SUMP-A INCREASING-PROGRESSION... moderately indicative, fairly strongly indicated

CTMT-ATMOSPHERIC-TEMP INCREASING-PROGRESSION. . . moderately indicative, fairly strongly indicated PRT-RUPTURE-DISC-RUPTURED is moderately suggested by the observed behaviour. The following behaviour is consistent with this hypothesis, but could alternatively be explained by other known influences:

PRT-PRESSURE DECREASING-PROGRESSION... moderately suggestive

E. M.Ruth et al. However, the following expected changes have not been observed:

CTMT-PRESSURE INCREASING-PROGRESSION .. .moderately indicative, fairly weakly indicated

CTMT-REClRCULATION-SUMP-AINCREASING-PROGRESSION . .. moderately indicative, fairly strongly indicated

CTMT-ATMOSPHERIC-TEMP INCREASING-PROGRESSION .. . moderately Downloaded by [University of Cincinnati Libraries] at 15:22 04 January 2015

indicative, fairly strongly indicated

PRIMARY-LEAK-TO-CONTAINMENTis moderately suggested by the observed behaviour. the following expected changes have not been observed:

CTMT-PRESSURE INCREASING-PROGRESS1O N . ..moderately indicative, fairly weakly indicated

CTMT-RECIRCULATION-SUMP-AINCREASING-PROGESSION ... moderately indicative, fairly strongly indicated CTMT-ATMOSPHERIC-TEMP INCREASING-PROGRESSION ... moderately indicative, fairly strongly indicated

INTERFACE-SYSTEM-BREAK-TO-CONTAINMENTis weakly suggested by the observed behaviour. the following expected changes have not been observed:

CI'MT-PRESSURE INCREASING-PROGRESSION .. .weakly indicative, fairly weakly indicated

CTMT-RECIRCULATION-SUMP-AINCREASING-PROGRESSION ...weakly indicative, fairly strongly indicated

CTMT-ATMOSPHERIC-TEMP

INCREASING-PROGRESSION...weakly

indicative, fairly strongly indicated

Observations at time 200.. As for the

.

CONTAINMENT-ACTIVITY ...

In this setting, there is only one possible explanation for this: Onset of PRT-RUPTURE-DISC-RUPTURED.

In the following, the program notices the increasing radiation in the auxiliary building. The several possibiliries mentioned include overflow of the surge rank, and also break of the piping in train-a or train-b of the RHR sysrem.

Cognitive simulation and task analysis Observations at time 372 concerning AWL-BLDG-RADIATION ... A HIGH-VALUE suggests an abnormal INCREASING-PROGRESSION influence.

There are several possible explanations for this:

CCW-SURGE-TANK-A-OVERFLOWING is strongly suggested by the observed behaviour.


the following expected change has not been observed:

RHR-PUMP-RM-SUMP-LEVEL-AINCREASING-PROGRESSION .. . strongly indicative, fairly strongly indicated

R HR-TRAIN-B-PI PING-RUPTURED is moderately suggested by the observed behaviour. the following expected change has not been observed:

RHR-PUMP-RM-SUMP-LEVEL-B INCREASING-PROGRESSION ... moderately indicative, fairly strongly indicated

RHR-TRAIN-A-PIPING-RUPTURED is moderately suggested by the observed behaviour. the following expected change has not been observed:

RHR-PUMP-RM-SUMP-LEVELA INCREASING-PROGRESSION . . . moderately indicative, fairly strongly indicated

The above commentary was made by a decision analyst. In rhe following comment by the program's 'decision masrcr: two of these possibilifies continue to be considered even afrer applying the constrainrs imposed by the scenario strucrures. This is because f he prospect of a break in the train-a piping is still a possible consequence of the overpressurization of the RHR frain-a. Damage to train-b is nor consideredfurther, however, because there is no evidence of txcessive pressure therc. This demonstrates how the expecf ations genera fed based on scenarios enable the set of possible explana ions to be reduced, but still allow for multiple a11m a tive explanaf ions. Observations at time 372.. . As for the AUX-BLDG-RADATION . ..

In this setting, there are only two possible explanations for this: Onset of RH R-TRAIN-A-PIPING-RUPTURED. Progression to CCW-SURGE-TANK-A-OVERFLOWING.

Finally rhe program concludes that the surge rank is the one that is culpable in this care because the surge tank level had already reached the 100% stage, making overflow inevitable.

.

1190

E. M . Roth et al.


With this evidence, the conclusion of confidence.

CCW-SURGE-TANK-Acan be made with

some

3.4. Cognitive fask analysis of the accident scenario In both cases CES successfulIy diagnosed both the primary coolant leak from the RCS into the RHR and the break in the RHR. It followed much the same line of reasoning as that of the instructor crews. However, it diagnosed the problems much earlier in the evolution of the events than the instructor crews did. The process of building the CES knowledge base to the point where it could handle the events, and analysing the factors that led CES to diagnose the events more readily than the operators were able to, provided insight into the knowledge and processing capabilities required to handle the events successfully. Juxtaposition of the CES performance with the crew performance brought out in bold relief the attentional and information processing bottlenecks that prevented the instructor crews from synthesizing the pattern of symptoms into a correct diagnosis as early in the event as

CES. To a large extent the success of CES in diagnosing these events depended on its ability to focus on the symptoms specific to the RHR early in the event. This allowed it to hypothesize an ISLOCA as soon as the first RHR symptoms appeared. It also allowed it to envision potential future disturbances that could result from the RHR pressure build-up. These projections enabled CES to form expectations about future symptoms that could arise in the PRT (due to the RHR relief valve opening), in containment (due to the PRT rupturing) and in the auxiliary building (due to the R H R rupturing). As a result, when these symptoms became manifest CES was able to absorb them into an already existing hypothesis-the ISLOCA. The human operator crews, while clearly possessing the same knowledge of system interconnection as encoded in CES, and the same qualitative reasoning ability to envision potential future repercussions of the initial ISLOCA, failed to activate this knowledge early on. In contrast to the behaviour of CES, when the first RHR symptoms appeared, the instructor crews neither searched for the source of the disturbance, nor pursued potential consequences of it. On the surface, the two ISLOCA incidents would seem to be straightforward to diagnose and manage-there are relevant alarms and parameter displays in the control room, and there are procedures available for responding to this class of events. There is an alarm that indicates abnormal pressure in the RHR system. This is the first alarm that goes off at the start of the incident (approximately 5 s into the event). In addition there are meters in the control room that allow operators to track RHR status (e.g., RHR discharge pump pressure). In spite of the availability of what would appear to be a clear leading indicator of a problem in the RHR and the availability of procedural guidance, the instructor crews failed to diagnose the leak from the primary system into the RHR system until late in the event. In the first case the crew did not identify the ISLOCA until 16 min in the event. In case 2, while the instructor crew correctly diagnosed and responded to the repercussions of the ISLOCA, they never explicitly pursued the source of the coolant water in the RHR-that is the ISLOCA. In contrast CES diagnosed the ISLOCA as soon as pressure symptoms in the RHR began to appear. It then projected potential future consequences of the ISLOCA and was quick to observe and explain plant symptoms that later appeared that were consistent with its expectations.



1191

Contrasting the performance of CES with the performance of the instructor crews reveals a variety of factors that limited the ability of the instructor crews to follow the same straight path as CES. It reveals several areas where human performance is vulnerable and points to a variety of ways to improve the person-machine system. The cognitive demands of the incident will be highlighted by tracing the dynamic flow of events. The first symptom is a high pressure alarm in the R H R system which the instructor crews as well as CES noted in both cases. While this alarm suggests a problem in the RHR system, it is rapidly followed by symptoms that suggest a primary system break inside containment. The pattern of findings-primary system level decreasing, primary system pressure decreasing, indication of radiation within containment-are the classic signature of a primary system break into containment. Note that this conclusion while consistent with a salient subset of the anomalies does not account for the R H R symptoms. At the first evidence of primary system disturbance the attentional resources of the instructor crews appear to have been diverted away from the RHR. From that point on, until more severe symptoms of the RHR break occurred, the instructor crews appeared to 'forget' about the RHR problem. They neither tried to pursue the source of the R H R pressure build-up, nor tried to anticipate potential consequences of the RHR disturbance. Instead their main attentional focus was on trying to diagnose and attempt to respond to the primary system symptoms. In contrast CES doggedly pursued the RHR symptoms. The success of CES resulted mainly from three factors: (1) its ability to diagnose the R H R ISLOCA early on-the instructor crews attended to RHR alarm but then didn't follow u p once symptoms in containment arose; (2) its ability to project potential consequences into the future that allowed it to absorb and connect seemingly disparate findings; and (3) its ability to notice anomalies as soon as they arose. The failure of the instructor crews to pursue the R H R symptoms becomes more readily understandable when one considers the operational context. Because of the importance of maintaining primary system integrity, all the attentional resources of the crew appear to have been absorbed by that problem. Furthermore, during NPP emergencies crew monitoring and response activity are supposed to be guided by the emergency procedures. In this case the procedures served to further focus the crew on the primary system symptoms at the expense of pursuing the R H R symptoms. Neither crew of instructors, who clearly possessed enough knowledge of the interconnections to entertain the possibility of an ISLOCA to account for the RHR symptoms, and who were very knowledgeable in the specific procedures for this type of incident, called to mind the possibility at this stage. CES runs showed us that it could be set up to make the diagnosis at this point, but only if it could generate or call to mind the hypothesis of an ISLOCA at this point. As the incident progresses additional symptoms appear that point to disturbances in other plant systems. In the first case, when the RHR piping breaks, the reactor coolant water spills into the auxiliary building resulting in an RHR Room Sump Level High alarm and auxiliary building radiation alarms. In the second case symptoms appear in a third system-component cooling water (CCW). However, in the case of the instructor crews, these alarms occur against the background of an initial plausible diagnosis that accounts for the anomalies in the major plant system. The additional anomalies in the RHR system re-start diagnostic search on the question of what influences could account for the R H R or CCW findings.


1192

E. M. Roth et al.

At this point the crews as well as CES have another cognitive task-how to put together two or three subsets of findings. This is an abductive reasoning problem-is there one underlying explanation for all of the subsets of findings o r are there several factors at work? This task arises and is difficult because, while the source of the problems is in the RHR system, symptoms are manifested in multiple systems that are normally not connected. As a result the operator needs to integrate seemingly independent problems actively to assess the situation correctly. This requires knowledge of the physical connections among the systems and active diagnostic effort that goes beyond the guidance provided in the procedures. In particular, diagnosis requires activating knowledge about the points of interface among the primary system (RCS), the'^^^ system, and the CCW system in order to put the various subsets of findings together. Both CES and the crews had to go through this reasoning task. CES, at its current stage of development, is able to carry out the necessary information processing and knowledge activation very quickly. This is because of the 'envisioning' mechanism that enabled CES to project possible future disturbances that could result from the initial RHR pressure buildup. The crews, on the othei hand, at first entertained the possibility of separate independent faults. As the severity of the symptoms increased the instructor crews had to pull back actively and attempt to resynthesize the pattern of symptoms. They took advantage of physical schematics of the relevant systems as an external knowledge base and a source of retrieval cues to activate knowledge relevant to the problem at hand and as a visualization aid to support reasoning through the possible flow paths among the systems. There are a variety of ways that this diagnostic cognitive task can break down and eventually we plan to be able to use CES as a tool to explore the consequences of these. The dynamics of the event (i-e.,the order of appearance and tempo of symptoms and their effect on the operator's attentional focus) are another important factor that makes it difficult for operators to put the whole picture together. Operator attention is divided and diverted away from the RHR system as symptoms begin to appear in other systems. When symptoms appear inside containment, the procedures direct the operator to procedures for responding to primary system break inside containment, thus further diverting the operator's attention away from the RHR problem. In the particular procedures employed in these exercises, there was a specific procedure for diagnosing and responding to ISLOCA events. However, once the operator is directed to the primary break inside containment procedure, there is no provision within the structure of the procedure to redirect himlher to the ISLOCA procedure. In order to get to the correct procedure, the operators must diagnose the situation alone, recognize that there is a relevant procedure for this condition, and take the initiative to switch to that procedure. Static analyses can easily underestimate the role of dynamic factors in human behaviour. A cognitive simulation forces the analyst to think about the temporal evolution of the incident and the temporal aspects of a crew's response. In spite of the fact that the two crews were made up of training instructors, who are proficient in the plant and the use of the procedures, and who are highly familiar with the range of NPP accidents that are simulated on the plant simulator, the ISLOCAs were not diagnosed until fairly late into the event-substantially after the pressure build up in the RHR led to a break. The crew in case 1 did not attempt to check on the status of the R H R isolation valves until approximately 16 min into the


event. The crew in the second_case did not check on the status of the valves during the time period observed. The diagnostic process can affect the selection and execution of appropriate recovery actions. Even if operators correctly diagnose the incident, a delay in diagnosis can mean that what was at one time an isolatable leak can no longer be isolated directly, or, in the case of an unisolatable leak, action to conserve RWST coolant water and to perform a plant cooldown may not be taken soon enough. As it happens, in the specific scenarios we ran, the leaks into the RHR were not isolatable. The results suggest that, even in cases where the leaks into the RHR are isolatable, realization of a need to check on the status of the isolation valves may not arise sufficiently soon to prevent a break in the RHR. Note that the human performance data used in this analysis are limited on a variety of dimensions. First the crews were made up of instructors rather than operators. Perhaps, more importantly, the crews were made up of only two people, which meant that they had fewer attentional resources. Normally an operator crew would comprise three to five people. Similarly, there are interpretative limits to the behaviour of CES on these incidents. The current working version of CES does not capture the cognitive demands associated with attentional control well, does not include all of the tasks and the associated workload operators must perf'orm, and is too good and too efficient at knowledge retrieval. Together the two sets of data provide a converging picture of the cognitive demands and vulnerabilities for human performance in this class of incidents. One of the clear strengths of CES is that it provides a tool for assessing the extent to which the environment supports the diagnostic task confronted by the problemsolver. The process of building CES to handle the accident situation successfully provides insight into the NPP knowledge an operator must have, and the evidence she or he must attend to and integrate, in order to diagnose and handle the incident of interest correctly. In order for CES to diagnose the fault and subsequent break correctly, it was necessary to encode in CES information about the physical interconnections among various plant systems, and about the disturbances that result from breaks between these systems (i-e., the equivalent of a mental model of the system and its interconnections). It was also necessary to activate reasoning mechanisms that allowed CES to project into the future disturbances that result form the break. This was critical to enable CES to anticipate future disturbances and integrate seemingly independent symptoms into a coherent picture (cf., Klein 1990, for evidence of the importance of mental simulations for future projections in human diagnosis). An implication of the CES exercise is that in order for people to diagnose and handle the ISLOCA correctly, they would need to access the same type of knowledge and utilize the same type of logic. Successful diagnosis requires accurate knowledge of the physical interconnections among the systems and potential flow paths among normally separate systems in order to account for the full set of symptoms observed. In Rasmussen's terminology, accurate diagnosis and response in this incident requires knowledge-based behaviour (Rasmussen 1986). Observation of the instructor crews confirms this. The crews used the same knowlege and reasoning to solve the problem, although it took them much longer to access the relevant knowledge and form the necessary connections. Activating and using this knowledge in a high tempo multi-task situation is critical and is affected by available external displays, aids, and knowledge bases. In '


1193

E. M. Roth et al.


1194

the case of the instructor crews activation of the ISLOCA hypothesis was substantially delayed. As a result the primary system symptoms were initially interpreted independently of the R H R symptoms. In contrast CES activated the ISLOCA hypothesis at the first sign of RHR symptoms. As a result it was able to explain the primary system symptoms as a consequence of the ISLOCA. For operators the ISLOCA to the R H R is not a hypothesis that would be readily called to mind in this context. One reason is that the RHR system is not active during normal operation, and the isolation valves between the RCS and the RHR are supposed to be closed at all times. The operator does get a high R H R discharge pressure alarm, suggesting a possible problem in the RHR, but the significance of this alarm is difficult to interpret given that the R H R system is supposed to be inactive. The alarm response book that is intended to provide guidance on alarm interpretation only discusses the meaning of this alarm for cases where the RHR is in operation. In addition the high RHR discharge pressure is a single alarm that is rapidly followed by alarms relating to systems inside containment that draw attention away from the RHR system. A second major reason the ISLOCA hypothesis would not be one of the first things called to mind is that operators are not sensitized to the possibility of this type of malfunction. Operators d o not rcceive as much training on this type of event as on the standard accident scenarios. In fact many training simulators do not model this type of event well, if at all. In summary, juxtaposition of the human performance data and the CES performance suggest that successful diagnosis requires: knowledge of the physical interconnections among the relevant systems; active search for a common root cause to account for the seemingly independent symptoms; call to mind the possibility of an ISLOCA, i. e., a primary system leak to the RHR; projecting potential future disturbances that may result from the initial RHR problem.

CES provides a way to assess the cognitive demands of the environment. A cognitive simulation can reveal:

A cognitive simulation such as

0 0 0

what knowledge an operator must possess to handle the event; what evidence must be observed and integrated; an absolute lower limit on how quickly a disturbance can be diagnosed (i-e., the time at which all the evidence required to diagnose the disturbance becomes available).

While CES was successful in revealing the knowledge and reasoning required to handle this class of incidents, it did not model the difficulty people had in accessing the relevant knowledge and integrating the evidence. CES was able to detect disturbances sooner, and follow implications of disturbances more thoroughly than the human crew could. This is because CES currently does not model the attention and processing resource limits of people. In its current state CES plays the role of a model of the cognitive environment that reveals the cognitive functions that must be performed to successfully handle a


1195


domain task, rather than a performance model that describes the detailed processes by which these functions are a c h i e ~ e d . ~ The current implementation of CES represents a working hypothesis of the set of competencies required to adequately respond to the demands of the problem-solving environment. When contrasted with empirical results of human crew performance it provides a way to disentangle performance limits that are due to inherent demands of the situation from limitations due to particular strategies or processing constraints. In turn the results of the analyses provide the basis for further development of the simulation. 4. Cognitive simulations and cognitive task analysis A cognitive task analysis depends on two mutually-reinforcing activities: an analysis of the cognitive demands imposed by the world that any intelligent agent would have to deal with (a model of the cognitive environment); and an empirical investigation of how practitioners, both experts and less skilled individuals, respond to the task demands (performance models). Taken in combination the analyses reveal the sources of task difficulty and enable identification of options to produce a better match between the cognitive demands of the task and the available resources (Roth and Woods 1989). They provide the basis for specifying what new information, representations, and advice should be provided. Cognitive simulations can aid in cognitive task analyses by revealing the knowledge and reasoning required to successfully respond to the task demands. They provide a tool for understanding the extent to which the environment supports the diagnostic task confronted by the problem solver. In the current example, CES provided an objective means for establishing some of the cognitive activities that any intelligent agent would have to perform in order to successfully handle the emergency event. As such it provided a tool for validating and extending the cognitive task analysis for this set of breaks that was performed based on discussions with instructors, review of procedures, and observations of crews in simulated emergencies. The value of a cognitive simulation is in helping to see the demands imposed by the problem-situation independent of the strategies that people bring to bear. A key element of the analysis is to compare the performance of the cognitive simulation with empirical data on human crew performance. It is the juxtaposition of data on human performance (even if limited in scope) with the performance of the cognitive simulation that allows the inherent demands of the cognitive environment to stand out. While there are clear attractions to the cognitive simulation strategy. there are also clear limitations that are important to keep in mind especially because they modulate how one should use the technique. First, given the breadth of human cognitive activities that come into play in complex task domains (Woods and Roth 1986), the evolution of knowledge in the field of cognitive science, and pragmatic

8Given that CES was able to synthesize the pattern of symptoms correctly in these events more early than the instructor crews, a question that might be asked is whether CES could serve as a 'computer advisor' or on-line decision-aid for operators. While the focus of this project is on using CES to reveal the cognitive demands o f the situation, we are also exploring the possibility of using the underlying reasoning engine as the basis for decision aids.


1196

E. M. Roth et al.

factors in large software development projects, it is very difficult t o see a cognitive simulation as a finished system. Rather, cognitive simulations are always in a state of evolution. Second, it must be kept in mind that cognitive simulations are instantiations of concepts about human cognitive activities, not the concepts themselves. As a result, it is better to see cognitive simulation as a tool for modelling rather than as a strong model in itself. Rather than ask validation questions-is this the correct model?-we contend that cognitive simulation is one part of the process of building and using a model of the cognitive demands and practitioner information processing activities. The goals of the simulation enterprise are to learn from juxtaposing the behaviour of the simulation and the behaviour of human practitioners. By examining the similarities and contrasts between simulation behaviour and crew behaviour one can extract information about error, expertise, and potential improvements to the person-machine system. At the same time the results of the analyses can serve to guide the continuing evolution of the cognitive simulation to better capture both cognitive task demands and human strategies for meeting those demands. In other words, cognitive simulation can be part of the process of theory-based empirical investigations and data-based theory building that is the classic model for scientific growth. Acknowledgements This research is being supported by the Office of Nuclear Regulatory Research, US Nuclear Regulatory Commission. The authors would like to thank Dr Thomas G. Ryan, who has served as NRC Project Manager, and Dr Paul Lewis, who is the current NRC Project Manager, for their guidance and support on this project.

References 1990, Robust fault diagnosis of physical systems in operation, Doctoral Dissertation, State University of New Jersey, Rutgers. AMENDOU, A., BERSIN,U., CACCIABUE, P. and MANCINI,G. 1987, Modelling operators in accident conditions: advances and perspectives on a cognitive model, Infernational Journal oJMan-Machine Srudies, 27, 599-6 1 2. COOK,R. I., WOODS,D. D. and MCDONALD,J. S. 199 1, Human Per/ormance in Anesthesia. A Corpus of Cares, Cognitive Systems Engineering Laboratory Report, Ohio State University, Columbus, Ohio. Coo=, M. N. and MCDONU, J. E. 1987, The application of psychological scaling techniques to knowledge elicitation, Inrernalional Journal of Man-Machine Studies, 26, 533-550. CORKER, K., DAVIS,L., PAPAZUN,8. and PEW,R. 1986, Development of an advanced task analysis methodology and demonstration for army aircrew/aircraft integration, Technical Report BBN 6 124, Bolt Beranek and Newman, Cambridge, MA. Ehsrur, J. R. 1987, Engineering human factors into the Westinghouse advanced control room, Nuclear Engineering lnlernarional, 32, 35-38. ELKIND, J., CARD, S., HOCHBERG, J. and HUEY, B. (eds) 1990, Human Perfarmance Models for Cornpurer Aided Engineering (Academic Press, New Y ork). Fomus, K. 1988, Qualitative physics, past, present and future, in Exploring Arlijicial Intelligence (Morgan Kaufmann, San Mateo, CA), 239-296. GADD,C. S. and POPLE,H. E. 1990, Evidence from internal medicine teaching rounds of the multiple roles of diagnosis in the transmission and testing of medical expertise, in N. Frederiksen, R. Glaser, A. Lesgold and M. G. Shafto (eds) Diagnostic Monitoring o/Skill and Knowledge Acquisirion (Erlbaum, Hillsdale, NJ), 89- 1 12. HOWD, J . H ., HOLYOAK, IL J ., N I S B R. ~ , E. and THAGARD, P. R. 1986, induction (MITPress, Cambridge, MA). HOUAGEL, E. and W o o ~ s ,D.D. 1983, Cognitive systems engineering: new wine in new bottles, Internarional Journal of Man-Machine Siudies, 18, 583-600. A e m , K. H.


Cognirive simulation and task analysis

1197

H ~ I N SE., 1991, HOW a cockpit remembers its speed, Technical Report, Distributed Cognition Laboratory, University of California at San Diego. JOHNSON, P. E., MOW,J. B. and THOMPSON, W. B. 1988, Garden path errors in diagnostic reasoning, in L. Bolec and M. J. Coombs (eds), Expert System Applications (SpringerVerlag, New York), 395-428. KLEIN, G. A. 3989, Recognition-primed decisions, in W. B. Rouse (ed.) Advances in Man-Machine Research, Vol. 5 (JAl Press, Greenwich, CT). WIN, G. A. 1990, Mental simulation and decision-making, Proceedings of the 5rh Mid-Central Ergonom ics/Human Factors ConJerence, Dayton, OH). KLEIN, G. A., CALDEIRWOOD, R. and MACGREGOR, D. 1989, Critical decision method for eliciting knowledge, IEEE Systems. Man, and Cybernetics, SMC-19,462-472. LAIRD,J. E., NEWELL, A. and R o s m m ~ P. , S. 1987, SOAR: an architecture for general intelligence, Artijicial Inrelligence, 33, 1-64. L m w , A,, LUOIE,S., LOGAN,D. and EGGAN,G. 1990, Applying cognitive task analysis and research methods to assessment, in N. Frederiksen, R. Glaser, A. Lesgold and M. G. Shafto (eds) Diagnostic Moniroring of Skill and Knowledge Acquisit ion (Erlbaum, Hillsdale, NJ), 325-350. LIND, M. 199 I, Development of interfaces for supervisory control, in G. Weir and J. Alty (eds), Human-Computer Interaction and Complex Systems (Academic Press, London). MEANS, B., MUMAW, R., ROTH, C., SCHUGER,M., MCWIUIAMS,E., GAGNE,E., RICE,V., ROSENTHAL, D. and HEON,S. 1988, ATC Training Analysis Study: Design of the NextGeneration Air Traflc Controller Training System, Hill Technical Report, HumRRO International, Inc., Alexandria, VA. M r r - a u t , C. M. and h l ~ wR., A. 1986, A discrete control model of operator function: a methodology for information display design, IEEE Sysrems, Man, and Cybernerics, SMC-16,343-357. MITCHELL,C. and SNSI, D. 1987, Use of model-based qualitative icons and adaptive windows in workstations for supervisory control systems, IEEE Transactions on Systems, Man, and Cybernetics, SMC-17,573-593. NEWELL, A. 1990, Unijed Theories of Cognirion (Harvard University Press, Cambridge, MA). NEWEU,A. and SIMON,H. A. 1972, Human Problem-Solving (Prentice-Hall, Englewood Cliffs, NJ). PEW,R., M~LLER, D. C. and FEEHRER, C. E. 198 1, Evaluarion of Proposed Conrrol Room Improvemenrs Through Analysis of Critical Operator Decisions, N P - 1 982, Electric Power Research Institute, Palo Alto, CA. RASMUSSEN,J. 1986, Information Processing and Human-Machine Interaction, An Approach to Cognitive Engineering (North-Holland, New York). REDDING, R. E. 1989, Perspectives on cognitive task analysis, the state oft he a n , Proceedings of the Human Factors Society 33rd Annual Meeting, 1 348- 1 3 5 2. ROTH, E. M., POPLE,H. E. and WOODS,D. D., Extending the Modelling Capabilities of /he Cognitive Environment Sirnulorion, Modeling an ISLOCA Scenario, NUREGICR-5 593, in preparation. Rom, E. M. and WOODS,D. D. 1988, Aiding human performance, I. Cognitive analysis, Le Travail Humain, 51, 39-64. Rom, E. M. and Woo~s,D. D. 1989, Cognitive task analysis: an approach to knowledge acquisition for intelligent system design, in G. Gujda and C. Tasso (eds) Topics in Expert System Design (North-Holland, New York), 233-264. SARTER,N. and WOODS,D. D. 1991, Pilo1 Inreracrion wirh Cockpit Automation, Cognitive Systems Engineering Laboratory Report, The Ohio State University, Columbus, OH. SIMON,H. A. 1 969, The Sciences of the Artificial (MITPress, Cambridge MA). THOMPSON, W. B., JOHNSON, P. E. and MOEN, J. B. 1983, Recognition-based diagnostic reasoning, in Proceedings of rhe Eight International Joinr Conference on Artijcial Intelligence (Karlsruhe, West Germany), 236-238. WOODS,D. D. 1988, Coping with complexity: the psychology of human behaviour in complex systems, in L. P. Goodstein, H. B. Andersen and S. E. Olsen (eds) Tarks,Errors, and Mental Models (Taylor & Francis, London), 128- 1 48. WOODS,D. D. 1991, Representation aiding: a ten year retrospective, in Proceedings of IEEE International Conference on Sysrems, Man, and Cybernetics ( I E E E , New York), 1173-1 176.


1198


WOODS,D. D. in press, Process Tracing Methods for the Study of Cognition Outside of the Experimental Psychology Laboratory, in G. A. Klein, J. Orasanu and R. Caldewood (eds) Decision-Making in Action: Mod~lsand Merhods (Ablex, New Jersey). WOODS, D. D. and HOUNAGEL,E. 1987, Mapping cognitive demands in complex problemsolving worlds, Inrernational Journal of Man-Machine Studies, 26, 25 7-2 75. WOODS, D. D. and Rom, E. M. 1986, Models of cognitive behavior in nuclear power plant personnel, Technical Report NUREG-CR-4532, US Nuclear Regulatory Commission, Washington DC. WOODS,D. D. and Rom, E. M. 1988a, Cognitive systems engineering, in M. Helander (ed.) Handbook of Human-Cornpurer Interacrion (North-Holland, New York), 3-43. WOODS,D. D. and ROTH,E. M. l988b, Aiding human performance, 11. From cognitive analysis to support systems, Le Travail Humain, 51, 1 39- 1 7 1. WOODS, D. D., O'BRIEN,J. F. and HANES,t.F. 1987, Human factors challenges in process control: the case of nuclear power plants, in G. Salvendy (ed.) Handbook of Human Factors (John Wiley, New York), 1 724- 1770. WOODS,D. D., ROTH, E. M. and POPU, H. 1987, Cognitive environment simulation: an artificial intelligence system for human performance assessment, NUREGICR-4862, US Nuclear Regulatory Commission, Washington, DC. WOODS,D. D., Rom, E. M. and POPLE,H. 1988, Modeling human intention formation for human reliability assessment, Reliability Engineering and Systems Safety, 22, 169-200. WOODS,D. D., POPLE,H. E. and ROTH,E. M. 1990, The cognitive environment simulation as a tool for modeling human performance and reliability, NUREGKR-52 13, US Nuclear Regulatory Commission, Washington, DC. -

Validation of a Cognitive Task Simulation and Rehearsal Tool for Open Carpal Tunnel Release.

A multi-site cognitive task analysis for biomedical query mediation.

Toward a cognitive task analysis for biomedical query mediation.

TMS as a Tool for Examining Cognitive Processing.

Measuring cognitive load: performance, mental effort and simulation task complexity.

Validating Touch Surgery™: A cognitive task simulation and rehearsal app for intramedullary femoral nailing.

Orbitofrontal cortex as a cognitive map of task space.

Can a virtual reality cognitive training application fulfill a dual role? Using the virtual supermarket cognitive training application as a screening tool for mild cognitive impairment.

Cognitive Aids in Medicine Assessment Tool (CMAT): preliminary validation of a novel tool for the assessment of emergency cognitive aids.

A competency framework for colonoscopy training derived from cognitive task analysis techniques and expert review.

Cognitive architectures as a tool for investigating the role of oscillatory power and coherence in cognition.

Materials design. Simulation as a tool for biopolymers design.

Development of a computerized tool for the chinese version of the montreal cognitive assessment for screening mild cognitive impairment.

The Oxford Cognitive Screen (OCS): validation of a stroke-specific short cognitive screening tool.

Inter-hemispheric recruitment as a function of task complexity, age and cognitive reserve.

Exploring adolescent cognitive control in a combined interference switching task.

Cognitive tool for dealing with unexpected difficult airway.

Task matters: influence of different cognitive tasks on cognitive-motor interference during dual-task walking in chronic stroke survivors.

Effect of type of cognitive task and walking speed on cognitive-motor interference during dual-task walking.

Cognitive distance, absorptive capacity and group rationality: a simulation study.

Cognitive Performance during a 24-Hour Cold Exposure Survival Simulation.

Validation of the Edinburgh Cognitive and Behavioural Amyotrophic Lateral Sclerosis Screen (ECAS): A cognitive tool for motor disorders.

Is state-trace analysis an appropriate tool for assessing the number of cognitive systems?

Use of the Internet as a prevention tool against cognitive decline in normal aging.