564594

HFSXXX10.1177/0018720814564594Human FactorsEMR Clutter

Clutter in Electronic Medical Records: Examining Its Performance and Attentional Costs Using Eye Tracking Nadine Moacdieh and Nadine Sarter, University of Michigan, Ann Arbor, Michigan Objective: The objective was to use eye tracking to trace the underlying changes in attention allocation associated with the performance effects of clutter, stress, and task difficulty in visual search and noticing tasks. Background: Clutter can degrade performance in complex domains, yet more needs to be known about the associated changes in attention allocation, particularly in the presence of stress and for different tasks. Frequently used and relatively simple eye tracking metrics do not effectively capture the various effects of clutter, which is critical for comprehensively analyzing clutter and developing targeted, real-time countermeasures. Method: Electronic medical records (EMRs) were chosen as the application domain for this research. Clutter, stress, and task difficulty were manipulated, and physicians’ performance on search and noticing tasks was recorded. Several eye tracking metrics were used to trace attention allocation throughout those tasks, and subjective data were gathered via a debriefing questionnaire. Results: Clutter degraded performance in terms of response time and noticing accuracy. These decrements were largely accentuated by high stress and task difficulty. Eye tracking revealed the underlying attentional mechanisms, and several display-independent metrics were shown to be significant indicators of the effects of clutter. Conclusion: Eye tracking provides a promising means to understand in detail (offline) and prevent (in real time) major performance breakdowns due to clutter. Application: Display designers need to be aware of the risks of clutter in EMRs and other complex displays and can use the identified eye tracking metrics to evaluate and/or adjust their display. Keywords: display design, visual search, attention, medical informatics

Address correspondence to Nadine Moacdieh, Department of Industrial and Operations Engineering, University of Michigan, 1205 Beal Ave., Ann Arbor, MI 48109; [email protected]. HUMAN FACTORS Vol. 57, No. 4, June 2015, pp. 591­–606 DOI: 10.1177/0018720814564594 Copyright © 2015, Human Factors and Ergonomics Society.

Introduction

Display clutter represents a problem for operators in many data-rich, safety-critical domains such as aviation (Kim & Kaber, 2009), security (Zhu & Sun, 2012), and medicine (Singh, Spitzmueller, Petersen, Sawhney, & Sittig, 2013). It leads to significant attention and performance decrements, including degraded object recognition (Bravo & Farid, 2006; Yeh, Merlo, Wickens, & Brandenburg, 2003), delayed visual search (Beck, Trenchard, van Lamsweerde, Goldstein, & Lohrenz, 2012; Henderson, Chanceaux, & Smith, 2009; Moacdieh & Sarter, 2012; Neider & Zelinsky, 2011), increased memory load (Westerbeek & Maes, 2011), and confusion (Alexander, Stelzer, Kim, & Kaber, 2008; Lohrenz, Trafton, Beck, & Gendron, 2009). Despite the agreement that “clutter” is a concern, there is no consensus on how to define, measure, or overcome this multifaceted problem. Based on a recent review of the literature, we refer to clutter as the presence of performance and attentional decrements that result from the interaction between high density of irrelevant data and poor display organization (Moacdieh & Sarter, 2014; Rosenholtz, Li, & Nakano, 2007). Some factors, such as high workload and inexperience, have been shown to exacerbate the effects of clutter (Naylor, 2010). However, the impact of other factors like task difficulty or stress (i.e., the interaction between physical and/or psychological demands, people’s perception of how well they can cope, and their perception of how important these demands are; Staal, 2004) remains less clear. The combination of clutter and stress may present a significant risk in domains where the ability to quickly and accurately extract important information is critical for safety and efficiency. This research therefore aimed to assess the performance

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

592 June 2015 - Human Factors

Figure 1. Fixations are usually depicted as a circle whose diameter is proportional to fixation duration. Saccades are represented as lines between two successive fixations, whereas areas of interest are typically drawn as rectangles. All of the fixations and saccades together create the scanpath.

effects and identify the underlying attentional mechanisms resulting from clutter, stress, and task difficulty in the context of search and noticing tasks. We focused on these tasks given that they are most commonly associated with and affected by clutter (e.g., Rosenholtz et al., 2007). Background

Research on display clutter typically adopts image processing, performance evaluation, and/ or subjective evaluation as assessment techniques. Image processing provides a mathematically defined, display-based measure of clutter (e.g., Camp, Moyer, & Moore, 2010; Rosenholtz et al., 2007). From a human factors perspective, such a measure is of limited use as it does not reflect how factors like stress interact with design to shape performance and attention. Performance evaluation can fill that gap (e.g., Chen & Barnes, 2012; Wickens, Nunes, Alexander, & Steelman, 2005). However, frequently used measures such as response time and accuracy assess performance outcome only; they do not trace the underlying attentional mechanisms associated with clutter. Similarly, subjective evaluation using questionnaires and interviews (e.g., Kaufmann & Kaber, 2010) fails to trace the effects of clutter on attention management and cannot be used in real time. Capturing the attentional mechanisms underlying the performance costs of clutter is important since (a) it can suggest targeted countermeasures and thus inform the redesign of dis-

plays and (b) it helps detect the effects of clutter early on so that the display can be adjusted in real time, forming the basis for an adaptive display. One means of tracing attentional allocation is using eye tracking (see Figure 1; for more details, see McCarley & Kramer, 2006; Poole & Ball, 2005). Several eye tracking metrics have been shown to be valid indicators of the effects of stress on attention allocation (e.g., Di ­Nocera, Camilli, & Terenzi, 2007). Eye tracking has also been used to study clutter (e.g., Beck et al., 2012; Beck, Lohrenz, & Trafton, 2010; ­Grahame, Laberge, & Scialfa, 2004; Neider & Zelinsky, 2011; Zhu & Sun, 2012). But relatively few and simple eye tracking metrics, such as fixation number and mean fixation duration, tend to be used, which may not provide the detail necessary to capture the various aspects of clutter. In addition, many of the metrics used to date rely on areas of interest (AOIs), experimenter-defined areas of the display on which analysis takes place (e.g., Beck et al., 2010; Hegarty, De Leeuw, & Bonura, 2008; ­Josephson & Holmes, 2006). However, AOI-dependent metrics require high eye tracker accuracy to be reliable (Goldberg & Helfman, 2011) and cannot be easily generalized across displays. Finally, the metrics that best reflect the effects of clutter early in the search process have yet to be identified, although these would be invaluable for adapting displays prior to significant performance decrements.

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

EMR Clutter

593

Table 1: Overview of the Eye Tracking Metrics Used in This Study Eye Tracking Metric

Notes

Location metrics Total number of fixations Fixation rate (/sec) Convex hull area (pixels2) Spatial density

Nearest neighbor index (NNI)

Directness metrics Scanpath length (pixels) Scanpath length per second (pixels/sec) Mean saccade length (pixels) Backtrack rate (/sec) Rate of transitions (/sec) Duration metrics Mean fixation duration (sec)

All the fixations within a defined period Total number of fixations divided by the amount of time Minimum convex area which contains the fixation points (Hegarty et al., 2008) Grid cells of size 65 × 45 pixels were created to cover the whole displaya; spatial density is the number of grid cells containing gaze points divided by the total number of cells (Cowen, Ball, & Delin, 2002) The ratio between (a) the average of the observed minimum distances between points and (b) the mean random distance expected if the distribution were random (see Di Nocera et al., 2007, for more details); higher values indicate more dispersion of fixations The sum of all saccade lengths within a defined period The total saccade length per second Mean of all the saccades within a defined period A backtrack is defined as an angle between two saccades that is greater than 90° (Goldberg & Kotval, 1999) Rate of transitions between = grid cells (as defined for spatial density) Mean duration of all fixations within a defined period

a

Grid size width was chosen to be around half the width of the medical history column (see later description of the display), and the length was equivalent to three rows.

To understand the effects of clutter on attention, one needs to examine how clutter affects the location/spread, directness, and duration of eye movements (Moacdieh & Sarter, 2014; see Table 1). Location metrics depend only on fixation coordinates; they show whether clutter causes a dispersion of eye movements across the display, thus preventing the user from focusing on important information. Directness measures differ in that the sequence of fixations is taken into account; these measures can indicate whether clutter made search less ordered or systematic. Finally, duration measures indicate how long a person looked at a particular area and relate clutter primarily to the difficulty extracting information from the display. Increased spread, less directness, and increased duration of fixations

are the potential clutter-induced attentional changes that we focus on in this research. Research Goals

The objectives of our research were to (a) determine the effects of display clutter, stress, and task difficulty on search and noticing performance and (b) examine the attentional mechanisms underlying these performance costs using AOI-independent eye tracking metrics. We defined visual search as the process of actively looking for and locating a given target whose location is unknown (Treisman, 1991), whereas noticing referred to “the conscious registration of some event” (Schmidt, 1995, p. 29), in our case important or critical events, without being given a specific target. The scope of this

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

594 June 2015 - Human Factors

study was limited to analyzing the effects of the excess data aspect of clutter, the most prominent aspect of the phenomenon in the literature to date. Our expectations were that (a) clutter would lead to worse visual search and noticing performance, and that this would be accentuated by high stress and task difficulty, and (b) AOI-independent eye tracking metrics would help explain this degradation in performance by revealing changes in attention allocation. The application domain for this research was electronic medical records (EMRs) used in the emergency department (ED) of a large research hospital. EMRs are digital versions of a patient’s medical health records that are used by physicians to diagnose and treat patients (King, Patel, & Furukawa, 2012). Studies have shown that EMR clutter, and specifically data overload, leads to increased search time for information (Duftschmid et al., 2013), decreased search accuracy (Zeng, Cimino, & Zou, 2002), and more time spent using the EMR overall (Murphy, Reis, Sittig, & Singh, 2012). However, the attentional mechanisms underlying these performance costs remain unclear, as does the interaction between clutter and stress, which, in the ED, is created by a high volume of patients and the serious and time-critical procedures involved (Trzeciak & Rivers, 2003). Methods

Simulated EMR

We replicated the medical history page of the EMR currently being used by all research participants. The page was created using Windows Presentation Foundation with XAML and C#, and was populated with fabricated but realistic patient data. Following discussions and observations with ED physicians, we identified this page as being one of the pages most affected by data overload. It displays the patient’s medical, surgical, social, and family history (see Figure 2; copyright restrictions prevent us from displaying the actual page). To the right of the medical and surgical history columns, there is a column displaying the date of each condition or procedure. Similarly, the social history column has another column to the right that provides brief details about each item (e.g., how many packs of cigarettes a day if the person is a smoker). The family history section also consists of two columns, one showing a particular medical condition and the other indicating the family member who suffered from it. Within each of the sections, the data in each column are not ordered; rather, the order of presentation depends on when it was entered by medical providers. In this experiment, the page was nonscrollable, and participants could not use the links to navigate to other pages. All the information needed for the experiment was present on the page.

Participants

Experiment Setup

The participants in this experiment were 15 medical practitioners (7 female and 8 male). They included 11 residents, 1 staff physician, and 3 physician assistants who were employed in the ED of a large research hospital. Their average age was 30.1 years (SD = 3.3), and their average years of experience in this ED was 4.3 years (SD = 4.0). All participants had been trained to use the EMR currently employed at the hospital. Their average experience using the EMR was 2.0 years (SD = 0.75). All participants gave informed consent and received compensation for their time. This study was approved by the University of Michigan Health Sciences and Behavioral Sciences Institutional Review Board.

The EMR was displayed on a 19-inch monitor with a resolution of 1,280 × 1,024 pixels. An Applied Science Laboratories D-6 desktopmounted eye tracker was used (sampling rate: 60 Hz; accuracy: less than 1° visual angle; precision: 0.5° visual angle). The eye tracker was placed directly in front of the computer monitor such that the aperture was around 2 inches from the screen. The participants were seated at a distance of around 24 inches from the eye tracker, making for a visual angle of around 60° with the screen in the horizontal and vertical directions. Calibration took place at the start of the experiment using a 9-point grid. The duration of the calibration procedure varied across participants but generally took around 5 minutes.

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

EMR Clutter

595

Figure 2. Outline of the page used in the experiment (right), including an example of the medical history entries in a low or high clutter condition (left; the number next to each entry is an identifier for each disease/complaint).

Experiment Design

The three independent variables were clutter (low, high), stress (low, high), and type of search task (simple, difficult). These were varied within participants, leading to a 2 × 2 × 2 full-factorial design. The combination of these three variables translated to eight versions of the EMR page, each containing data for a different fictitious patient scenario. For each patient scenario, there was a corresponding (a) summary of the patient’s symptoms, similar to what a physician would receive in a triage note, (b) search target, that is, information that a physician might look for in a similar situation, (c) noticing target, that is, information that a physician needed to make the correct diagnosis, and (d) suggested diagnosis. Each patient scenario corresponded to one experiment trial, and participants were exposed to the same one trial per condition. The suggested diagnosis was presented to participants as the initial assessment by a medical student. Participants were told to first answer the search question (i.e., find the search target) and then use the rest of their time to assess the likelihood of the diagnosis based on the available EMR data. Note that they were not requested to make a final diagnosis about the patient, as

such a task would entail examining more than just one EMR page. Participants provided all their answers verbally. Collaborating physicians in the ED confirmed that the patient scenarios were realistic and mimicked some of the tasks they perform using their EMR. Table 2 shows all the scenario details, including the text that was presented to physicians and the search and noticing targets. The scenarios are equivalent apart from the manipulated variables and involve searching for a target in one of the eight columns related to medical, surgical, social, or family history. The three experimental variables were manipulated as follows: Clutter. Clutter was operationally defined as the number of text entries (medical/surgical condition, social history entry, or family history entry) in the EMR display. High clutter was created by adding information that was irrelevant to the current condition of the patient. For example, for a patient presenting with abdominal pain, irrelevant information could include entries about skin conditions or ear disorders. The low versus high clutter tasks were equivalent in that (a) there was the same amount of relevant information (i.e., the same number of relevant

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

596 June 2015 - Human Factors Table 2: Text of the Experiment Trials Shown to Participants and the Targets That They Had to Search for or Detect Diagnosis

Clutter

Given Text

Noticing Target

Simple tasks Peptic Low A 45-year-old female has severe abdominal pain, vomiting, Cholecystectomy ulcer clutter and nausea. Your medical student has evaluated the patient and believes she is suffering from cholecystitis. You open the patient’s chart to check her medical history. Does the patient have a history of gastric reflux? Appendectomy High An 80-year-old female has severe abdominal pain and clutter nausea. Your medical student has evaluated the patient and believes she is suffering from appendicitis. You open the patient’s chart to check her medical history. Does the patient have a history of gastritis? Hypertension Ischemic Low A 69-year-old male presents to the emergency department stroke clutter (ED) with dizziness. Your medical student has evaluated the patient and thinks he could be suffering from an inner ear disorder. You open the patient’s chart to review his medical history. What year was the patient diagnosed with diabetes? Atrial fibrillation High A 71-year-old male presents to the ED with a severe clutter sudden headache. Your medical student has evaluated the patient and thinks he could be suffering from migraines. You open the patient’s chart to review his medical history. What year was the patient diagnosed with hypertension? Difficult tasks DVT Low A 50-year-old female has pain and swelling in her left calf. Tibia surgery clutter Your medical student has seen the patient and believes she may have a deep vein thrombosis (DVT). You open the patient’s chart to review her medical history. You also want to see if anyone in her family has had a DVT. Who is the closest relative to have had a DVT? High Knee surgery A 73-year-old male has pain, redness, and swelling in his clutter right calf. Your medical student has seen the patient and believes he has a DVT. You open the patient’s chart to review his medical history and that of his family. Who is the closest relative to have had a pulmonary embolism? Father has aortic Aortic Low A 60-year-old male presents to the ED with acute onset aneurysm dissection clutter of severe chest pain radiating to the back, nausea, and numbness. Your medical student has seen the patient and believes he has aortic dissection. You open the patient’s chart to check for a history of aortic issues. What year was his most recent aortic bypass surgery? High A 79-year-old male presents to the ED with acute onset of Mother had clutter aortic severe chest pain radiating to the back and shortness of dissection breath. Your medical student has seen the patient and believes he has aortic dissection. You open the patient’s chart to check for a history of aortic issues. What year was his most recent aortic root repair? Note. High stress scenarios are in italics; search target shown in bold and underlined here. Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

EMR Clutter

597

entries), (b) the eccentricity of the search and noticing targets was held to within less than a fifth of a page of difference (always in the same position horizontally and never more than 300 pixels apart vertically in a 1,680 × 1,050 pixel image), and (c) the two questions were comparable (e.g., they both required search for an entry in the medical history section). There was an average of 15 (SD = 2.3) entries in low clutter and an average of 41 (SD = 2.8) entries in high clutter. Stress. Stress was induced using a combination of techniques. First, the high stress scenarios involved life-threatening patient conditions, as opposed to less serious conditions for low stress tasks. Second, a time limit of 20 seconds was imposed during the high stress scenarios. This limit was determined empirically, following pilot tests with physicians. Third, participants were told that the person with the best performance in these high stress tasks (in terms of search time and accuracy) would be paid an extra $50, in addition to the compensation for participating in the experiment. Fourth, participants were instructed to be as fast as possible, whereas in the low stress condition, participants were asked only to be both fast and accurate. Finally, the high stress scenarios were accompanied by ventilator and heart rate monitor alerts, sounds that physicians would typically associate with critical conditions. Task. Both simple and difficult search tasks were used. Simple search tasks involved finding only one given piece of information in the display (e.g., “Does the patient have a history of gastritis?”), whereas difficult tasks involved finding several instances of a given item and performing some comparison or judgment (e.g., “When was the patient’s most recent aortic root repair?”). The dependent measures consisted of performance, eye tracking, and subjective data. The performance measures were search time, or the time to find the search target (excluding cases where the answer was incorrect, a response was not provided, or the trial timed out), screen time (the total amount of time spent scanning the display, including both search and noticing tasks), search accuracy (whether participants correctly identified the search target), and noticing accu-

racy (whether participants detected the important information in each EMR page). For both types of accuracy, an error was defined as any case where the participant either did not respond or provided the incorrect answer. With regard to eye tracking, all of the measures in Table 1 were calculated. Finally, subjective measures included clutter ratings and workload assessments using the NASA Task Load Index (TLX). Experiment Procedure

First, participants were given a training session that included a sample task scenario. This was done to familiarize them with the different tasks and required responses. Participants were informed that this study was intended to evaluate the design of EMRs. Next, the eye tracker was calibrated, after which the experiment began. For each trial, the screen first displayed an introduction page that had a description of the patient’s symptoms, the initial diagnosis made by a medical student, the specific question related to the proposed diagnosis (see Table 2), and whether participants had 20 seconds or unlimited time for that task. The presentation order of the variables was counterbalanced by dividing participants into two groups, each of which did the experiment in a fixed random order. The order of the second group was the inverse of the first. When participants were ready, they pressed a button to replace the question screen with the corresponding EMR page. Participants then searched through that page to find the answer to the given question. They were instructed that the search task was their priority, and they needed to provide the answer verbally first and also think aloud throughout. After participants had provided their answer to the search task, they continued scanning the display to look for other case-relevant information for as long as they needed (limited to 20 seconds in the high stress conditions). The EMR page was then replaced by a question page that asked participants to explain whether and why they thought the initial diagnosis was likely or not. They completed all trials with no breaks in between, with the eight pages taking around 10 minutes. After completing all trials, participants filled out a debriefing questionnaire in which they entered clutter ratings for sample images

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

598 June 2015 - Human Factors b

Search me (low stress) 16 14 12 10 8 6 4 2 0

Low cluer High cluer

Simple task

Search me (sec)

Search me (sec)

a

Search me (high stress) 16 14 12 10 8 6 4 2 0

Low cluer High cluer

Difficult task

Simple task

Difficult task

Figure 3. Response time in seconds in (a) low stress and (b) high stress.

and NASA-TLX ratings for the low and high stress conditions.

Unless otherwise specified, results were analyzed using a 2 × 2 × 2 repeated-measures analysis of variance (ANOVA), with Bonferroni corrections applied for multiple statistical tests. Significance was set at p < .05, and partial eta-squared (ηp2) was used as a measure of effect size. The ANOVA results are reported for statistically significant results only, with some descriptive values highlighting notable trends. We refer to the different conditions using L/H for low/high stress and S/D for simple or difficult tasks. Error bars on graphs indicate the standard error of the mean (SEM). Performance Results

Search time. We considered only correct answers, all of which were given within less than the 20-second time limit imposed in the high stress situation. Clutter caused a slight overall increase in search time from 5.85 seconds (SEM = 1.56) in the low clutter condition to 7.78 seconds (SEM = 2.07) in the high clutter condition, with a significant simple simple main effect of clutter in the high stress and difficult task condition only, F(1, 10) = 9.095, p = .013 (Figure 3b). In this case, four participants’ data were discarded due to incorrect answers. Screen time. We considered only low stress conditions (i.e., cases with no time limit) for screen time to avoid including the cutoff time of 20 seconds in the results. One participant’s screen time could not be calculated due to a

Screen me (sec)

Results

Screen me (low stress) 50 40 30 20 10 0 Simple task Low cluer

Difficult task High cluer

Figure 4. Screen time in seconds for the low stress conditions.

technical error during the experiment. Clutter significantly increased the amount of time spent extracting information from the display, from an overall average of 20.05 seconds (SEM = 5.35) in low clutter to 26.38 seconds (SEM = 7.05) in high clutter, main effect of clutter: F(1, 14) = 11.789, p = .004, ηp2 = .331. There was also a simple simple main effect of clutter in the simple task and low stress condition, F(1, 14) = 11.841, p = .016, ηp2 = .458, with high clutter resulting in longer screen time (see Figure 4). Search and noticing accuracy. To determine the effects of clutter, stress, and task difficulty on the likelihood of missing important information, we performed a mixed-model binary logistic regression for each page with clutter, stress, and task as fixed effects and participants as the random effects. For the search tasks, clutter did not result in a significant increase in the number of misses (2 and 6 misses in the low and high clutter condition, respectively, from a total of 60 summed across all 15 participants in each

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

EMR Clutter

Number of misses

14

b

Number of misses (low stress)

14

12 10 8

Low cluer

6

High cluer

4 2

Number of misses

a

599

Number of misses (high stress)

12 10 8

Low cluer

6

High cluer

4 2 0

0 Simple task

Difficult task

Simple task

Difficult task

Figure 5. Number of noticing misses in (a) low stress and (b) high stress (summed across all participants).

condition). For the noticing tasks, clutter resulted in a significant increase in the number of misses in all four conditions (see Figure 5). Regression analysis showed that clutter (odds ratio [OR] = 5.25, 95% confidence interval [CI] = 2.153– 12.856, p < .001), stress (OR = 3.83, 95% CI = 1.584–9.263, p = .003), and task difficulty (OR = 3.83, 95% CI = 1.584–9.263, p = .003) were all significant predictors of misses. Eye Tracking Results

One participant’s data had to be discarded due to poor eye tracking discrimination, resulting in a total of 14 participants for the eye tracking analysis. After eliminating the periods corresponding to eye blinks, the raw eye tracking data (gaze points) were separated into fixations (minimum duration for fixations was 100 ms) and saccades (gaze points had to be within a radius of 2° visual angle; Goldberg & Kotval, 1999). We first examined the search eye tracking metrics (i.e., from start until search target detection) to determine the underlying cause of degraded search performance. We then calculated the overall eye tracking metrics (i.e., throughout the entire time participants spent looking at the screen) to examine the reasons behind poor noticing performance. Finally, we calculated early eye tracking metrics using data from only the first 4 seconds of search to identify metrics that can capture the effects of clutter early on. We chose this time window as it was shorter than almost all the response times but still long enough to calculate the metrics. Only 9 out of 111 (around 8%) of responses were made in less than 4 seconds. In these cases, the actual response time was used instead of 4 seconds.

Search eye tracking metrics. Results showed a main effect of clutter on four location metrics: number of fixations, convex area, spatial density, and NNI (see Table 3). For the directness metrics, there was a significant main effect of clutter on scanpath length per second and mean saccade length. The duration metric was not significantly affected by clutter. Some of the main effects need to be interpreted with caution since there were also significant interaction effects for a number of metrics (see Figure 6, left, and Table 3). First, there were significant clutter × task interactions for the number of fixations (Figure 6a, left), spatial density (Figure 6b, left), and scanpath length (Figure 6c, left) and simple main effects of clutter in the difficult conditions, number of fixations: F(1, 13) = 8.75, p = .011, ηp2 = .402; spatial density: F(1, 13) = 8.807, p = .011, ηp2 = .393; scanpath length: F(1, 13) = 5.359, p = .038, ηp2 = .292. As can be seen in Figures 6a, 6b, and 6c (right), the increase in the metrics in the difficult condition can be largely attributed to the increase in the high stress and difficult task condition. There was also a significant clutter × stress interaction effect for mean saccade length (see Figure 7 and Table 3). However, in this case, clutter caused a significant decrease in mean saccade length in the low stress condition, simple main effect of clutter in low stress: F(1, 13) = 10.451, p = .007, ηp2 = .446. Overall eye tracking metrics. For the location metrics, there was a significant main effect of clutter such that the number of fixations, convex hull area, spatial density, and NNI were higher in high clutter (see Table 4). For the

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

600 June 2015 - Human Factors Table 3: Mean Values of the Search Metrics in the Different Clutter Conditions Eye Tracking Metric Location metrics Total number of fixations

Low Clutter Mean (SEM)

High Clutter Mean (SEM)

12.0 (1.65)

14.73 (2.0)

Main Effects of Clutter

Interaction Effectsa

F(1, 13) = 7.684, p = .016, ηp2 = .371

Clutter × task: F(1, 13) = 7.047, p = .02, ηp2 = .352 Fixation rate 2.00 (0.14) 1.87 (0.13) Not significant — b Convex hull area 2,817 (524) 3,834 (841) F(1, 10) = 4.958, p = — .05, ηp2 = .331 Spatial density 0.0148 (0.00170) 0.0178 (0.00230) F(1, 13) = 6.822, p = Clutter × task: F(1, 13) = 8.429, p = .012, .022, ηp2 = .344 ηp2 = .393 Nearest neighbor 1.87 (0.0248) 2.58 (0.0287) F(1, 13) = 12.13, p = — index .006, ηp2 = .326 Directness metrics Scanpath length 400 (52.3) 464.85 (75.1) Not significant Clutter × task: F(1, 13) = 7.877, p = .015, ηp2 = .377 Scanpath length 70.8 (6.00) 56.8 (5.39) F(1, 13) = 10.452, p = — per second .007, ηp2 = .446 Mean saccade 36.9 (3.75) 30.6 (2.52) F(1, 13) = 5.058, p = Clutter × stress: F(1, 13) = 12.902, p = length .042, ηp2 = .28 .003, ηp2 = .498 Backtrack rate 0.902 (0.0990) 0.819 (0.102) Not significant — Rate of transitions 1.70 (0.136) 1.62 (0.138) Not significant — Duration metrics Mean fixation 0.426 (0.0415) 0.468 (0.0804) Not significant — duration a

Also see Figure 6. Note that in three cases the convex hull area could not be calculated because there were not sufficient fixation points to create the triangulation matrix.

b

directness metrics, only the mean saccade length significantly decreased in high clutter. Finally, the duration metric was not significantly affected by clutter. In this overall phase, we did not analyze the clutter × stress and clutter × task interaction effects given the unequal amount of time that participants could spend on the screen in the low and high stress cases. Early eye tracking metrics. Finally, we calculated once again the metrics that showed a significant main or interaction effect for clutter in the search or overall phase, but this time over a period of 4 seconds. None of the metrics showed significant main effects of clutter; however, convex hull area, NNI, scanpath length per second, and mean

saccade length showed a similar trend to what we observed during the search and overall phases (see Table 5). Significant interaction effects were found only for the mean saccade length. There were both significant clutter × stress interactions and clutter × task interactions (see Figure 8 and Table 5). Clutter significantly decreased the mean saccade length overall in both the simple task, simple main effect of clutter: F(1, 13) = 5.564, p = .07, ηp2 = .3 (Figure 8a) and low stress, simple main effect of clutter: F(1, 13) = 7.243, p = .036, ηp2 = .358 ­(Figure 8b) conditions, as well as the low stress and simple task condition in particular, simple simple main effect of clutter: F(1, 13) = 12.405,

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

EMR Clutter

a

601

Number of fixaons

Number of fixaons (cluer*task) 30

25 20 Low cluer

15

High cluer

10 5

Number of fixaons

Number of fixaons

30

25 20 Low cluer

10

High cluer

5 0

0 Simple

b

Difficult

LS

LD

0.03

0.025

0.025

0.02 0.015

Low cluer

0.01

High cluer

0.005

0.02 Low cluer

0.01

High cluer

0.005 0

Simple

LS

Difficult

Scanpath length (cluer*task)

LD

HS

HD

Scanpath length 1000

800 600

Low cluer

400

High cluer

200 0 Simple

Scanpath length (pixels)

1000 Scanpath length (pixels)

HD

0.015

0

c

HS

Spaal density

Spaal density (cluer*task) 0.03 Spaal density

Spaal density

15

800 600

Low cluer

400

High cluer

200 0

Difficult

LS

LD

HS

HD

Figure 6. Significant interaction effects for the (a) number of fixations, (b) spatial density, and (c) scanpath length (left) and most noticeable increase in the high stress, difficult task (HD) conditions (right). LS = low stress, simple task; LD = low stress, difficult task; HS = high stress, simple task.

Subjective Results

Participants were asked to rate the amount of clutter for one of the low clutter and one of the high clutter EMR pages on a 10-point scale, with 10 referring to maximum clutter. The ratings were analyzed using a Wilcoxon test for ordinal data. High clutter resulted in a significant increase in ratings from a median of 3 (SD = 1.6) in low clutter to 8 (SD = 2.3) in high clutter (median increase = 3; z = 3.316, p = .001). Participants also rated their perceived mental workload in the low stress and high stress

Mean saccade length (cluer*stress) Mean saccade length (pixels)

p = .016, ηp2 = .488 (Figure 8c). The opposite trend was seen in the case of high stress and difficult tasks (Figure 8d).

50 40 30

Low cluer

20

High cluer

10 0 Low stress

High stress

Figure 7. Significant interaction effects for the mean saccade length.

conditions using NASA-TLX scales (scale: 0 to 20 [largest effect]). These rankings were also

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

602 June 2015 - Human Factors Table 4: Mean Values of the Overall Eye Tracking Metrics in the Different Clutter Conditions Eye Tracking Metric

Low Clutter Mean (SEM)

Location metrics Total number of fixations 35.1 (3.54) Fixation rate 1.78 (0.121) Convex hull area 8,255 (1,529) Spatial density 0.0346 (0.00270) Nearest neighbor index 0.31 (0.0301) (NNI) Directness metrics Scanpath length 1506 (306) Scanpath length per 280 (44.4) second Mean saccade length 39.0 (2.18) Backtrack rate 0.95 (0.0870) Rate of transitions 1.63 (0.109) Duration metrics Mean fixation duration 0.430 (0.0320)

High Clutter Mean (SEM)

Main Effects of Clutter

48.2 (6.79) 1.79 (0.0922) 12,833 (1,644) 0.0474 (0.00460) 0.39 (0.0281)

F(1, 13) = 13.657, p = .003, ηp2 = .512 Not significant F(1, 13) = 41.78, p < .001, ηp2 = .763 F(1, 13) = 27.493, p < .001, ηp2 = .679 F(1, 13) = 19.629, p = .001, ηp2 = .602

1630 (220) 251 (47.1)

Not significant Not significant

34.5 (2.16) 0.92 (0.0766) 1.70 (0.0850)

F(1, 13) = 10.317, p = .007, ηp2 = .442 Not significant Not significant

0.440 (0.0366)

Not significant

Table 5: Mean Values of the Early Eye Tracking Measures in the Different Clutter Conditions Eye Tracking Metric Location metrics Total number of fixations Convex hull area Spatial density Nearest neighbor index Directness metrics Scanpath length Scanpath length per second Mean saccade length

Low Clutter Mean (SEM)

High Clutter Mean (SEM)

Main Effects of Clutter

8.01 (0.684)

7.92 (0.642)

Not significant

—

2232 (605) Not significant 0.0107 (0.000800) Not significant 1.89 (0.0341) Not significant

— —  

1866 (350) 0.011 (0.000900) 1.65 (0.0302)

Interaction Effects With Clutter

286 (26.8) 76.2 (8.63)

263 (32.1) 67.2 (7.90)

Not significant Not significant

— —

35.6 (4.02)

33.0 (4.16)

Not significant

Clutter × stress: F(1, 13) = 10.575, p = .006, ηp2 = .449; Clutter × task: F(1, 13) = 6.766, p = .022, ηp2 = .342

analyzed using a Wilcoxon test. As can be seen in Table 6, there were significant effects of stress on all six scales, with high stress notably resulting in increased impressions of temporal demand and frustration.

Discussion and Conclusion

Our goals in this study were twofold. First, we wanted to assess the effects of clutter, in combination with stress and task difficulty,

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

EMR Clutter

40 30

Low cluer

20

High cluer

10 0

c

Mean saccade length (cluer*stress)

Mean saccade length (pixels)

50

Simple

Mean saccade length (pixels)

b

Mean saccade length (cluer*task)

50 40 30

Low cluer

20

High cluer

10 0 Low stress

Difficult

d

Mean saccade length (low stress)

High stress

Mean saccade length (high stress) 60

60 50 40 30

Low cluer

20

High cluer

10 0

Mean saccade length (pixels)

Mean saccade length (pixels)

a

603

50 40 Low cluer

30

High cluer

20 10 0

Simple task

Difficult task

Simple task

Difficult task

Figure 8. Early values of mean saccade length: (a) clutter × task interaction, (b) clutter × stress interaction, and (c) values in the low and high stress conditions.

on visual search and noticing performance. The subjective data confirm that we were able to manipulate clutter and stress, as intended. Results showed that, with high clutter, screen time increased significantly; also, participants missed more critical information. These noticing misses occurred even without a time constraint. As expected, high stress and task difficulty exacerbated the performance effects of clutter. Clutter significantly affected search time only in the high stress and difficult task condition, and search time tended to increase overall with high stress, despite the presence of a time limit and the instruction to search as quickly as possible. This result supports previous studies that have highlighted the detrimental effects of stress and time pressure on performance (e.g., Quigley, Nelson, Carriere, Smilek, & Purdon, 2012). Search accuracy, on the other hand, was not significantly affected by clutter or stress. One possible explanation for this is that the search tasks used in this study were not sufficiently difficult. In terms of the speed–accuracy trade-off, participants seemed to compromise speed in favor of accuracy.

Our second goal was to explain the observed performance effects using AOI-independent eye tracking metrics and to determine which metrics best reflect the effects of clutter on attention allocation. Both during search and overall, performance effects were best reflected in location metrics (specifically, the total number of fixations, convex hull area, spatial density, and NNI). This finding suggests that participants were distracted by unimportant data. The resulting large spread of fixations likely resulted in the increased response time and number of misses, consistent with results from previous studies (Moacdieh, Prinet, & Sarter, 2013; Moacdieh & Sarter, 2012). High clutter also affected ­directness measures. Specifically, mean saccade length and scanpath length per second decreased significantly. Thus, although fixations overall were spread widely across the screen, successive fixations were not very far apart. It seems that participants adopted a slower and possibly more orderly search to compensate for high clutter. Moreover, the duration metric was not significantly affected by clutter. This suggests that extracting and understanding

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

604 June 2015 - Human Factors Table 6: Results of the NASA Task Load Index (NASA-TLX) Ratings Along the Different Dimensions NASA-TLX Scale (0–20) Mental demand Temporal demand Performance Effort Frustration

Low Stress Median (SD)

High Stress Median (SD)

Median Increase

Wilcoxon Exact Sign Test

8 (4.2) 4 (4.0) 6 (2.5) 7 (4.1) 5 (4.0)

12 (3.9) 15 (3.6) 11 (4.2) 12 (3.9) 13 (4.3)

 4 10  4  4  6

z = 2.677, p = .007 z = 3.411, p = .001 z = 3.304, p = .001 z = 3.303, p = .001 z = 3.279, p = .001

information was not a problem for participants. We also examined the eye tracking metrics over the first 4 seconds of each trial. There were no significant effects of clutter in this case, but convex hull area, NNI, scanpath length per second, and mean saccade length revealed similar trends to those observed in the search and overall phases. Finally, the performance decrements resulting from high stress and task difficulty were also reflected in the eye tracking metrics. The number of fixations, spatial density, and scanpath length were all significantly affected by clutter in the difficult task conditions. The effect of clutter also appeared to be most noticeable in the high stress and complex task condition. Thus, any algorithms looking to identify the effects of clutter for a judgment and multiple comparison task, especially under high stress, should consider using these metrics. On the other hand, mean saccade length would appear to be less useful in the case of high stress and difficult tasks as the effect of clutter was significant only in low stress and simple task conditions. The practical implications of this study include the use of eye tracking metrics to create intelligent, adaptive displays where adjustments to information presentation are triggered in real time. The fact that the early eye tracking metrics revealed a similar trend suggests that adaptations may be triggered early on and before significant performance effects occur, although more research is needed on this topic. The metrics can also be used offline to detect and analyze the effects of clutter on the spread, directness, and duration of search, providing display designers and researchers with a framework for the comprehensive evaluation of clutter.

The main limitation of this experiment is the relatively small number of participants. Combined with the Bonferroni corrections, this meant that the differences between several mean values narrowly failed to reach significance. Also, asking participants to think aloud may have been disruptive, although physicians may often talk to other medical personnel while using their EMR. Acknowledgments We would like to thank Travis Ganje, Benjamin Bassin, and David Somand for their helpful insights and feedback. We would also like to thank the medical personnel who participated in the experiment.

Key Points •• Electronic medical record clutter degrades response time and noticing accuracy, particularly in conditions of high stress and task difficulty. •• Eye tracking metrics suggest that the performance decrements are due to an increased spread of eye fixations and a slower and more methodical search. •• A number of eye tracking metrics reflect the effects of clutter on the search process and can help inform the design of displays, as well as form the basis for early detection and adjustment of displays in real time. •• In particular, all of the metrics identified are AOIindependent, meaning that they are not limited to particular displays or domains.

References Alexander, A. L., Stelzer, E. M., Kim, S. H., & Kaber, D. B. (2008). Bottom-up and top-down contributors to pilot perceptions of display clutter in advanced flight deck technologies. In Proceedings of the Human Factors and Ergonomics Society annual meeting (pp. 1180–1184). Thousand Oaks, CA: Sage. doi:10.1177/154193120805201806

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

EMR Clutter

605

Beck, M. R., Lohrenz, M. C., & Trafton, J. G. (2010). Measuring search efficiency in complex visual search tasks: Global and local clutter. Journal of Experimental Psychology: Applied, 16, 238–250. doi:10.1037/a0019633 Beck, M. R., Trenchard, M., van Lamsweerde, A., Goldstein, R. R., & Lohrenz, M. (2012). Searching in clutter: Visual attention strategies of expert pilots. In Proceedings of the Human Factors and Ergonomics Society annual meeting (pp. 1411–1415). Thousand Oaks, CA: Sage. doi:10.1177/1071181312561400 Bravo, M. J., & Farid, H. (2006). Object recognition in dense clutter. Attention, Perception, & Psychophysics, 68, 911–918. doi:10.3758/BF03193354 Camp, H. A., Moyer, S., & Moore, R. K. (2010). Comparing masked target transform volume (MTTV) clutter metric to human observer evaluation of visual clutter. In Proceedings of SPIE defense, security, and sensing (pp. 76620A-1–12). Orlando, FL: International Society for Optics and Electronics. doi:10.1117/12.851799 Chen, J. Y., & Barnes, M. J. (2012). Supervisory control of multiple robots effects of imperfect automation and individual differences. Human Factors, 54, 157–174. doi:10.1177/0018720811435843 Cowen, L., Ball, L. J., & Delin, J. (2002). An eye movement analysis of web page usability. In People and Computers XVI— Memorable yet invisible (pp. 317–335). London, UK: Springer. Di Nocera, F., Camilli, M., & Terenzi, M. (2007). A random glance at the flight deck: Pilots’ scanning strategies and the real-time assessment of mental workload. Journal of Cognitive Engineering and Decision Making, 1, 271–285. doi:10.1518/1555 34307X255627 Duftschmid, G., Rinner, C., Kohler, M., Huebner-Bloder, G., Saboor, S., & Ammenwerth, E. (2013). The EHR-ARCHE project: Satisfying clinical information needs in a Shared Electronic Health Record System based on IHE XDS and Archetypes. International Journal of Medical Informatics, 82, 1195–1207. doi:10.1016/j.ijmedinf.2013.08.002 Goldberg, J., & Helfman, J. (2011). Eye tracking for visualization evaluation: Reading values on linear versus radial graphs. Information Visualization, 10, 182–195. doi:10.1177/1473871611406623 Goldberg, J. H., & Kotval, X. P. (1999). Computer interface evaluation using eye movements: Methods and constructs. International Journal of Industrial Ergonomics, 24, 631–645. doi:10.1016/S0169-8141(98)00068-7 Grahame, M., Laberge, J., & Scialfa, C. T. (2004). Age differences in search of web pages: The effects of link size, link number, and clutter. Human Factors, 46, 385–398. doi:10.1518/ hfes.46.3.385.50404 Hegarty, M., De Leeuw, K., & Bonura, B. (2008, November). What do spatial ability tests really measure. Paper presented at the 49th meeting of the Psychonomic Society, Chicago, IL. Henderson, J. M., Chanceaux, M., & Smith, T. J. (2009). The influence of clutter on real-world scene search: Evidence from search efficiency and eye movements. Journal of Vision, 9, 32. doi:10.1167/9.1.32 Josephson, S., & Holmes, M. E. (2006). Clutter or content? How on-screen enhancements affect how TV viewers scan and what they learn. In Proceedings of the 2006 symposium on eye tracking research & applications (pp. 155–162). San Diego, CA: ACM Press. doi:10.1145/1117309.1117361 Kaufmann, K. A., & Kaber, D. B. (2010). The influence of individual differences in perceptual performance on pilot percep-

tions of head-up display clutter. In Proceedings of the Human Factors and Ergonomics Society annual meeting (pp. 70–74). Thousand Oaks, CA: Sage. doi:10.1177/154193121005400116 Kim, S.-H., & Kaber, D. B. (2009). Assessing the effects of conformal terrain features in advanced head-up displays on pilot performance. In Proceedings of the Human Factors and Ergonomics Society annual meeting (pp. 36–40). Thousand Oaks, CA: Sage. doi:10.1177/154193120905300109 King, J., Patel, V., & Furukawa, M. (2012). Physician adoption of electronic health record technology to meet meaningful use objectives: 2009–2012 (ONC Data Brief No. 7). Washington, DC: Office of the National Coordinator for Health Information Technology. Lohrenz, M. C., Trafton, J. G., Beck, M. R., & Gendron, M. L. (2009). A model of clutter for complex, multivariate geospatial displays. Human Factors, 51, 90–101. doi:10.1177/0018720809333518 McCarley, J. S., & Kramer, A. F. (2006). Eye movements as a window on perception and cognition. In M. Rizzo & R. Parasuraman (Eds.), Neuroergonomics: The brain at work (pp. 95–112). New York, NY: Oxford University Press. Moacdieh, N. M., Prinet, J. C., & Sarter, N. B. (2013). Effects of modern primary flight display clutter: Evidence from performance and eye tracking data. In Proceedings of the Human Factors and Ergonomics Society annual meeting (pp. 11–15). Thousand Oaks, CA: Sage. doi:10.1177/1541931213571005 Moacdieh, N. M., & Sarter, N. B. (2012). Eye tracking metrics: A toolbox for assessing the effects of clutter on attention allocation. In Proceedings of the Human Factors and Ergonomics Society annual meeting (pp. 1366–1370). Thousand Oaks, CA: Sage. doi:10.1177/1071181312561391 Moacdieh, N., & Sarter, N. (2014). Display clutter: A review of definitions and measurement techniques. Human Factors. Advance online publication. doi:0018720814541145. Murphy, D. R., Reis, B., Sittig, D. F., & Singh, H. (2012). Notifications received by primary care practitioners in electronic health records: A taxonomy and time analysis. The American Journal of Medicine, 125(2), 209.e1–209.e7. doi: 10.1016/j. amjmed.2011.07.029 Naylor, J. (2010). The influence of dynamics, flight domain and individual flight training & experience on pilot perception of clutter in aviation displays (Master’s thesis). Available from http://repository.lib.ncsu.edu/ir/handle/1840.16/6103 Neider, M. B., & Zelinsky, G. J. (2011). Cutting through the clutter: Searching for targets in evolving complex scenes. Journal of Vision, 11, 7. doi:10.1167/11.14.7 Poole, A., & Ball, L. J. (2005). Eye tracking in human-computer interaction and usability research. In C. Ghaoui (Ed.), Encyclopedia of human computer interaction (pp. 211–219). Hershey, PA: Idea Group. Quigley, L., Nelson, A. L., Carriere, J., Smilek, D., & Purdon, C. (2012). The effects of trait and state anxiety on attention to emotional images: An eye-tracking study. Cognition & Emotion, 26, 1390–1411. doi:10.1080/02699931.2012.662892 Rosenholtz, R., Li, Y., & Nakano, L. (2007). Measuring visual clutter. Journal of Vision, 7, 1–22. doi:10.1167/7.2.17. Schmidt, R. W. (1995). Consciousness and foreign language learning: A tutorial on the role of attention and awareness in learning. In R. W. Schmidt (Ed.), Attention and awareness in foreign language learning (pp. 1–63). Honolulu: University of Hawaii, National Foreign Language Resource Center. Singh, H., Spitzmueller, C., Petersen, N. J., Sawhney, M. K., & Sittig, D. F. (2013). Information overload and missed test results

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015

606 June 2015 - Human Factors in electronic health record–based settings. Journal of the American Medical Association Internal Medicine, 173, 702–704. doi:10.1001/2013.jamainternmed.61 Staal, M. A. (2004). Stress, cognition, and human performance: A literature review and conceptual framework (NASA Tech. Memo. 212824). Moffett Field, CA: NASA Ames Research Center. Treisman, A. (1991). Search, similarity, and integration of features between and within dimensions. Journal of Experimental Psychology: Human Perception and Performance, 17, 652. doi:10.1037/0096-1523.17.3.652 Trzeciak, S., & Rivers, E. P. (2003). Emergency department overcrowding in the United States: An emerging threat to patient safety and public health. Emergency Medicine Journal, 20, 402–405. doi:10.1136/emj.20.5.402 Westerbeek, H. G. W., & Maes, A. (2011). Referential scope and visual clutter in navigation tasks. In K. van Deemter, A. Gatt, R. van Gompel, & E. J. Krahmer (Eds.), Proceedings of the Workshop on the Production of Referring Expressions (PRECogSci 2011) (pp. 1–6). Boston, MA: Cognitive Science Society. Wickens, C. D., Nunes, A., Alexander, A. L., & Steelman, K. (2005). 3D navigation and integrated hazard display in advanced avionics: Workload, performance, and situation awareness display clutter issues in SVS design (AHFD-05-19/ NASA-05). Urbana-Champaign: University of Illinois Human Factors Division. Yeh, M., Merlo, J. L., Wickens, C. D, & Brandenburg, D. L. (2003). Head up versus head down: The costs of imprecision, unreliability, and visual clutter on cue effectiveness for

­display signaling. Human Factors, 45, 390–407. doi:10.1518/ hfes.45.3.390.27249 Zeng, Q., Cimino, J. J., & Zou, K. H. (2002). Providing conceptoriented views for clinical data using a knowledge-based system: An evaluation. Journal of the American Medical Informatics Association, 9, 294–305. doi:10.1197/jamia.M1008 Zhu, B., & Sun, X. (2012). Effects of superposition on oculomotor guidance and target recognition. In Proceedings of the Human Factors and Ergonomics Society annual meeting (pp. 333–337). Thousand Oaks, CA: Sage. doi:10.1177/1071181312561077

Nadine Moacdieh is a PhD candidate in the Department of Industrial and Operations Engineering, Center for Ergonomics, at the University of Michigan. She earned her MSE in industrial and operations engineering from the University of Michigan, Ann Arbor in 2012. Nadine Sarter is a professor in the Department of Industrial and Operations Engineering, Center for Ergonomics, at the University of Michigan. She earned her PhD in industrial and systems engineering from Ohio State University in 1994. Date received: July 22, 2014 Date accepted: November 21, 2014

Downloaded from hfs.sagepub.com at UNIV NEBRASKA LIBRARIES on September 7, 2015