This article was downloaded by: [The University of British Columbia] On: 14 October 2014, At: 13:31 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
The Journal of General Psychology Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/vgen20
Time Pressure Heuristics Can Improve Performance Due to Increased Consistency a
Stephen Rice & David Trafimow
a
a
New Mexico State University Published online: 20 Sep 2012.
To cite this article: Stephen Rice & David Trafimow (2012) Time Pressure Heuristics Can Improve Performance Due to Increased Consistency, The Journal of General Psychology, 139:4, 273-288, DOI: 10.1080/00221309.2012.705187 To link to this article: http://dx.doi.org/10.1080/00221309.2012.705187
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content.
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
The Journal of General Psychology, 2012, 139(4), 273–288 C 2012 Taylor & Francis Group, LLC Copyright
Time Pressure Heuristics Can Improve Performance Due to Increased Consistency STEPHEN RICE DAVID TRAFIMOW New Mexico State University
ABSTRACT. Our goal is to demonstrate that potential performance theory (PPT) provides a unique type of methodology for studying the use of heuristics under time pressure. While most theories tend to focus on different types of strategies, PPT distinguishes between random and nonrandom effects on performance. We argue that the use of a heuristic under time pressure actually can increase performance by decreasing randomness in responding. We conducted an experiment where participants performed a task under time pressure or not. In turn, PPT equations make it possible to parse the observed change in performance from the unspeeded to the speeded condition into that which is due to a change in the participant’s randomness in responding versus that which is due to a change in systematic factors. We found that the change in randomness was slightly more important than the change in systematic factors. Keywords: consistency, PPT, pressure, time
RESEARCHERS WHO STUDY THE USE OF HEURISTICS under time pressure generally argue that time pressure influences performance by causing a systematic change in the processing of information (Maule, Hockey & Bdzola, 2000; Svenson & Maule, 1993; Maule & Edland, 1997). Although we agree that this is so, we also believe that time pressure can influence the randomness of performance. We propose that a recently published theory, potential performance theory (PPT; Trafimow & Rice, 2008, 2009), can be used as a methodology to study how time pressure causes random effects as well as systematic ones in task performance. With this goal in mind, after reviewing briefly the relevant literature, we explain how PPT works and how it can be used to study the use of heuristics under time
Address correspondence to Stephen Rice, Department of Psychology, MSC 3452, New Mexico State University, PO Box 30001, Las Cruces, NM 88003-8001, USA; sc [email protected] (e-mail). 273
274
The Journal of General Psychology
pressure. We then present an experiment that demonstrates the potential of PPT as a methodology rather than simply as a theory.
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
Time Pressure Past research has shown how time pressure can produce a profound effect on decision-making and judgment (Maule et al., 2000; Svenson & Maule, 1993; Maule & Edland, 1997). It is intuitive that time pressure can reduce the accuracy of the decision-making process (Payne, Bettman & Johnson, 1993) or result in less extreme judgments (Kaplan, Wanshula & Zanna, 1993). However, not all the effects of time pressure are negative (Maule et al., 2000). Svenson and Benson (1993) revealed that when under time pressure, participants were less likely to be negatively affected by a framing bias, resulting in superior overall decisions. It is believed that time pressure may cause people to fundamentally change the way they process information. Decision-making and judgments tasks are subjected to at least two different types of cognitive processing (Evans, 2007; Evans, 2008; Evans, Handley & Bacon, 2009; Kahneman & Frederick, 2002; Sloman, 1996; Stanovich, 1999)—analytic processing and heuristic processing. Analytic processing involves slow, sequential, and less automatic thinking, where people make a methodical decision based on information and situations that they may not have encountered before. Heuristic processing, on the other hand, involves quick, automatic, and unconscious thinking (Goldstein & Gigerenzer, 2002; Hogarth, 1981). For our purposes, the important distinction between the two types of processing is the amount of time involved in them. When under time pressure, people often do not have time to finish the slower analytic processing (Harreveld, Wagenmakers & Maas, 2007), which forces them to consider switching strategies in order to save time (Rieskamp & Hoffrage, 2008). One different strategy might be to simply speed up the gathering of information needed for the analytic processing (Ben-Zur & Breznitz, 1981; Edland, 1994; Kerstholt, 1995; Payne, Bettman & Johnson, 1988) while also selecting what they consider to be the important information in a shorter amount of time (Ben-Zur & Breznitz, 1981; B¨ockenholt & Kroeger, 1993; Kerstholt, 1995; Payne, et al., 1988; Wright, 1974). One way to facilitate this approach might be to rely more on cues that point to what they perceive to be the more important information at hand (Maule, 1994; Payne et al., 1988; Rieskamp & Hoffrage, 2008). Another strategy that people under time pressure might take is to switch to heuristic processing. It is believed that heuristic processing reflects a strategy of rapid, automatic cognitive processing that requires few mental resources and is distinct from the slower analytic processing described earlier (Todd & Gigerenzer, 2000). In cases where the heuristic processing system is superior to the analytic processing system, stressors that hinder the analytic system may actually improve
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
Rice & Trafimow
275
overall performance (De Neys, 2006), implying that people can be induced into heuristic processing. Imagine a situation where a person has to say whether a stimulus is present or absent and is aided by an automated device that recommends whether the stimulus is present or absent at some level of reliability. The person could choose to make decisions entirely on his or her own, to simply agree with the automated aid and shortcut the decision process, or to undergo a more complex process that takes own opinions and recommendations from the automated aid into account. If the heuristic is to simply comply more with an automated aid that is superior to own non-augmented performance, then not only will people reduce the amount of time needed to make a decision, but their decisions will be superior to those people who chose the slower, deliberative approach that resulted in non-compliance with the automated aid. In fact, this is exactly what Rice and Keller (2009) found. They had participants search through aerial images of Baghdad for enemy weapons. This task was augmented by a diagnostic aid that provided recommendations during each trial. The aid was 100%, 95%, 80%, or 65% reliable. In one condition, participants were allowed plenty of time to examine the image and make their decisions. In a second condition, participants were put under time pressure and only allowed 2 seconds to make a decision. The authors reported that when under time pressure, participants were more likely to comply with the aid than their counterparts who were not under time pressure, regardless of the aid’s reliability. One weakness of the study was that the authors were unable to determine how much of the improved performance was due to systematic effects of using the heuristics, or if part of the improvement was due to simply being more consistent in using the heuristic. Fortunately, a new theory of task performance has been developed in psychology that may allow us to re-examine these issues and parse the systematic and random factors that comprise human performance. This new theory is termed potential performance theory (PPT) (Trafimow & Rice, 2008, 2009). Potential Performance Theory (PPT) According to PPT, all task performance is due to two general underlying classes of causes (Trafimow & Rice, 2008); these are random and nonrandom. The totality of random effects is indexed by consistency. To obtain consistency as it is defined by PPT, there must be at least two blocks of trials and consistency is defined as the correlation between both of them (Trafimow & Rice, 2008, 2009, 2011). As previously stated, PPT suggests that consistency plays an influential role in one’s observed task performance. It is worth mentioning that consistency in terms of PPT is across blocks of trials, which should not be confused with any other sense of the term. Different fields of research have used the term “consistency” to refer to several different phenomena. For instance, attribution research uses consistency to refer to a person responding similarly across different situations
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
276
The Journal of General Psychology
(Kelley & Michela, 1980; Orvis, Cunningham, & Kelley, 1975). According to work on the Lens Model (Brunswik, 1952), consistency is used to refer to the correlation between predicted judgments and actual judgments, or the predicted criterion value and the actual criterion value. Furthermore, personality research uses consistency to refer to stability of personality traits. PPT uses yet another sense of the term, where consistency refers to the correlation across two blocks of matched trials. We further emphasize that this is a correlation that is obtained for each individual that participates in the experiment. Thus, for example, if there are 25 participants who complete two blocks of 50 trials where “yes” or “no” responses are possible, the correlation of yes and no responses across the two blocks of 50 trials is each participant’s consistency coefficient (and there would be 25 such coefficients). A high consistency coefficient indicates that there is little randomness in the participant’s responses whereas a low consistency coefficient indicates that there is much randomness. Thus, consistency can be considered to be an inverse measure of randomness. Lest consistency coefficients be confused with Cronbach’s alpha, we hasten to point out an important difference. Cronbach’s alpha concerns how well items go with each other; it is an intra-class coefficient that measures the internal consistency of items making up a measure. But when the issue is to correct for randomness in performance, internal consistency is not relevant; rather, a consistency coefficient across blocks is required. Another way to put it is that Cronbach’s alpha is insufficient to handle “transient error” (error across occasions), and so it is necessary to have at least two blocks of trials. In addition to randomness, there also are nonrandom or systematic effects that are indexed by potential scores. Conceptually, the idea of PPT is to take observed scores—each participant’s proportion of successes—and “correct” for randomness using the consistency coefficient. This correction provides the best possible estimate of how each participant would have performed if there were no randomness. These estimates of random-free performance are called potential scores (please see Appendix A for the formulas needed to calculate potential scores). A high potential score indicates that the participant would perform at a high level in the absence of randomness whereas a low potential score indicates that the participant would perform at a low level, even with the deleterious effects of randomness removed. It is important to understand that it is a simple fact of mathematics that increasing randomness will push scores in the direction of chance performance. Therefore, if base performance is greater than the chance level, as it usually is, increasing randomness will decrease observed performance. Consequently, one way to think about PPT is that it corrects for the deleterious effects of randomness by providing for the calculation of potential scores. Trafimow and Rice (2008) provided the basic mathematics for calculating potential scores. Subsequently, Trafimow and Rice (2009) provided mathematics that allow researchers to compute, across conditions, the amount of change in observed scores that would
Rice & Trafimow
277
happen if only consistency changed, if only potential scores changed, or the joint effects of both kinds of change.
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
Current Study A PPT perspective suggests that, as with everything, there are two general reasons for human-automation performance to be at less than optimal levels. Human-automation performance might be less consistent (more random) than automation by itself or there might be systematic problems with combining a human and automation. Similarly, if using a speeded task influences humanautomation performance by causing participants to use a heuristical approach, it could be through influencing consistency, changing potential scores, or both. The experiment to be presented tests both random and systematic effects. Our main hypothesis is that at least some of the effect of using a speeded task on observed performance will be due to random effects. Stated in a more operational vein, we hypothesize that using a speeded task will increase human-automation performance and that this increase will be due, in substantial part, to an increase in the consistency of using the speeded heuristic. Method Participants Sixteen participants (eight females) from a major university in the southwestern United States took part in the experiment for course credit. The mean age was 20.06 (SD = 2.27). All participants were tested for normal or corrected to normal vision. Apparatus and Stimuli The experimental display (a partial replication of the Rice & Keller, 2009 study) was presented via E-Prime 1.1 on a Dell computer with a 22” monitor, using 1024 × 768 resolution. Target-absent images were created by using 30 aerial photographs of Baghdad. Target-present images were created by digitally superimposing a simulated helicopter onto the 30 aerial photographs using Photoshop CS3. Thus, there were 60 photographic stimuli—30 with helicopters and 30 without helicopters. The target helicopter occupied approximately 2 degrees of visual angle both horizontally and vertically. Figure 1 presents an example target-present stimulus. Automated Aid A simulated diagnostic aid was used to present recommendations to participants before each trial. The aid would either say, “The automation detects a
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
278
The Journal of General Psychology
FIGURE 1. A sample target-present stimulus. The helicopter is in the top right corner. The actual image was in color.
helicopter” or “The automation has determined there is no helicopter.” The aid was set to 95% accuracy, erring only by providing false alarms on 5% of the trials. The aid was always perfectly consistent, in that if it produced an error on a particular image during the first block of trials, it would produce the same error on the next block of trials. Procedure Participants first signed a consent form, and were then seated 21” from the display. Position was controlled with a chin rest. Instructions were given within the e-prime program and all questions were answered verbally by the experimenter. Participants were serially, and randomly, presented with 60 aerial photographs of Baghdad, in which half of the photographs contained an enemy helicopter. Participants were instructed that if they detected a helicopter, they should press the “J” key on the provided keyboard. If they determined that no helicopter was present, they should press the “F” key. Participants were asked to maintain as high accuracy as possible within the time constraints. Participants were also informed
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
Rice & Trafimow
279
that they would be assisted by an automated diagnostic aid that would give them a recommendation on whether or not a helicopter was present. They were told exactly how reliable the aid was and what types of errors they could expect it to make (false alarms). Lastly, participants were told that they would either have 2 seconds or 8 seconds to view each image after the automation provided its recommendation. Each of the 100 trials began with a slide presenting the recommendation of the automation. This slide remained for 1000 ms, followed by the aerial image. In the Speeded condition (participants were under time pressure), each photograph remained present for 2 seconds. In the Unspeeded condition, each photograph remained present for 8 seconds. After the allotted time, the image was replaced with a Choice screen in which participants determined whether or not they saw a target. Following a participant response, a feedback slide was presented. This slide stated, “Correct!” in green letters or “Incorrect!” in red letters, with a running total of overall accuracy. This slide remained present for 1000 ms, followed by the next trial. After 100 trials, the experiment ended automatically. Upon completion, participants were then debriefed and dismissed. Design A within-participants design was used, in which all participants were exposed to both the Unspeeded and Speeded conditions twice. The 4 blocks (2 for each condition) were randomly ordered for each participant. Participants were told at the beginning of each independent block whether they had 2 or 8 seconds to view the image. Results We performed two categories of analyses. First, we performed traditional PPT analyses on observed scores (percent correct), potential scores (estimates of the percent that would have been correct in the absence of randomness), and consistency scores (within-participants correlations across blocks of trials). Appendix A provides details of the mathematical formulae that were used to obtain potential scores. Second, we performed sequential analyses to test the assumption that the trials are locally independent. Group Analyses The Group data can be found in Figure 2 and Table 1. Consistent with previous research (Rice & Keller, 2009; Rice, Trafimow, Keller, Hunt & Geels, 2011), observed scores were greater in the speeded than in the unspeeded condition (M = 93% and M = 83%), t(15) = 4.44, d = 2.29, p < .001. Given this, it is not surprising that potential scores were also greater in the speeded than in the
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
280
The Journal of General Psychology
FIGURE 2. Group data from the experiment. Mean observed scores indicate the mean percent correct. Mean potential scores indicate the mean percent correct in the absence of randomness.
unspeeded condition (M = 99% and M = 93%), t(15) = 2.42, d = .80, p = 01. In essence, the potential scores indicate that, in the absence of randomness, performance in the speeded condition would be close to perfect whereas this would not be so in the unspeeded condition. Of primary interest, consistency coefficients also were greater in the speeded than in the unspeeded condition (M = .77 and
TABLE 1. Group Data From the Experiment
Speeded Unspeeded
Observed Score
Potential Score
Consistency
93 (.04) 83 (.09)
99 (.02) 93 (.09)
.77 (.10) .61 (.22)
Mean observed scores indicate the mean percent correct. Mean potential scores indicate the mean percent correct in the absence of randomness. Mean consistency scores indicate the mean within-participants correlation across the two blocks of trials. Standard deviations are in parentheses.
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
Rice & Trafimow
281
M = .61), t(15) = 3.27, d = .93, p = .003. Clearly, performances were more consistent, and less random, in the speeded than in the unspeeded condition. In order to show that not all of the increase in observed score performance was due to an increase in potential scores, we conducted a two-way ANOVA using Observed/Potential scores as one Factor and Speeded/Unspeeded as the other factor, with performance being the dependent variable. Not surprisingly, there was both a main effect of Observed/Potential scores, F(1, 15) = 57.27, p < .001, ηp 2 = .79, and a main effect of Speeded/Unspeeded, F(1, 15) = 12.77, p = .003, ηp 2 = .46. Most importantly, there was a significant interaction between the two factors, F(1, 15) = 6.25, p = .024, ηp 2 = .29, indicating that at least some of the improvement to observed scores was due to an increase in consistency in the Speeded condition. How much were observed scores influenced by the changes in both systematic and random effects across the two conditions? According to the procedure proposed by Trafimow and Rice (2009), it is possible to calculate the effects of the change in potential scores on observed scores, keeping consistency constant. And it is possible to calculate the effect of the change in consistency on observed scores, keeping potential scores constant. Controlling for consistency, the increase in observed scores is from 83% to 87% from the unspeeded to the speeded condition. The effect of the change in consistency, controlling for potential scores, is from 83% to 88% from the unspeeded to the speeded condition. Of course, because observed scores jumped from 83% to 93% from the unspeeded to the speeded condition, it should be clear that the interactive effect of increases in potential scores and consistency coefficients mattered. This effect is consistent with PPT effects that Trafimow and Rice (2009) demonstrated. Note that the unique effect of the change in consistency on observed scores slightly exceeds that of the change in potential scores. Stated more generally, the decrease in randomness engendered by the speeded condition was at least as important as the increase in systematic effects in explaining the increase in observed performance.
Sequential Analyses PPT assumes that each trial is independent; the response to trial N has no influence on the response to trial N+1. If this assumption were shown to be false with respect to the present data, the strength of our conclusions would be compromised. Therefore, we devoted the following analyses to supporting that the trials really were independent. First, we constructed a N by N+1 matrix for each participant and computed a correlation coefficient to see if trial N predicted trial N+1. The mean correlation coefficient was −.032 and not discernibly different from zero. Second, we used the adjusted ratio of clustering (ARC) to look for sequential effects. To compute an ARC score, one counts the number of repetitions of
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
282
The Journal of General Psychology
responses in the relevant categories for each participant. An ARC score of 0 indicates that the number of repetitions is at the chance level whereas an ARC score of 1 indicates that there is the maximum possible number of repetitions. The mean ARC score was -.031 and not discernibly different from zero. Therefore, both types of sequential analyses failed to uncover any evidence for sequential effects, and there is little reason to fear that the PPT equations were misapplied to our data. In general, although the mean observed scores, potential scores, and consistency coefficients are important, they can hide some of the action. Individuals varied widely with respect to how much the increase in performance from the unspeeded to the speeded condition was due to decreasing randomness (increasing consistency) or increasing advantageous systematic factors (increasing potential scores). Put another way, although changes in randomness and changes in systematic factors were about equally important, on average, there were individuals who deviated substantially from average. Discussion Theoretical and Methodological Issues We commenced by citing research showing that observed scores improve in a speeded condition relative to an unspeeded one (Rice & Keller, 2009; Rice et al., 2011b). Although there are explanations based on changes in systematic effects, to our knowledge, no one has suggested that a change in random effects could matter. A PPT theoretical perspective suggests the possibility that any changes in observed performance across conditions in any experiment might be due, at least in part, to a change in randomness (Rice et al., 2010, 2011a, 2011b, 2012; Trafimow & Rice, 2008, 2009, 2011). Applying this idea to the present task suggests the possibility that an important cause for improvement in observed performance from the unspeeded to the speeded condition might be because the speeded condition reduces randomness. We found that, on average, the change in randomness is as important as the change in systematic effects for causing the improvement in observed performance. Further PPT analyses show that individuals vary widely in the extent to which reductions in randomness, or increases in advantageous systematic effects, are responsible for the improvement in performance from the unspeeded to the speeded condition. From the point of view of heuristics, our findings seem easy to explain. In the speeded condition, participants had insufficient time to use alternative ways to perform the task, and so they used the heuristic of complying with the automation. Given that this is so, it is possible to ask what is gained by using the present PPT methodology. To answer this question, consider that what often seems to be a simple mechanism sometimes becomes seen as being more complicated as more fundamental underlying principles are discovered. For example, Aristotle
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
Rice & Trafimow
283
explained the falling of objects in a simplistic way; objects fall because it is in their nature to fall. Aristotle argued that heavier objects fall faster than less heavy ones because heavy objects have more of this nature than do less heavy ones. Of course, with the discoveries of Galileo, we now see that Aristotle’s simple mechanism is really due to the confluence of different forces such as inertia and gravity. In addition, we now know that with friction controlled for, Aristotle’s prediction is simply wrong. The present research depends heavily on the ability of PPT to account for all randomness and the reader might question whether this is so. There is evidence in the form of mathematical proofs, computer simulations, and empirical evidence. Interestingly, the original theory was based on a general mathematical proof of this point (Trafimow & Rice, 2008). More recently, Trafimow, MacDonald, and Rice (2012) provided a more specific mathematical proof of the same point from a signal detection theory perspective; this latter proof showed that under conditions where the only factor that impedes performance is random noise, using potential scores results in perfect corrected performance. Trafimow and colleagues (2012) also provided computer simulations that further supported that PPT does account for all randomness (mean corrected performance really was perfect across thousands of computer generated experiments), and these computer simulations buttress earlier more general simulations performed by Trafimow and Rice (2011). Even in the speeded condition of the present experiment, correcting for the effects of randomness caused potential scores to be close to perfect (M = 99%). Limitations As is usually the case, there are limitations. The most obvious limitation is that we used introductory undergraduate students, and so the findings might not generalize to real work situations. A second limitation is that participants did not have prior experience with the task, which might have increased the randomness in their performances relative to if they had been more experienced (e.g., Trafimow & Rice, 2009). There also is a limitation of more theoretical importance. Specifically, we tested only one type of heuristic, whereas there is a large class of potential heuristics that are theoretically available for testing. We hope and expect that future research will remedy this lack by including tests of additional heuristic processes. Conclusion We see the present experiment as a demonstration of a larger PPT issue. Trafimow and Rice (2008) originally developed PPT in the context of morality. They commenced from the standpoint that moral decisions could be made in the context of an absolute standard (e.g., the bible, Kant’s moral imperative, and so on) or a relative one (e.g., the opinion of another person). In the former case,
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
284
The Journal of General Psychology
“success” or “failure” would be defined via agreement or disagreement with the absolute standard, whereas in the latter case, success or failure would be defined via agreement or disagreement with another entity (usually another person). The original PPT equations were developed to be sufficiently flexible to be used either way. In addition, Trafimow and Rice suggested PPT as a general theory, rather than just a theory of morality, though they originally derived their equations in the context of the morality literature. As often is so, a methodology came with the theory, and the methodology involves presenting at least two blocks of trials instead of only one block of trials. From there, PPT equations take over. Because journals differ on whether they favor contributions to theory versus empirical contributions, there has been a tendency to present PPT as a theory in journals that emphasize theoretical contributions, whereas PPT has been presented as a methodology when the focus is on empirical contributions to substantive areas. We believe that the distinction between theory and method is too strong. It is inevitable that theories will imply methodologies. Going the other way, all methodologies involve some kind of underlying theory, though that theory might be well or ill specified. In the present article, we attempted a Goldilocks effect—to hit a happy medium between using PPT from a theoretical versus methodological perspective, and to show that considering PPT simultaneously as both a theory and a methodology facilitates interesting research, while adding applied value as well. AUTHOR NOTES Stephen Rice is an Associate Professor of Psychology at New Mexico State University. He received his Ph.D. in Experimental Psychology from the University of Illinois at Urbana-Champaign in 2006. His research interests include social cognition, human performance and potential performance theory. David Trafimow is a professor of psychology at New Mexico State University, a Fellow of the Association for Psychological Science, and an executive editor of the Journal of General Psychology. He received his PhD in psychology from the University of Illinois at Urbana-Champaign in 1993. His current research interests include attribution, attitudes, cross-cultural research, methodology, and potential performance theory. REFERENCES Ben Zur, H., & Breznitz, S. J. (1981). The effect of time pressure on risky choice behavior. Acta Psychologica, 47, 89–104. B¨ockenholt, U., & Kroeger, K. (1993). The effect of time pressure in multiattribute binary choice tasks. In O. Svenson & A. J. Maule (Eds.), Time pressure and stress in human judgment and decision making (pp. 195–214). New York, NY: Plenum Press. Brunswik, E. (1952). The conceptual framework of psychology: International encyclopedia of unified science. Chicago, IL: University of Chicago Press.
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
Rice & Trafimow
285
De Neys, W. (2006). Dual processing in reasoning—Two systems but one reasoner. Psychological Science, 17, 428–433. Edland, A. C. (1994). Time pressure and the application of decision rules. Scandinavian Journal of Psychology, 35, 281–291. Evans, J. St. B. T. (2007). Hypothetical thinking: Dual process in reasoning and judgement. Hove, UK: Psychology Press. Evans, J. St. B. T. (2008). Dual-processing accounts of reasoning, judgment and social cognition. Annual Review of Psychology, 59, 255–278. Evans, J. St. B. T., Handley, S. J., & Bacon, A. M. (2009). Reasoning under time pressure: A study of causal conditional inference. Experimental Psychology, 56(2), 77–83. Goldstein, D. G., & Gigerenzer, G. (2002). Models of ecological rationality: The recognition heuristic. Psychological Review, 109, 75–90. Hammond, K.R. (1980). Introduction to Brunswikian theory and methods. New Directions for Methodology of Social and Behavioral Science, 3, 1–11. Harreveld, F., Wagenmakers, E., & van der Maas, H. (2007). The effects of time pressure on chess skill: An investigation into fast and slow processes underlying expert performance. Psychological Research, 71, 591–597. Hogarth, R. (1981). Beyond discrete biases: Functional and dysfunctional aspects of judgmental heuristics. Psychological Bulletin, 90(2), 197–217. Kahneman, D., & Frederick, S. (2002). Representativeness revisited: Attribute substitution in intuitive judgement. In T. Gilovich, D. Griffin, & D. Kahneman (Eds.), Heuristics and biases: The psychology of intuitive judgment (pp. 49–81). Cambridge, UK: Cambridge University Press. Kaplan, M. F., Wanshula, L. T., & Zanna, M. P. (1993). Time pressure and information integration in social judgement. In O. Svenson & A. J. Maule, Time pressure and stress in human judgement and decision making (pp. 255–267). New York, NY: Plenum. Kelley, H. H., & Michela, J. L. (1980). Attribution theory and research. Annual Review of Psychology, 31, 457–501. Kerstholt, J. H. (1995). Decision making in a dynamic situation: The effect of false alarms and time pressure. Journal of Behavioral Decision Making, 8, 181–200. Maule, A. J. (1994). A componential investigation of the relation between structural modelling and cognitive accounts of human judgement. Acta Psychologica, 87, 199–216. Maule, A. J., & Edland, A. C. (1997). The effects of time pressure on judgment and decision making. In R. Ranyard, W. R. Crozier & O. Svenson, Decision making: cognitive models and explanation (pp. 189–204). London : Routledge & Kegan Paul. Maule, A., Hockey, G., & Bdzola, L. (2000). Effects of time-pressure on decision-making under uncertainty: Changes in affective state and information processing strategy. Acta Psychological, 104, 283–301. Orvis, B. R., Cunningham, J. D., & Kelley, H. H. (1975). A closer examination of causal inference: The roles of consensus, distinctiveness, and consistency information. Journal of Personality and Social Psychology, 32(4), 605–616. Payne, J. W., Bettman, J. R., & Johnson, E. J. (1988). Adaptive strategy selection in decision making. Journal of Experimental Psychology: Learning, Memory, & Cognition, 14, 534–552. Payne, J. W., Bettman, J. R., & Johnson, E. J. (1993). The adaptive decision maker. Cambridge, UK: Cambridge University Press. Rice, S., Geels, K., Trafimow, D. & Hackett, H. (2011). Our students suffer from both lack of knowledge and consistency: A PPT analysis of test-taking. US-China Education Review, 1(6), 845–855. Rice, S., & Keller, D. (2009). Automation reliance under time pressure. Cognitive Technology, 14(1), 36–44.
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
286
The Journal of General Psychology
Rice, S., Trafimow, D., Clayton, K., & Hunt, G. (2008). Impact of the contrast effect on trust ratings and behavior with automated systems. Cognitive Technology Journal, 13(2), 30–41. Rice, S., Trafimow, D. & Hunt, G. (2010). Using PPT to analyze sub-optimal humanautomation performance. Journal of General Psychology, 137(03), pp. 310–329. Rice, S., Trafimow, D., Keller, D., Hunt, G. & Geels, K. (2011). Using PPT to correct for inconsistency in a speeded task. The Journal of General Psychology, 138(1), 12–34. Rieskamp, J., & Hoffrage, U. (2008). Inferences under time pressure: How opportunity costs affect strategy selection. acta psychological, 127, 258–276. Sloman, S. A. (1996). The empirical case for two systems of reasoning. Psychological Bulletin, 119, 3–22. Stanovich, K. E. (1999). Who is rational? Studies of individual differences in reasoning. Mahway, NJ: Lawrence Elrbaum Associates. Svenson, O., & Benson, L. (1993). Framing and time pressure in decision making. In O. Svenson & A. J. Maule, Time pressure and stress in human judgement and decision making (pp. 133–144). New York, NY: Plenum. Svenson, O., & Maule A. J. (1993). Time pressure and stress in human judgement and decision making. New York, NY: Plenum. Todd, P. M., & Gigerenzer, G. (2000). Pr´ecis of Simple heuristics that make us smart. Behavioral and Brain Sciences, 23, 727–780. Trafimow, D., MacDonald, J., & Rice, S. (2012). Using PPT to account for randomness in perception. Attention, Perception and Psychophysics, 74, 1355–1365. Trafimow, D., & Rice, S. (2008). Potential Performance Theory: A general theory of task performance applied to morality. Psychological Review, 115(2), 447–462. Trafimow, D., & Rice, S. (2009). Potential Performance Theory (PPT): Describing a methodology for analyzing task performance. Behavior Research Methods, 41(2), 359– 371. Trafimow, D., & Rice, S. (2011). Using a sharp instrument to parse apart strategy and consistency: An evaluation of PPT and its assumptions. Journal of General Psychology, 138(3), 169–184. Wright, P. L. (1974). The harassed decision maker: time pressures, distractions and the use of evidence. Journal of Applied Psychology, 59, 555–561.
Original manuscript received January 12, 2012 Final version accepted June 18, 2012 APPENDIX A The PPT strategy is as follows. In a dichotomous task, there are two possible responses to each trial, and each possible response might be correct or incorrect. Thus, for each participant, it is possible to create a 2 (option chosen) by 2 (option correct) frequency table. Table 1 illustrates the frequency table with the four cells. The participant can choose the first option and be correct (cell a) or incorrect (cell c), or the participant can choose the second option and be correct (cell d) or incorrect (cell c). There are also margin frequencies constructed as follows. The first row frequency is the sum of cells a and b whereas the second row frequency is the sum of cells c and d (r1 and r2 , respectively). The first column frequency is the sum of cells a and c whereas the second column frequency is the sum of
Rice & Trafimow
287
cells b and d (c1 and c2 , respectively). Each participant’s frequency table can be converted to a correlation coefficient via Equation 1.
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
r=
|ad − bc| |ad − bc| = (a + b)(c + d)(a + c)(b + d) r1 r2 c1 c2
(1)
The second step, assuming that there were at least two blocks of trials in the experiment, is to obtain the consistency coefficient. This is simply the correlation, for each participant, across blocks of trials. The third step is to use the famous correction formula that was originally derived by Spearman (1904), is a standard derivation from classical true score theory, but also can be derived from more modern theories (see Allen & Yen, 1979; Cohen & Swerdlik, 1999; Crocker & Algina, 1986; Gulliksen, 1987; Lord & Novick, 1968 for reviews). The formula is provided in Equation 2, where R is the best estimate of the correlation coefficient that would be obtained in the absence of randomness and rxx is the consistency coefficient across blocks of trials that was computed on Step 2. R=√
r rxx
(2)
It is worth noting that Equation 2 is a special case of a more general correction formula. In the general case, there are two variables being correlated, and so there is a consistency coefficient associated with each variable. However, we are interested in only one variable; this is participant’s responding to the task trials. Therefore, for our present purposes, the simplified formula is sufficient. But it is possible to imagine a task where there is no correct answer, and so the criterion for “success” is agreement with another person. In this case, there would be a consistency coefficient for each person because each person would be capable of responding with some degree of randomness. As Trafimow and Rice (2008) showed, when there is no correct answer, the more general correction formula must be used, where each person’s responding is treated as a separate variable. Given that a corrected correlation coefficient has been obtained, the next step is to convert it back into a frequency table. But the new frequency table is not a table of observed frequencies, but rather of potential frequencies, which are the best estimates of the frequencies that would be obtained in the absence of randomness (i.e., with perfect consistency). These can be obtained via Equations 3–6 below. Equations 3–6 require fixing the margin frequencies at the obtained levels, similar to a Chi Square test or a Fisher’s Exact test, and using R to estimate the potential cell frequencies. Trafimow and Rice (2008) provide derivations of Equations 3–6 and Trafimow and Rice (2009) provide empirical tests that strongly validate the assumptions. Finally, Trafimow and Rice (2011) provide further validation in the form of computer simulations. Consistent with previous research, we use upper
288
The Journal of General Psychology
case letters (A, B, C, D, R1 , R2 , C1 , C2 ) to refer to the cell and margin frequencies in Equations 3–6 below. A=
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
B= C= D=
√ R R1 R2 C 1 C 2 + C 1 R1 (R1 + R2 ) √ R1 (R1 + R2 ) − (R R1 R2 C1 C2 + C1 R1 ) (R1 + R2 ) √ C 1 R 2 − R R1 R 2 C 1 C 2 (R1 + R2 ) √ C2 (R1 + R2 ) − R1 (R1 + R2 ) − R R1 R2 C1 C2 + C1 R1 (R1 + R2 )
(3) (4) (5) (6)
The foregoing steps for using PPT can be summarized as follows, providing that the participants have responded to two blocks of matched trials. First, the researcher uses Equation 1 to convert each participant’s observed performance table into a correlation coefficient. Second, the researcher obtains a consistency coefficient for each participant. Third, the researcher uses Equation 2 to obtain a corrected or potential correlation coefficient. Finally, the researcher uses Equations 3–6 to estimate the cell frequencies that would be obtained without randomness (i.e., potential cell frequencies). Once the potential cell frequencies have been obtained, each participant’s potential performance, or the performance level each person would have if the responses had been perfectly consistent across blocks can be calculated via Equation 7 below. potential performance =
A+D (A + B + C + D)
(7)
The Journal of General Psychology Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/vgen20
Time Pressure Heuristics Can Improve Performance Due to Increased Consistency a
Stephen Rice & David Trafimow
a
a
New Mexico State University Published online: 20 Sep 2012.
To cite this article: Stephen Rice & David Trafimow (2012) Time Pressure Heuristics Can Improve Performance Due to Increased Consistency, The Journal of General Psychology, 139:4, 273-288, DOI: 10.1080/00221309.2012.705187 To link to this article: http://dx.doi.org/10.1080/00221309.2012.705187
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content.
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
The Journal of General Psychology, 2012, 139(4), 273–288 C 2012 Taylor & Francis Group, LLC Copyright
Time Pressure Heuristics Can Improve Performance Due to Increased Consistency STEPHEN RICE DAVID TRAFIMOW New Mexico State University
ABSTRACT. Our goal is to demonstrate that potential performance theory (PPT) provides a unique type of methodology for studying the use of heuristics under time pressure. While most theories tend to focus on different types of strategies, PPT distinguishes between random and nonrandom effects on performance. We argue that the use of a heuristic under time pressure actually can increase performance by decreasing randomness in responding. We conducted an experiment where participants performed a task under time pressure or not. In turn, PPT equations make it possible to parse the observed change in performance from the unspeeded to the speeded condition into that which is due to a change in the participant’s randomness in responding versus that which is due to a change in systematic factors. We found that the change in randomness was slightly more important than the change in systematic factors. Keywords: consistency, PPT, pressure, time
RESEARCHERS WHO STUDY THE USE OF HEURISTICS under time pressure generally argue that time pressure influences performance by causing a systematic change in the processing of information (Maule, Hockey & Bdzola, 2000; Svenson & Maule, 1993; Maule & Edland, 1997). Although we agree that this is so, we also believe that time pressure can influence the randomness of performance. We propose that a recently published theory, potential performance theory (PPT; Trafimow & Rice, 2008, 2009), can be used as a methodology to study how time pressure causes random effects as well as systematic ones in task performance. With this goal in mind, after reviewing briefly the relevant literature, we explain how PPT works and how it can be used to study the use of heuristics under time
Address correspondence to Stephen Rice, Department of Psychology, MSC 3452, New Mexico State University, PO Box 30001, Las Cruces, NM 88003-8001, USA; sc [email protected] (e-mail). 273
274
The Journal of General Psychology
pressure. We then present an experiment that demonstrates the potential of PPT as a methodology rather than simply as a theory.
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
Time Pressure Past research has shown how time pressure can produce a profound effect on decision-making and judgment (Maule et al., 2000; Svenson & Maule, 1993; Maule & Edland, 1997). It is intuitive that time pressure can reduce the accuracy of the decision-making process (Payne, Bettman & Johnson, 1993) or result in less extreme judgments (Kaplan, Wanshula & Zanna, 1993). However, not all the effects of time pressure are negative (Maule et al., 2000). Svenson and Benson (1993) revealed that when under time pressure, participants were less likely to be negatively affected by a framing bias, resulting in superior overall decisions. It is believed that time pressure may cause people to fundamentally change the way they process information. Decision-making and judgments tasks are subjected to at least two different types of cognitive processing (Evans, 2007; Evans, 2008; Evans, Handley & Bacon, 2009; Kahneman & Frederick, 2002; Sloman, 1996; Stanovich, 1999)—analytic processing and heuristic processing. Analytic processing involves slow, sequential, and less automatic thinking, where people make a methodical decision based on information and situations that they may not have encountered before. Heuristic processing, on the other hand, involves quick, automatic, and unconscious thinking (Goldstein & Gigerenzer, 2002; Hogarth, 1981). For our purposes, the important distinction between the two types of processing is the amount of time involved in them. When under time pressure, people often do not have time to finish the slower analytic processing (Harreveld, Wagenmakers & Maas, 2007), which forces them to consider switching strategies in order to save time (Rieskamp & Hoffrage, 2008). One different strategy might be to simply speed up the gathering of information needed for the analytic processing (Ben-Zur & Breznitz, 1981; Edland, 1994; Kerstholt, 1995; Payne, Bettman & Johnson, 1988) while also selecting what they consider to be the important information in a shorter amount of time (Ben-Zur & Breznitz, 1981; B¨ockenholt & Kroeger, 1993; Kerstholt, 1995; Payne, et al., 1988; Wright, 1974). One way to facilitate this approach might be to rely more on cues that point to what they perceive to be the more important information at hand (Maule, 1994; Payne et al., 1988; Rieskamp & Hoffrage, 2008). Another strategy that people under time pressure might take is to switch to heuristic processing. It is believed that heuristic processing reflects a strategy of rapid, automatic cognitive processing that requires few mental resources and is distinct from the slower analytic processing described earlier (Todd & Gigerenzer, 2000). In cases where the heuristic processing system is superior to the analytic processing system, stressors that hinder the analytic system may actually improve
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
Rice & Trafimow
275
overall performance (De Neys, 2006), implying that people can be induced into heuristic processing. Imagine a situation where a person has to say whether a stimulus is present or absent and is aided by an automated device that recommends whether the stimulus is present or absent at some level of reliability. The person could choose to make decisions entirely on his or her own, to simply agree with the automated aid and shortcut the decision process, or to undergo a more complex process that takes own opinions and recommendations from the automated aid into account. If the heuristic is to simply comply more with an automated aid that is superior to own non-augmented performance, then not only will people reduce the amount of time needed to make a decision, but their decisions will be superior to those people who chose the slower, deliberative approach that resulted in non-compliance with the automated aid. In fact, this is exactly what Rice and Keller (2009) found. They had participants search through aerial images of Baghdad for enemy weapons. This task was augmented by a diagnostic aid that provided recommendations during each trial. The aid was 100%, 95%, 80%, or 65% reliable. In one condition, participants were allowed plenty of time to examine the image and make their decisions. In a second condition, participants were put under time pressure and only allowed 2 seconds to make a decision. The authors reported that when under time pressure, participants were more likely to comply with the aid than their counterparts who were not under time pressure, regardless of the aid’s reliability. One weakness of the study was that the authors were unable to determine how much of the improved performance was due to systematic effects of using the heuristics, or if part of the improvement was due to simply being more consistent in using the heuristic. Fortunately, a new theory of task performance has been developed in psychology that may allow us to re-examine these issues and parse the systematic and random factors that comprise human performance. This new theory is termed potential performance theory (PPT) (Trafimow & Rice, 2008, 2009). Potential Performance Theory (PPT) According to PPT, all task performance is due to two general underlying classes of causes (Trafimow & Rice, 2008); these are random and nonrandom. The totality of random effects is indexed by consistency. To obtain consistency as it is defined by PPT, there must be at least two blocks of trials and consistency is defined as the correlation between both of them (Trafimow & Rice, 2008, 2009, 2011). As previously stated, PPT suggests that consistency plays an influential role in one’s observed task performance. It is worth mentioning that consistency in terms of PPT is across blocks of trials, which should not be confused with any other sense of the term. Different fields of research have used the term “consistency” to refer to several different phenomena. For instance, attribution research uses consistency to refer to a person responding similarly across different situations
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
276
The Journal of General Psychology
(Kelley & Michela, 1980; Orvis, Cunningham, & Kelley, 1975). According to work on the Lens Model (Brunswik, 1952), consistency is used to refer to the correlation between predicted judgments and actual judgments, or the predicted criterion value and the actual criterion value. Furthermore, personality research uses consistency to refer to stability of personality traits. PPT uses yet another sense of the term, where consistency refers to the correlation across two blocks of matched trials. We further emphasize that this is a correlation that is obtained for each individual that participates in the experiment. Thus, for example, if there are 25 participants who complete two blocks of 50 trials where “yes” or “no” responses are possible, the correlation of yes and no responses across the two blocks of 50 trials is each participant’s consistency coefficient (and there would be 25 such coefficients). A high consistency coefficient indicates that there is little randomness in the participant’s responses whereas a low consistency coefficient indicates that there is much randomness. Thus, consistency can be considered to be an inverse measure of randomness. Lest consistency coefficients be confused with Cronbach’s alpha, we hasten to point out an important difference. Cronbach’s alpha concerns how well items go with each other; it is an intra-class coefficient that measures the internal consistency of items making up a measure. But when the issue is to correct for randomness in performance, internal consistency is not relevant; rather, a consistency coefficient across blocks is required. Another way to put it is that Cronbach’s alpha is insufficient to handle “transient error” (error across occasions), and so it is necessary to have at least two blocks of trials. In addition to randomness, there also are nonrandom or systematic effects that are indexed by potential scores. Conceptually, the idea of PPT is to take observed scores—each participant’s proportion of successes—and “correct” for randomness using the consistency coefficient. This correction provides the best possible estimate of how each participant would have performed if there were no randomness. These estimates of random-free performance are called potential scores (please see Appendix A for the formulas needed to calculate potential scores). A high potential score indicates that the participant would perform at a high level in the absence of randomness whereas a low potential score indicates that the participant would perform at a low level, even with the deleterious effects of randomness removed. It is important to understand that it is a simple fact of mathematics that increasing randomness will push scores in the direction of chance performance. Therefore, if base performance is greater than the chance level, as it usually is, increasing randomness will decrease observed performance. Consequently, one way to think about PPT is that it corrects for the deleterious effects of randomness by providing for the calculation of potential scores. Trafimow and Rice (2008) provided the basic mathematics for calculating potential scores. Subsequently, Trafimow and Rice (2009) provided mathematics that allow researchers to compute, across conditions, the amount of change in observed scores that would
Rice & Trafimow
277
happen if only consistency changed, if only potential scores changed, or the joint effects of both kinds of change.
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
Current Study A PPT perspective suggests that, as with everything, there are two general reasons for human-automation performance to be at less than optimal levels. Human-automation performance might be less consistent (more random) than automation by itself or there might be systematic problems with combining a human and automation. Similarly, if using a speeded task influences humanautomation performance by causing participants to use a heuristical approach, it could be through influencing consistency, changing potential scores, or both. The experiment to be presented tests both random and systematic effects. Our main hypothesis is that at least some of the effect of using a speeded task on observed performance will be due to random effects. Stated in a more operational vein, we hypothesize that using a speeded task will increase human-automation performance and that this increase will be due, in substantial part, to an increase in the consistency of using the speeded heuristic. Method Participants Sixteen participants (eight females) from a major university in the southwestern United States took part in the experiment for course credit. The mean age was 20.06 (SD = 2.27). All participants were tested for normal or corrected to normal vision. Apparatus and Stimuli The experimental display (a partial replication of the Rice & Keller, 2009 study) was presented via E-Prime 1.1 on a Dell computer with a 22” monitor, using 1024 × 768 resolution. Target-absent images were created by using 30 aerial photographs of Baghdad. Target-present images were created by digitally superimposing a simulated helicopter onto the 30 aerial photographs using Photoshop CS3. Thus, there were 60 photographic stimuli—30 with helicopters and 30 without helicopters. The target helicopter occupied approximately 2 degrees of visual angle both horizontally and vertically. Figure 1 presents an example target-present stimulus. Automated Aid A simulated diagnostic aid was used to present recommendations to participants before each trial. The aid would either say, “The automation detects a
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
278
The Journal of General Psychology
FIGURE 1. A sample target-present stimulus. The helicopter is in the top right corner. The actual image was in color.
helicopter” or “The automation has determined there is no helicopter.” The aid was set to 95% accuracy, erring only by providing false alarms on 5% of the trials. The aid was always perfectly consistent, in that if it produced an error on a particular image during the first block of trials, it would produce the same error on the next block of trials. Procedure Participants first signed a consent form, and were then seated 21” from the display. Position was controlled with a chin rest. Instructions were given within the e-prime program and all questions were answered verbally by the experimenter. Participants were serially, and randomly, presented with 60 aerial photographs of Baghdad, in which half of the photographs contained an enemy helicopter. Participants were instructed that if they detected a helicopter, they should press the “J” key on the provided keyboard. If they determined that no helicopter was present, they should press the “F” key. Participants were asked to maintain as high accuracy as possible within the time constraints. Participants were also informed
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
Rice & Trafimow
279
that they would be assisted by an automated diagnostic aid that would give them a recommendation on whether or not a helicopter was present. They were told exactly how reliable the aid was and what types of errors they could expect it to make (false alarms). Lastly, participants were told that they would either have 2 seconds or 8 seconds to view each image after the automation provided its recommendation. Each of the 100 trials began with a slide presenting the recommendation of the automation. This slide remained for 1000 ms, followed by the aerial image. In the Speeded condition (participants were under time pressure), each photograph remained present for 2 seconds. In the Unspeeded condition, each photograph remained present for 8 seconds. After the allotted time, the image was replaced with a Choice screen in which participants determined whether or not they saw a target. Following a participant response, a feedback slide was presented. This slide stated, “Correct!” in green letters or “Incorrect!” in red letters, with a running total of overall accuracy. This slide remained present for 1000 ms, followed by the next trial. After 100 trials, the experiment ended automatically. Upon completion, participants were then debriefed and dismissed. Design A within-participants design was used, in which all participants were exposed to both the Unspeeded and Speeded conditions twice. The 4 blocks (2 for each condition) were randomly ordered for each participant. Participants were told at the beginning of each independent block whether they had 2 or 8 seconds to view the image. Results We performed two categories of analyses. First, we performed traditional PPT analyses on observed scores (percent correct), potential scores (estimates of the percent that would have been correct in the absence of randomness), and consistency scores (within-participants correlations across blocks of trials). Appendix A provides details of the mathematical formulae that were used to obtain potential scores. Second, we performed sequential analyses to test the assumption that the trials are locally independent. Group Analyses The Group data can be found in Figure 2 and Table 1. Consistent with previous research (Rice & Keller, 2009; Rice, Trafimow, Keller, Hunt & Geels, 2011), observed scores were greater in the speeded than in the unspeeded condition (M = 93% and M = 83%), t(15) = 4.44, d = 2.29, p < .001. Given this, it is not surprising that potential scores were also greater in the speeded than in the
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
280
The Journal of General Psychology
FIGURE 2. Group data from the experiment. Mean observed scores indicate the mean percent correct. Mean potential scores indicate the mean percent correct in the absence of randomness.
unspeeded condition (M = 99% and M = 93%), t(15) = 2.42, d = .80, p = 01. In essence, the potential scores indicate that, in the absence of randomness, performance in the speeded condition would be close to perfect whereas this would not be so in the unspeeded condition. Of primary interest, consistency coefficients also were greater in the speeded than in the unspeeded condition (M = .77 and
TABLE 1. Group Data From the Experiment
Speeded Unspeeded
Observed Score
Potential Score
Consistency
93 (.04) 83 (.09)
99 (.02) 93 (.09)
.77 (.10) .61 (.22)
Mean observed scores indicate the mean percent correct. Mean potential scores indicate the mean percent correct in the absence of randomness. Mean consistency scores indicate the mean within-participants correlation across the two blocks of trials. Standard deviations are in parentheses.
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
Rice & Trafimow
281
M = .61), t(15) = 3.27, d = .93, p = .003. Clearly, performances were more consistent, and less random, in the speeded than in the unspeeded condition. In order to show that not all of the increase in observed score performance was due to an increase in potential scores, we conducted a two-way ANOVA using Observed/Potential scores as one Factor and Speeded/Unspeeded as the other factor, with performance being the dependent variable. Not surprisingly, there was both a main effect of Observed/Potential scores, F(1, 15) = 57.27, p < .001, ηp 2 = .79, and a main effect of Speeded/Unspeeded, F(1, 15) = 12.77, p = .003, ηp 2 = .46. Most importantly, there was a significant interaction between the two factors, F(1, 15) = 6.25, p = .024, ηp 2 = .29, indicating that at least some of the improvement to observed scores was due to an increase in consistency in the Speeded condition. How much were observed scores influenced by the changes in both systematic and random effects across the two conditions? According to the procedure proposed by Trafimow and Rice (2009), it is possible to calculate the effects of the change in potential scores on observed scores, keeping consistency constant. And it is possible to calculate the effect of the change in consistency on observed scores, keeping potential scores constant. Controlling for consistency, the increase in observed scores is from 83% to 87% from the unspeeded to the speeded condition. The effect of the change in consistency, controlling for potential scores, is from 83% to 88% from the unspeeded to the speeded condition. Of course, because observed scores jumped from 83% to 93% from the unspeeded to the speeded condition, it should be clear that the interactive effect of increases in potential scores and consistency coefficients mattered. This effect is consistent with PPT effects that Trafimow and Rice (2009) demonstrated. Note that the unique effect of the change in consistency on observed scores slightly exceeds that of the change in potential scores. Stated more generally, the decrease in randomness engendered by the speeded condition was at least as important as the increase in systematic effects in explaining the increase in observed performance.
Sequential Analyses PPT assumes that each trial is independent; the response to trial N has no influence on the response to trial N+1. If this assumption were shown to be false with respect to the present data, the strength of our conclusions would be compromised. Therefore, we devoted the following analyses to supporting that the trials really were independent. First, we constructed a N by N+1 matrix for each participant and computed a correlation coefficient to see if trial N predicted trial N+1. The mean correlation coefficient was −.032 and not discernibly different from zero. Second, we used the adjusted ratio of clustering (ARC) to look for sequential effects. To compute an ARC score, one counts the number of repetitions of
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
282
The Journal of General Psychology
responses in the relevant categories for each participant. An ARC score of 0 indicates that the number of repetitions is at the chance level whereas an ARC score of 1 indicates that there is the maximum possible number of repetitions. The mean ARC score was -.031 and not discernibly different from zero. Therefore, both types of sequential analyses failed to uncover any evidence for sequential effects, and there is little reason to fear that the PPT equations were misapplied to our data. In general, although the mean observed scores, potential scores, and consistency coefficients are important, they can hide some of the action. Individuals varied widely with respect to how much the increase in performance from the unspeeded to the speeded condition was due to decreasing randomness (increasing consistency) or increasing advantageous systematic factors (increasing potential scores). Put another way, although changes in randomness and changes in systematic factors were about equally important, on average, there were individuals who deviated substantially from average. Discussion Theoretical and Methodological Issues We commenced by citing research showing that observed scores improve in a speeded condition relative to an unspeeded one (Rice & Keller, 2009; Rice et al., 2011b). Although there are explanations based on changes in systematic effects, to our knowledge, no one has suggested that a change in random effects could matter. A PPT theoretical perspective suggests the possibility that any changes in observed performance across conditions in any experiment might be due, at least in part, to a change in randomness (Rice et al., 2010, 2011a, 2011b, 2012; Trafimow & Rice, 2008, 2009, 2011). Applying this idea to the present task suggests the possibility that an important cause for improvement in observed performance from the unspeeded to the speeded condition might be because the speeded condition reduces randomness. We found that, on average, the change in randomness is as important as the change in systematic effects for causing the improvement in observed performance. Further PPT analyses show that individuals vary widely in the extent to which reductions in randomness, or increases in advantageous systematic effects, are responsible for the improvement in performance from the unspeeded to the speeded condition. From the point of view of heuristics, our findings seem easy to explain. In the speeded condition, participants had insufficient time to use alternative ways to perform the task, and so they used the heuristic of complying with the automation. Given that this is so, it is possible to ask what is gained by using the present PPT methodology. To answer this question, consider that what often seems to be a simple mechanism sometimes becomes seen as being more complicated as more fundamental underlying principles are discovered. For example, Aristotle
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
Rice & Trafimow
283
explained the falling of objects in a simplistic way; objects fall because it is in their nature to fall. Aristotle argued that heavier objects fall faster than less heavy ones because heavy objects have more of this nature than do less heavy ones. Of course, with the discoveries of Galileo, we now see that Aristotle’s simple mechanism is really due to the confluence of different forces such as inertia and gravity. In addition, we now know that with friction controlled for, Aristotle’s prediction is simply wrong. The present research depends heavily on the ability of PPT to account for all randomness and the reader might question whether this is so. There is evidence in the form of mathematical proofs, computer simulations, and empirical evidence. Interestingly, the original theory was based on a general mathematical proof of this point (Trafimow & Rice, 2008). More recently, Trafimow, MacDonald, and Rice (2012) provided a more specific mathematical proof of the same point from a signal detection theory perspective; this latter proof showed that under conditions where the only factor that impedes performance is random noise, using potential scores results in perfect corrected performance. Trafimow and colleagues (2012) also provided computer simulations that further supported that PPT does account for all randomness (mean corrected performance really was perfect across thousands of computer generated experiments), and these computer simulations buttress earlier more general simulations performed by Trafimow and Rice (2011). Even in the speeded condition of the present experiment, correcting for the effects of randomness caused potential scores to be close to perfect (M = 99%). Limitations As is usually the case, there are limitations. The most obvious limitation is that we used introductory undergraduate students, and so the findings might not generalize to real work situations. A second limitation is that participants did not have prior experience with the task, which might have increased the randomness in their performances relative to if they had been more experienced (e.g., Trafimow & Rice, 2009). There also is a limitation of more theoretical importance. Specifically, we tested only one type of heuristic, whereas there is a large class of potential heuristics that are theoretically available for testing. We hope and expect that future research will remedy this lack by including tests of additional heuristic processes. Conclusion We see the present experiment as a demonstration of a larger PPT issue. Trafimow and Rice (2008) originally developed PPT in the context of morality. They commenced from the standpoint that moral decisions could be made in the context of an absolute standard (e.g., the bible, Kant’s moral imperative, and so on) or a relative one (e.g., the opinion of another person). In the former case,
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
284
The Journal of General Psychology
“success” or “failure” would be defined via agreement or disagreement with the absolute standard, whereas in the latter case, success or failure would be defined via agreement or disagreement with another entity (usually another person). The original PPT equations were developed to be sufficiently flexible to be used either way. In addition, Trafimow and Rice suggested PPT as a general theory, rather than just a theory of morality, though they originally derived their equations in the context of the morality literature. As often is so, a methodology came with the theory, and the methodology involves presenting at least two blocks of trials instead of only one block of trials. From there, PPT equations take over. Because journals differ on whether they favor contributions to theory versus empirical contributions, there has been a tendency to present PPT as a theory in journals that emphasize theoretical contributions, whereas PPT has been presented as a methodology when the focus is on empirical contributions to substantive areas. We believe that the distinction between theory and method is too strong. It is inevitable that theories will imply methodologies. Going the other way, all methodologies involve some kind of underlying theory, though that theory might be well or ill specified. In the present article, we attempted a Goldilocks effect—to hit a happy medium between using PPT from a theoretical versus methodological perspective, and to show that considering PPT simultaneously as both a theory and a methodology facilitates interesting research, while adding applied value as well. AUTHOR NOTES Stephen Rice is an Associate Professor of Psychology at New Mexico State University. He received his Ph.D. in Experimental Psychology from the University of Illinois at Urbana-Champaign in 2006. His research interests include social cognition, human performance and potential performance theory. David Trafimow is a professor of psychology at New Mexico State University, a Fellow of the Association for Psychological Science, and an executive editor of the Journal of General Psychology. He received his PhD in psychology from the University of Illinois at Urbana-Champaign in 1993. His current research interests include attribution, attitudes, cross-cultural research, methodology, and potential performance theory. REFERENCES Ben Zur, H., & Breznitz, S. J. (1981). The effect of time pressure on risky choice behavior. Acta Psychologica, 47, 89–104. B¨ockenholt, U., & Kroeger, K. (1993). The effect of time pressure in multiattribute binary choice tasks. In O. Svenson & A. J. Maule (Eds.), Time pressure and stress in human judgment and decision making (pp. 195–214). New York, NY: Plenum Press. Brunswik, E. (1952). The conceptual framework of psychology: International encyclopedia of unified science. Chicago, IL: University of Chicago Press.
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
Rice & Trafimow
285
De Neys, W. (2006). Dual processing in reasoning—Two systems but one reasoner. Psychological Science, 17, 428–433. Edland, A. C. (1994). Time pressure and the application of decision rules. Scandinavian Journal of Psychology, 35, 281–291. Evans, J. St. B. T. (2007). Hypothetical thinking: Dual process in reasoning and judgement. Hove, UK: Psychology Press. Evans, J. St. B. T. (2008). Dual-processing accounts of reasoning, judgment and social cognition. Annual Review of Psychology, 59, 255–278. Evans, J. St. B. T., Handley, S. J., & Bacon, A. M. (2009). Reasoning under time pressure: A study of causal conditional inference. Experimental Psychology, 56(2), 77–83. Goldstein, D. G., & Gigerenzer, G. (2002). Models of ecological rationality: The recognition heuristic. Psychological Review, 109, 75–90. Hammond, K.R. (1980). Introduction to Brunswikian theory and methods. New Directions for Methodology of Social and Behavioral Science, 3, 1–11. Harreveld, F., Wagenmakers, E., & van der Maas, H. (2007). The effects of time pressure on chess skill: An investigation into fast and slow processes underlying expert performance. Psychological Research, 71, 591–597. Hogarth, R. (1981). Beyond discrete biases: Functional and dysfunctional aspects of judgmental heuristics. Psychological Bulletin, 90(2), 197–217. Kahneman, D., & Frederick, S. (2002). Representativeness revisited: Attribute substitution in intuitive judgement. In T. Gilovich, D. Griffin, & D. Kahneman (Eds.), Heuristics and biases: The psychology of intuitive judgment (pp. 49–81). Cambridge, UK: Cambridge University Press. Kaplan, M. F., Wanshula, L. T., & Zanna, M. P. (1993). Time pressure and information integration in social judgement. In O. Svenson & A. J. Maule, Time pressure and stress in human judgement and decision making (pp. 255–267). New York, NY: Plenum. Kelley, H. H., & Michela, J. L. (1980). Attribution theory and research. Annual Review of Psychology, 31, 457–501. Kerstholt, J. H. (1995). Decision making in a dynamic situation: The effect of false alarms and time pressure. Journal of Behavioral Decision Making, 8, 181–200. Maule, A. J. (1994). A componential investigation of the relation between structural modelling and cognitive accounts of human judgement. Acta Psychologica, 87, 199–216. Maule, A. J., & Edland, A. C. (1997). The effects of time pressure on judgment and decision making. In R. Ranyard, W. R. Crozier & O. Svenson, Decision making: cognitive models and explanation (pp. 189–204). London : Routledge & Kegan Paul. Maule, A., Hockey, G., & Bdzola, L. (2000). Effects of time-pressure on decision-making under uncertainty: Changes in affective state and information processing strategy. Acta Psychological, 104, 283–301. Orvis, B. R., Cunningham, J. D., & Kelley, H. H. (1975). A closer examination of causal inference: The roles of consensus, distinctiveness, and consistency information. Journal of Personality and Social Psychology, 32(4), 605–616. Payne, J. W., Bettman, J. R., & Johnson, E. J. (1988). Adaptive strategy selection in decision making. Journal of Experimental Psychology: Learning, Memory, & Cognition, 14, 534–552. Payne, J. W., Bettman, J. R., & Johnson, E. J. (1993). The adaptive decision maker. Cambridge, UK: Cambridge University Press. Rice, S., Geels, K., Trafimow, D. & Hackett, H. (2011). Our students suffer from both lack of knowledge and consistency: A PPT analysis of test-taking. US-China Education Review, 1(6), 845–855. Rice, S., & Keller, D. (2009). Automation reliance under time pressure. Cognitive Technology, 14(1), 36–44.
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
286
The Journal of General Psychology
Rice, S., Trafimow, D., Clayton, K., & Hunt, G. (2008). Impact of the contrast effect on trust ratings and behavior with automated systems. Cognitive Technology Journal, 13(2), 30–41. Rice, S., Trafimow, D. & Hunt, G. (2010). Using PPT to analyze sub-optimal humanautomation performance. Journal of General Psychology, 137(03), pp. 310–329. Rice, S., Trafimow, D., Keller, D., Hunt, G. & Geels, K. (2011). Using PPT to correct for inconsistency in a speeded task. The Journal of General Psychology, 138(1), 12–34. Rieskamp, J., & Hoffrage, U. (2008). Inferences under time pressure: How opportunity costs affect strategy selection. acta psychological, 127, 258–276. Sloman, S. A. (1996). The empirical case for two systems of reasoning. Psychological Bulletin, 119, 3–22. Stanovich, K. E. (1999). Who is rational? Studies of individual differences in reasoning. Mahway, NJ: Lawrence Elrbaum Associates. Svenson, O., & Benson, L. (1993). Framing and time pressure in decision making. In O. Svenson & A. J. Maule, Time pressure and stress in human judgement and decision making (pp. 133–144). New York, NY: Plenum. Svenson, O., & Maule A. J. (1993). Time pressure and stress in human judgement and decision making. New York, NY: Plenum. Todd, P. M., & Gigerenzer, G. (2000). Pr´ecis of Simple heuristics that make us smart. Behavioral and Brain Sciences, 23, 727–780. Trafimow, D., MacDonald, J., & Rice, S. (2012). Using PPT to account for randomness in perception. Attention, Perception and Psychophysics, 74, 1355–1365. Trafimow, D., & Rice, S. (2008). Potential Performance Theory: A general theory of task performance applied to morality. Psychological Review, 115(2), 447–462. Trafimow, D., & Rice, S. (2009). Potential Performance Theory (PPT): Describing a methodology for analyzing task performance. Behavior Research Methods, 41(2), 359– 371. Trafimow, D., & Rice, S. (2011). Using a sharp instrument to parse apart strategy and consistency: An evaluation of PPT and its assumptions. Journal of General Psychology, 138(3), 169–184. Wright, P. L. (1974). The harassed decision maker: time pressures, distractions and the use of evidence. Journal of Applied Psychology, 59, 555–561.
Original manuscript received January 12, 2012 Final version accepted June 18, 2012 APPENDIX A The PPT strategy is as follows. In a dichotomous task, there are two possible responses to each trial, and each possible response might be correct or incorrect. Thus, for each participant, it is possible to create a 2 (option chosen) by 2 (option correct) frequency table. Table 1 illustrates the frequency table with the four cells. The participant can choose the first option and be correct (cell a) or incorrect (cell c), or the participant can choose the second option and be correct (cell d) or incorrect (cell c). There are also margin frequencies constructed as follows. The first row frequency is the sum of cells a and b whereas the second row frequency is the sum of cells c and d (r1 and r2 , respectively). The first column frequency is the sum of cells a and c whereas the second column frequency is the sum of
Rice & Trafimow
287
cells b and d (c1 and c2 , respectively). Each participant’s frequency table can be converted to a correlation coefficient via Equation 1.
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
r=
|ad − bc| |ad − bc| = (a + b)(c + d)(a + c)(b + d) r1 r2 c1 c2
(1)
The second step, assuming that there were at least two blocks of trials in the experiment, is to obtain the consistency coefficient. This is simply the correlation, for each participant, across blocks of trials. The third step is to use the famous correction formula that was originally derived by Spearman (1904), is a standard derivation from classical true score theory, but also can be derived from more modern theories (see Allen & Yen, 1979; Cohen & Swerdlik, 1999; Crocker & Algina, 1986; Gulliksen, 1987; Lord & Novick, 1968 for reviews). The formula is provided in Equation 2, where R is the best estimate of the correlation coefficient that would be obtained in the absence of randomness and rxx is the consistency coefficient across blocks of trials that was computed on Step 2. R=√
r rxx
(2)
It is worth noting that Equation 2 is a special case of a more general correction formula. In the general case, there are two variables being correlated, and so there is a consistency coefficient associated with each variable. However, we are interested in only one variable; this is participant’s responding to the task trials. Therefore, for our present purposes, the simplified formula is sufficient. But it is possible to imagine a task where there is no correct answer, and so the criterion for “success” is agreement with another person. In this case, there would be a consistency coefficient for each person because each person would be capable of responding with some degree of randomness. As Trafimow and Rice (2008) showed, when there is no correct answer, the more general correction formula must be used, where each person’s responding is treated as a separate variable. Given that a corrected correlation coefficient has been obtained, the next step is to convert it back into a frequency table. But the new frequency table is not a table of observed frequencies, but rather of potential frequencies, which are the best estimates of the frequencies that would be obtained in the absence of randomness (i.e., with perfect consistency). These can be obtained via Equations 3–6 below. Equations 3–6 require fixing the margin frequencies at the obtained levels, similar to a Chi Square test or a Fisher’s Exact test, and using R to estimate the potential cell frequencies. Trafimow and Rice (2008) provide derivations of Equations 3–6 and Trafimow and Rice (2009) provide empirical tests that strongly validate the assumptions. Finally, Trafimow and Rice (2011) provide further validation in the form of computer simulations. Consistent with previous research, we use upper
288
The Journal of General Psychology
case letters (A, B, C, D, R1 , R2 , C1 , C2 ) to refer to the cell and margin frequencies in Equations 3–6 below. A=
Downloaded by [The University of British Columbia] at 13:31 14 October 2014
B= C= D=
√ R R1 R2 C 1 C 2 + C 1 R1 (R1 + R2 ) √ R1 (R1 + R2 ) − (R R1 R2 C1 C2 + C1 R1 ) (R1 + R2 ) √ C 1 R 2 − R R1 R 2 C 1 C 2 (R1 + R2 ) √ C2 (R1 + R2 ) − R1 (R1 + R2 ) − R R1 R2 C1 C2 + C1 R1 (R1 + R2 )
(3) (4) (5) (6)
The foregoing steps for using PPT can be summarized as follows, providing that the participants have responded to two blocks of matched trials. First, the researcher uses Equation 1 to convert each participant’s observed performance table into a correlation coefficient. Second, the researcher obtains a consistency coefficient for each participant. Third, the researcher uses Equation 2 to obtain a corrected or potential correlation coefficient. Finally, the researcher uses Equations 3–6 to estimate the cell frequencies that would be obtained without randomness (i.e., potential cell frequencies). Once the potential cell frequencies have been obtained, each participant’s potential performance, or the performance level each person would have if the responses had been perfectly consistent across blocks can be calculated via Equation 7 below. potential performance =
A+D (A + B + C + D)
(7)
