Commentary/Newell & Shanks: Unconscious influences on decision making policy) - often by assigning a points value to indicate each cue's weight. Conceptuahsing self-insight research as "double model recovery" emphasises some limitations in previous work that bear on N&S's relevance and sensitivity criteria for the adequate assessment of awareness (target article. Table 1), and points to avenues for developing better methods for assessing selF-insiglit. First, this double-model-recovery framework highlights a critical question: why-as seems standard - instantiate the implicit policy (from statistical model recovery) as the "correct" model, and therefore assume that any discrepancy between implicit and explicit policies represents the judge's failure to recover die "true" model? Model-recovery exercises in cognitive science usually consider multiple "families" of candidate model. Typically, lens model research considers a single family of models: compensatory linear rules that integrate afixednumber of cues - though it does consider different family members, wliicli differ according to the number of cues used. Altemative families of non-Hnear (configurai) or non-compensatoiy judgment models are less frequently considered in the multiple-cue judgment literature, even though several alternatives can be modelled, such as judgments made according to the similarity of each case to a prototype, judgments made following a non-exliaustive lexicographic search through cues, and judgments where cues are selected probabilistically and tlierefore different cues are used for different cases. In contrast, some research on multi-attribute choice does consider different families of models: for instance, comparing altemative models reflecting whether a compensatory or lexicographic decision rule is being applied (e.g.. Broder 2003). Additionally, this research on recovering choice processes liiglJights that different models reflecting quite distiuct processes often fit the data similarly well. Therefore, even when a compensatory linear model fits the data, die judge may nonetheless have followed a quite different process in milking his or her judgments. In such cases, any elicitation procedure that presupposes the compensatory hnear combination of a fixed number of cues fails N&S's relevance criterion because the behaviours being probed are not those that drove the judgment. This is liable to generate a poor match between tlie implicit and explicit policies. T^hus, by following a restricted approach to modelling the judge to dictate the constraints of that judge's self-description, we create an insensitive assessment of awareness and may misattribute poor modelling as poor self-insight. Second, a double-model-recovery framework emphasises the potential for mis-recovery of the original judgment process by either recoveiy technique (statistical or human). As many others have done, I have pitted human judges against statistical mies in multiple-cue judgments and - as is typical - have found that "statistical judges" outperform their hum.ui competitors (Dawes et al. 1989). However, in one investigation (see Rakow et al. 2003), our statistical judge showed the same apparent lack of self-insight as its human counteiparts. A seven-cue predictive model derived using logistic regression generated predicted probabilities (that an applicant would be offered a place at a given university) for a series of cases, each defined by multiple cues. Human judges also provided die same set of judgments. Using the same linear regression analysis applied to the human participants, the implicit policy for the statistical judge declared only five cues to be used reliably (i.e., significant). Thus the statistical judge showed the typical pattem of limited self-insight that human judges display, appaiently overestimating the number of cues that it used! Thus, just as assessments of awareness may fail N&S's sensitivity criterion, so too, insufficiently sensitive model recoveiy via linear regression could contribute to an inappropriate conclusion of "limited self-insight" (for a technical discussion of this problem, see Beckstead 2007). Third, we can consider strategies for assisting human judges in recovering (describing) tlieir judgment policies, which may, also, influence the candidate models for the statistical element of die double recovery exercise. In a recent study, we asked mental health practitioners to self-identify with descriptions of altemative
BEHAVIORAL AND BRAIN SCIENCES (2014) 37:1
families of judgment models - (non-)compensatory and (non-) exliaustive-which drew on analogies to common decision aids such as "balance sheets" and "trouble-shooting guides." Many of our assessors selected those options that implied contingent information search or non-compensatory infomiation integration in their own (triage) judgments. Thus, if required to describe themselves in tenns of a compensatory model always using a fixed number of cues (as per most self-insight research - though arguably failing the relevance criterion), inevitably some participants were forced to misrepresent their policy. It would tlierefore be unsurprising if judges displayed "poor self-insight." Much work has been done on altemative strategies for eliciting the subjective weights for compensatory linear judgment policies (e.g.. Cook & Stewart 1975). However, we need improved (i.e., relevant and sensitive) elicitation methods that allow for a wider range of information search and integration processes to be identified when judges are asked to describe their judgment policies. Fair assessment of a judge's self-insight requires that hotli the statistical exercise of deriving the implicit judgment policy and the elicitation exercise whereby the judge describes liis or her own judgments allow-as far as possible - recovery of tlie processes by which the original judgments were made.
What we (don't) know about what we know doi: 10.1017/S0140525X13000836 Shiomi Sher^ and Piotr Winkielirian"''' ^Department of Psychology, Pomona College, Claremont, CA 91711; ''Department of Psychology, University of California, San Diego, La Jolla, CA 92093-0109; "University of Sociai Sciences and Humanities, 03-815 Warsaw, Poland. [email protected] [email protected]