This article was downloaded by: [New York University] On: 07 October 2014, At: 12:53 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

The Journal of General Psychology Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/vgen20

Color Selectivity of the Spatial Congruency Effect: Evidence from the Focused Attention Paradigm a

Elena Makovac & Walter Gerbino

a

a

University of Trieste Published online: 20 Dec 2013.

To cite this article: Elena Makovac & Walter Gerbino (2014) Color Selectivity of the Spatial Congruency Effect: Evidence from the Focused Attention Paradigm, The Journal of General Psychology, 141:1, 18-34, DOI: 10.1080/00221309.2013.837025 To link to this article: http://dx.doi.org/10.1080/00221309.2013.837025

PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content.

Downloaded by [New York University] at 12:53 07 October 2014

This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

The Journal of General Psychology, 2014, 141(1), 18–34 C 2014 Taylor & Francis Group, LLC Copyright 

Downloaded by [New York University] at 12:53 07 October 2014

Color Selectivity of the Spatial Congruency Effect: Evidence from the Focused Attention Paradigm ELENA MAKOVAC WALTER GERBINO University of Trieste

ABSTRACT. The multisensory response enhancement (MRE), occurring when the response to a visual target integrated with a spatially congruent sound is stronger than the response to the visual target alone, is believed to be mediated by the superior colliculus (SC) (Stein & Meredith, 1993). Here, we used a focused attention paradigm to show that the spatial congruency effect occurs with red (SC-effective) but not blue (SC-ineffective) visual stimuli, when presented with spatially congruent sounds. To isolate the chromatic component of SC-ineffective targets and to demonstrate the selectivity of the spatial congruency effect we used the random luminance modulation technique (Experiment 1) and the tritanopic technique (Experiment 2). Our results indicate that the spatial congruency effect does not require the distribution of attention over different sensory modalities and provide correlational evidence that the SC mediates the effect. Keywords: focused attention paradigm, multisensory response enhancement, random luminance modulation technique, superior colliculus, tritanopic technique

MOST ORGANISMS MONITOR THEIR ENVIRONMENTS through several senses and combine information from multiple sources to build coherent and unified representations of objects and space. This ability, called multisensory integration (MI), has been broadly investigated in humans and other animals (Stein, 2012; Stein & Meredith, 1993; Welch & Warren, 1986), and may be the rule in ordinary perception and attention, constituting a pervasive aspect of the binding problem (Brockmole & Franconeri, 2009). In perceptual integration, visual and auditory information can be combined to yield an integrated percept different from the one obtained on the basis of unisensory information only (McGurk & MacDonald, 1976). In attentional orienting, an uninformative sound activating an exogenous (bottom-up) shift of attention towards a certain location can enhance the Address correspondence to Elena Makovac, Department of Life Sciences, Psychology Unit “Gaetano Kanizsa,” University of Trieste, via Weiss 21, 34128, Trieste, Italy; elena. [email protected] (e-mail). 18

Downloaded by [New York University] at 12:53 07 October 2014

Makovac & Gerbino

19

processing of a visual target subsequently presented in the same location, showing that spatial attention is at least partially supramodal (P´erez-Bellido, Soto-Faraco, & L´opez-Moliner, 2013; for a review, see Spence & McDonald, 2004). An important integration effect is the multisensory response enhancement (MRE), occurring when the response to spatially and temporally close crossmodal stimuli exceeds the response to the most effective unisensory stimulus; whereas spatially and temporally disparate stimuli produce either depression or no change in the neural response (Calvert & Thesen, 2004; Stein & Meredith, 1993). Multisensory integration follows three rules (Bolognini & Ladavas, 2005; Spence & Driver, 1997, 2004; Stein, 2012; Stein, Meredith, Huneycutt, & McDade, 1989) that, with reference to audio-visual events, can be described as follows. Spatial Rule Acoustic and visual stimuli originating from the same spatial position fall within the excitatory receptive fields of the same bimodal neurons and enhance their responses (Meredith & Stein, 1996). Temporal Rule Maximal MI is achieved when periods of peak activity of the unisensory discharge train overlap. This occurs when stimuli are presented simultaneously, although small temporal discrepancies between stimuli might fall within a temporal window for MI (Kadunce, Vaughan, Wallace, Benedek, & Stein, 1997; Meredith & Stein, 1996). Inverse Effectiveness Rule The integration between acoustic and visual stimuli is stronger when the stimuli presented in isolation evoke weak neural (Perrault, Vaughan, Stein, & Wallace, 2003) or behavioral (Bolognini, Frassinetti, Serino, & Ladavas, 2005) responses. Even when acoustic and visual stimuli evoke no responses in the unisensory condition their combination becomes effective and produces a surprisingly vigorous response. When acoustic and visual stimuli are weak and in close spatiotemporal proximity, the MRE is superadditive. There is a broad agreement on the idea that the rules of MI apply to both neural and behavioral responses (Maravita, Bolognini, Bricolo, Marzi, & Savazzi, 2008; Stein, Huneycutt, & Meredith, 1988; Alais, Newell, & Mamassian, 2010). The behavioral consequences of MI include the speeding up of the response (Todd, 1912) and the improvement of stimulus detection revealed by both manual responses and saccadic movements (Diederich & Colonius, 2004). The MRE is stronger for temporally and spatially overlapping stimuli (Colonius & Diederich, 2012a, 2012b; Diederich & Colonius, 2004; Teder-S¨alej¨arvi et al., 2005), and for participants exhibiting the weakest responses in the unisensory conditions, consistently with the inverse effectiveness rule (Holmes, 2007, 2009). It is widely held that the superior colliculus (SC) plays a central role in MI and in the generation of spatial orienting responses. Evidence comes from animal lesions and, most compellingly, from single-unit recording in the cat (Burnett, Stein,

Downloaded by [New York University] at 12:53 07 October 2014

20

The Journal of General Psychology

Perrault, & Wallace, 2007; Stein & Meredith, 1993). The SC is believed to play an important role in sensorimotor integration and orienting behavior. Traditionally, it is divided into superficial layers (containing mostly visual neurons) and deep layers (containing multisensory and premotor neurons; Tardif, Delacuisine, Probst, & Clarke, 2005). Superficial layers are connected mainly with structures involved in visual perception, whereas neurons of the deep layers are mostly motor-related (Huerta & Harting, 1984). The multisensory stimulation enhances the firing rate of multisensory neurons and consequently influences the level of activation of premotor neurons in the SC, closely linked to the initiation of a movement. The premotor neurons, which are the main connection between the sensory enhancement and the motor behavior mediated by the SC, are facilitated when activated by audio-visual stimuli (Peck, 1987). To test the role of the SC in MI one can use visual stimuli of different spectral content. Importantly for our investigation, electrophysiological studies have reported that the retino-collicular pathway does not contain projections from short-wavelength cones (S-cones; Calkins, 2001; de Monasterio, 1978; Marrocco & Li, 1977; Schiller & Malpeli, 1977; Stockman, MacLeod, & DePriest, 1991). The S-cones are thought primarily to mediate color perception, and their signals are carried by morphologically distinct types of ganglion cells, which project to the koniocellular layers of the lateral geniculate nucleus and thence to layers 2 and 3 of the striate cortex (Dacey & Lee, 1994; Hendry & Reid, 2000). On the contrary, the magnocellular pathway constitutes the main input to the retinocollicular projection, and receives its inputs mainly from the long-wavelength (L) and medium-wavelength (M) cones. Since the magnocellular pathway is not color opponent and receives little input from S-cones, the initial sensory activity in SC is color blind (Marrocco & Li, 1977; Ottes, Van Gisbergen, & Eggermont, 1987). Chromatic changes visible only to S-cones should be invisible to the retinocollicular pathway (Sumner, 2006; Sumner, Adamjee, & Mollon, 2002; Sumner, Nachev, Vora, Husain, & Kennard, 2004). Therefore, the role of the SC in different tasks could be tested by selectively activating the S-cones. S-cones may still provide some input to the SC and to the luminance pathway, as suggested by some researchers (Calkins, 2001; Stockman, MacLeod, & DePriest, 1991). But such input is small and can be masked by luminance noise (Barbur, Harlow, & Plant, 1994; Birch, Barbur, & Harlow, 1992; Mollon, 1982). Given these assumptions, several authors tried to manipulate the chromaticity and luminance of the stimuli, in order to indirectly investigate the role of the SC. For example, Corballis (1998) measured the redundancy gain in split-brain subjects and found that it was greatly diminished when the stimuli were equiluminant with the background (i.e., undetected by the SC). Leo, Bertini, di Pellegrino, & Ladavas (2008) asked healthy observers to detect auditory targets, visual targets (either red, invisible to S-cones, or purple, visible to S-cones), as well as spatially congruent/incongruent audio-visual targets. A spatial congruency effect occurred with red but not purple targets; the MRE effect was superadditive only for red targets in the congruent

Downloaded by [New York University] at 12:53 07 October 2014

Makovac & Gerbino

21

condition, as expected if the spatial rule were mediated by the activation of the SC. It is worth pointing out that the studies evaluating the role of the SC in the MRE and the spatial congruency effect utilized a redundant target paradigm, in which participants should divide their attention between auditory and visual modalities. Whether the results can be replicated within a focused attention paradigm (when attention can be focused only on one modality, since the second is task-irrelevant) is still uncertain and will be addressed in the present study. In fact, the role of attention in MI is controversial (Bertelson & Radeau, 1981; McDonald Teder-S¨alej¨arvi, Di Russo, & Hillyard, 2003; Shore & Simic, 2005; Talsma & Woldorff, 2005). According to Alais, Newell, & Mamassian (2005) Audio-visual integration is not automatic and can be more or less effective when tasks are performed in multiple modalities. Mozolic et al. (2008) utilized a cued discrimination task and provided evidence that the MRE occurs only when participants divide their attention between auditory and visual modalities. In contrast, evidence from studies of spatial attention and the ventriloquist effect (i.e., the illusory spatial misallocation of a sound toward a spatially disparate visual stimulus) indicates that the ability to localize a sound depends on automatic sensory processes, not influenced by directing attention to different visual locations (Bertelson, Vroomen, de Gelder, & Driver, 2000; Vroomen, Bertelson, & de Gelder, 2001a). However, one can expect that the spatial congruency effect in a simple detection task is highly automatic, and as such does not require the distribution of attention over auditory and visual modalities. To support such a hypothesis, we run two experiments with the aim of replicating the results obtained in the above-described studies with a redundant target paradigm, but using a focused attention paradigm in which participants should explicitly focus on visual targets, ignoring the auditory modality. SC-efficient (red, mediated mainly by L-cones) vs. SC-inefficient (blue, mediated mainly by S-cones) targets were compared, minimizing the luminance contrast of SC-inefficient targets relative to the background, by means of the random luminance modulation technique (Barbur et al., 1994; Birch et al., 1992; Mollon, 1982) in Experiment 1, and the tritanopic technique (Cavanagh, MacLeod, & Anstis, 1987) in Experiment 2. A stronger MRE for spatially congruent vs. incongruent stimuli was obtained only with SC-efficient targets. No spatial rule was evident with SC-inefficient stimuli. Moreover, the MRE in the spatially congruent condition was stronger for SC-efficient that SCinefficient targets. Admittedly, results obtained in a focused attention paradigm cannot provide evidence of superadditivity (given the absence of response times to auditory targets alone). Therefore, if MI is operationally defined by a superadditive MRE, no direct conclusion about MI can be drawn. However, the possible presence of a spatial congruency effect can be considered as an indirect evidence of MI, under the assumption that the spatial rule is an intrinsic property of MI.

22

The Journal of General Psychology

Downloaded by [New York University] at 12:53 07 October 2014

Experiment 1 The experiment was based on the comparison of go responses in three sets of positive trials. Unisensory visual (US) trials: the visual (red or blue) target was presented alone, on the left/right of the fixation cross; Multisensory congruent (MSC) trials: the auditory stimulus and the visual (red or blue) target were presented simultaneously in the same spatial position, on the left/right of the fixation cross; Multisensory incongruent (MSI) trials: the auditory stimulus and the visual (red or blue) target were presented simultaneously but in opposite positions (A-left and V-right, or vice versa). To make responses contingent on the presentation of visual targets, as implied by the focused attention paradigm, we added a set of Unisensory auditory catch trials, in which the auditory stimulus was presented alone, from either the left or right loudspeaker, but the participant should not respond. We expected participants to be equally accurate in all experimental conditions, if targets of different colors had equal brightness. However, a combination of effects were expected on RTs: no difference between red vs. blue targets in US trials; significant (i.e., larger than zero) MRE effects in both congruent and incongruent conditions; a larger MRE effect for red targets in the congruent condition; no difference between red vs. blue targets in the incongruent condition. Method Participants Eleven healthy right-handed observers (age range 19-25 years, 9 females) took part in the experiment. All had normal hearing and normal or corrected-to-normal visual acuity. Participants were students, na¨ıve as to the purpose of the experiment, who received course credit for their participation. Every participant gave his/her informed consent individually, prior to the beginning of the experiment. Stimuli and Procedure Auditory and visual stimuli were generated by a PC equipped with standard acoustic and graphic software. SuperLab 4.8 (Cedrus Corporation, San Pedro, CA) was utilized to present stimuli and collect responses. Visual stimuli were displayed on Sony Trinitron (Sony Corporation, Japan) CPD-G200P CRT 17 inch monitor, set at 50% brightness, 90% contrast, 1280 × 768 pixel resolution, 60 Hz. CIE1931 coordinates of monitor primaries were as follows: R (x = .623, y = .339); G (x = .284, y = .589); B (x = .152, y = .067). The participant was seated at a distance of 57 cm from the center of the screen and required to hold constant fixation on a central white cross (70 cd/m2) against a black background (5 cd/m2). The task consisted of pressing the central button of the keyboard with the index finger of the dominant hand as quickly as

Downloaded by [New York University] at 12:53 07 October 2014

Makovac & Gerbino

23

FIGURE 1. Schematic timetable of a positive trial in Experiment 1 (not in scale). Each trial was initiated by the onset of a central fixation cross. The visual target was shown after an interval varying between 1367 and 2633 ms, including a blank field and the display of two grey-cell matrices. The target was a red or blue square, corresponding to the central cell of one of two matrices. Loudspeakers were placed just above the matrices. In multisensory positive trials an auditory stimulus was presented simultaneously with the visual target, in the same or opposite position. Negative (catch) trials were characterized by the absence of the visual target and presence of the auditory stimulus.

possible to respond to any visual target (either red or blue, in either unisensory or multisensory conditions) briefly shown at the left/right of the fixation cross with a 11.5◦ eccentricity, and suppressing any overt response to the auditory stimulus alone. Target stimuli were 1 × 1◦ red (long-wavelength) or blue (short-wavelength) squares, displayed for 67 ms as the central cell of a 3 × 3 matrix. Each cell of the matrix subtended a 1 × 1◦ visual angle. Figure 1 shows the sequence of events in a trial: the fixation point lasted 500 ms, followed by a blank interval with a random duration of 133–1132 ms; then the two matrices were displayed for 733–1000 ms, with the luminances of the peripheral cells refreshed every 67 ms by randomly picking up 8 luminance values in the 6.4–10.8 cd/m2 range, with the only exception of the last 67 ms, in which the 8 luminances of the peripheral cells were fixed (average luminance = 8.6 cd/m2); finally, the target stimulus (red or blue) was displayed in the central cell. The random luminance modulation technique allowed us to isolate the chromatic contribution of blue targets from their associated luminance component (Barbur, 2004; Birch et al., 1992; Mollon, 1982).

Downloaded by [New York University] at 12:53 07 October 2014

24

The Journal of General Psychology

Multisensory positive trials were characterized by the simultaneous presentation of an auditory stimulus and the visual target, in the same position or controlaterally to each other. Participants were presented also with negative (catch) trials, in which only the random luminance matrices were displayed, without any visual target or auditory stimulus. The auditory stimulus was a white noise burst of 60 dB emitted for 67 ms by one of two piezoelectric loudspeakers (4 W) located 5 cm above the positions of the matrices, where the visual target might appear. After 20 min of dark adaptation the participant entered a 38-trial training session in which three criteria should be reached: a maximum of two RTs over 500 ms, a maximum of two RTs under 120 ms, and a maximum of two errors (either misses or false alarms). If the participant failed to meet any of the three criteria, the training session was repeated. The experimental session included 416 trials (104 for each of the 4 conditions). To generate red and blue targets of equal brightness we used the minimum motion technique (Anstis & Cavanagh, 1983). Four frames (each containing a row of eight adjacent 1 × 1◦ squares) were exposed in a repetitive sequence: frames 1 and 3 contained opposite-phase luminance gratings composed of light/dark green squares (18.5 and 5.5 cd/m2), while frames 2 and 4 contained gratings composed of red/blue squares with a quarter-cycle phase shift. When red and blue squares had different brightness, motion was perceived towards either the left or right according to the spatiotemporal proximity of homologous border contrasts. The luminance of blue squares was set to 8.6 cd/m2. The individual equal-brightness point to be utilized in red-target trials was computed as the average of three adjustments obtained in a preliminary session in which the participant asked the experimenter to adjust the luminance of red squares until sideways motion was substituted by flicker. The luminances of red targets resulting from the application of the minimum motion technique ranged between 7.7 and 12.0 cd/m2. Importantly, since the point of equal brightness changes with eccentricity, the minimum motion measurements were performed with peripheral stimuli, separately for left and right sides. Results and Discussion Figure 2A illustrates the pattern of Miss percentages and RTs of Hits in the 6 conditions of the Stimulation × Color design. The average False Alarm percentage in catch trials was 1.9%. A preliminary test of the distribution of Misses, run on transformed data [x = arcsin(pMiss )0.5], showed that 2 out of 6 mean Miss percentages differed from zero (being associated to t(10) values larger than 3.28, correspondent to the critical value of p = .008, one-tailed, chosen after the Bonferroni correction). The two mean Miss percentages significantly larger than zero were MissMSC red = 3.15% and MissMSI red = 5.42% (t(10) = 4.20 and 4.34, p = .002 and .001, respectively). A 2-way ANOVA on transformed data revealed

Downloaded by [New York University] at 12:53 07 October 2014

Makovac & Gerbino

FIGURE 2. Panel A shows mean Miss percentages and RTs for correct responses (± 1 s.e.m.) in the three sets of positive trials of Experiment 1, for red and blue targets. In unisensory (US) trials the target color did not affect RTs and errors. Relative to US trials, RTs were faster in both multisensory congruent (MSC) and incongruent (MSI) trials, while the number of errors increased significantly in MS trials for red but not blue targets. Panel B shows the distribution of MRE amounts, indicating that performance was better in MSC than MSI trials with red but not blue targets.

25

Downloaded by [New York University] at 12:53 07 October 2014

26

The Journal of General Psychology

a marginal main effect of Stimulation (F(2,20) = 3.30, p = .06), no main effect of Color (F(1,10) = 1.31, p = .28), and no 2-way interaction (F(2,20) = 2.64, p = .10). This pattern of Miss percentages suggests the possibility that the irrelevant sound, in combination with the random luminance modulation technique, acted as a distractor, making visual targets less detectable in multisensory trials. As regards the expected effects on RTs of Hits, a planned-comparison analysis showed that the spatial congruency of the acoustic source improved the detection of red (RTMSC red vs. MSI red = 311 vs. 323 ms: F(2,20) = 10.06, p < .01) but not blue (RTMSC blue vs. MSI blue = 317 vs. 319 ms: F < 1) targets.1 Furthermore, in US trials no significant difference was found between response times for red and blue targets (RTUS red vs. blue = 348 vs. 348 ms: t(10) = .13), suggesting that the minimum motion technique was an effective means of matching the effectiveness of red and blue targets. This pattern is consistent with the results of the 3 × 2 within-subjects ANOVA on RTs, which included a main effect of Stimulation (F(2,20) = 48.20, p < .01), no main effect of Color (F < 1), and no Stimulation × Color interaction (F(2,20) = 2.53, p = .11). The proportional amount of MRE was calculated in relation to the unisensory baseline for each participant separately, as a function of target color and spatial congruency of the acoustic source, according the following formula by Stein & Meredith (1993), adapted to the focused attention paradigm: 100·(RTUS RTMS )/RTUS . All four average MRE values were positive, confirming that multisensory stimulation always produced a speed gain, for both red and blue targets and in both multisensory—spatially congruent and incongruent—conditions (all t(10) values larger than 5, one-tailed, p < .01). A planned-comparison analysis confirmed that the MRE was stronger in the congruent than incongruent condition for red (10.7 vs. 7.2%: F(1,10) = 8.02, p < .02) but not blue (8.8 vs. 8.5%: F < 1) targets. This pattern was consistent with the results of the 2 × 2 ANOVA on MRE values, showing a marginal main effect of Congruency (F(1,10) = 4.13, p = .07), no effect of Color (F < 1), and a marginal Congruency × Color interaction (F(1,10) = 3.67, p = .08). To summarize the results of Experiment 1, the RT pattern followed all our expectations. In US trials responses to red and blue targets were equally fast. The expected response speed enhancement was obtained in both multisensory conditions (congruent and incongruent), but it was larger in MSC than MSI trials only for red targets. Furthermore, in MSC trials responses were faster for red than blue targets, as expected if MI follows the spatial rule, consistent with neural evidence on SC functions. Note that the pattern of improvements in response speed was not mirrored by a similar pattern of accuracy data. Misses were approximately the same across conditions and possibly increased (not decreased) in multisensory trials. Let us anticipate that such a tendency will not be replicated in Experiment 2, suggesting that the possible distracting role of the sound in multisensory trials of Experiment

Makovac & Gerbino

27

1 could be a by-product of the combination of the sound and the random luminance modulation technique.

Downloaded by [New York University] at 12:53 07 October 2014

Experiment 2 Experiment 2 was conceived as a conceptual replication of Experiment 1, to provide converging evidence on color selectivity of the spatial congruency effect and, consequently, on SC-mediated MI. To decrease the responsiveness of the luminance pathway and to differentiate the SC-efficiency of red/blue targets, as an alternative to the random luminance modulation technique of Experiment 1 we used the tritanopic technique (Cavanagh et al., 1987). The basic idea behind the tritanopic technique is the following. The input to the luminance pathway mainly comes from L- and M-cones (Drum, 1983; Eisner & MacLeod, 1980; Lee & Stromeyer, 1989). On the other hand, the blue/yellow chromatic pathway is strongly driven by S-cones. Thus, an intense yellow adapting field shown before a blue increment can isolate the response of the S-cones very effectively (Stiles, 1959; Wald, 1966). The yellow field drives the responses of Land M-cones to very high levels and elevates their contrast thresholds, reducing the efficiency of the blue target and making it virtually invisible to the luminance pathway. This is called a tritanopic stimulus because it activates only the S-cones and, therefore, would be invisible to a tritanope (an individual lacking S-cones). Apart from the elimination of the randomly modulated grey-cell matrices and the introduction of the yellow field, the method, expectations, and transformations used in statistical analyses were the same as in Experiment 1. Method Participants Twelve observers participated in Experiment 2. Nine had already participated in Experiment 1, while the other three were new to the task (20, 21, and 24 year old, one male and two females, all right-handed with normal hearing and normal or corrected-to-normal visual acuity). Stimuli and Procedure With respect to Experiment 1 there were three main changes in the visual stimuli. The background was yellow (RGB values: 200, 160, 0; 47.1 cd/m2). The grey-cell matrices were substituted by black outline squares (outline thickness = 3 pixels; matrix side = 3◦ ). RGB values and luminances of red and blue targets were as follows: red (255, 160, 0; 55.5 cd/m2); blue (200, 160, 200; 55.7 cd/m2). The temporal sequence of events included in every trial was the same as in previous experiments. A black (5.2 cd/m2) fixation cross replaced the white cross used in Experiment 1.

28

The Journal of General Psychology

Downloaded by [New York University] at 12:53 07 October 2014

Results Figure 3A illustrates the pattern of Miss percentages and RTs of Hits in the 6 conditions of the Stimulation × Color design. The average False Alarm percentage in catch trials was 1.8%. A preliminary test of the distribution of Misses showed that the three Miss percentages for red targets differed from zero (t(11) values larger than 3.21, correspondent to the critical value of p = .008, one-tailed, chosen after the Bonferroni correction), while Miss percentages for blue targets did not. A 2-way ANOVA on transformed Miss percentages showed that neither the main effect of Stimulation (F(2,22) = 2.47, p = .11) nor the 2-way interaction (F < 1) were significant, whether the main effect of Color was significant (F(1,11) = 13.50, p < .01), attributable to a larger number of missed red targets in all conditions (MissUS red vs. blue: t(11) = 2.37, p < .05; MissMSC red vs. blue: t(11) = 3.92, p < .01; MissMSI red vs. blue: t(11) = 2.76, p < .05; two-tailed). The distribution of RTs in US trials go responses were significantly slower for red than blue targets (RTUS red vs. blue = 358 vs. 338 ms: t(11) = 7.26, two-tailed, p < .01). The 3 × 2 within-subjects ANOVA showed a main effect of Stimulation (F(2,22) = 38.18, p < .01), a main effect of Color (F(1,11) = 77.44, p < .01), and a Stimulation × Color interaction (F(2,22) = 2.22, p = .13). Figure 3B shows the amounts of MRE in the four conditions of the Congruency × Color design. As in Experiment 1, all average MRE values were positive, indicating that the multisensory stimulation always produced a speed gain, for red and blue targets, in both spatial congruency conditions (all t(11) values larger than 5, one-tailed, p < .01). A planned-comparison analysis showed that the MRE was larger in congruent than incongruent trials for red (7.7 vs. 4.5 %: F(1,11) = 11.60, p < .01) but not blue (5.7 vs. 5.4%: F < 1) targets, and larger for red than blue targets in congruent (F(1,11) = 4.50, p < .05) but not incongruent (F < 1) trials. This pattern was consistent with the outcome of the 2 × 2 ANOVA on MRE values, showing a marginal main effect of Congruency (F(1,11) = 4.06, p = .07), no effect of Color (F < 1), and a marginal Congruency × Color interaction (F(1,11) = 4.62, p = .054). Like in Experiment 1 the pattern of MRE values supported the hypotheses described in the introduction. The response speed enhancement was obtained in both multisensory conditions, congruent and incongruent; furthermore, its amount was larger in congruent than incongruent trials only for red targets, as expected on the basis of the color selectivity of the spatial congruency effect, consistent with the SC-mediated MI. Differently from Experiment 1, a significant difference in accuracy for red vs. blue targets emerged in Experiment 2. The average proportion of Misses was slightly lower for blue than red targets in Experiment 1, but the difference did not reach the level of significance. In Experiment 2, instead, detection accuracy was significantly better for blue than red targets, contrary to the superiority of red targets in response speed. This difference might be attributed to the relative ineffectiveness

Downloaded by [New York University] at 12:53 07 October 2014

Makovac & Gerbino

FIGURE 3. Panel A shows mean Miss percentages and RTs for correct responses (± 1 s.e.m.) in the three sets of positive trials of Experiment 2, for red and blue targets. In unisensory (US) trials red targets led to more errors and longer RTs. The number of errors was the same in the three positive conditions, and was larger for red than blue targets. Relative to US trials, RTs were shorter in both multisensory congruent (MSC) and multisensory incongruent (MSI) trials. Panel B shows the distribution of MRE amounts indicating that, as in Experiment 1, performance was better in MSC than MSI trials with red but not blue targets.

29

30

The Journal of General Psychology

of the red increment, despite its nominal equivalence to the blue increment, as a consequence of the saturation of the luminance channel by the yellow background. As regards the slight tendency to an accuracy loss in multisensory trials observed in Experiment 1, this was not replicated in Experiment 2, supporting our suggestion that it could depend on the temporal aspects of the random luminance modulation technique.

Downloaded by [New York University] at 12:53 07 October 2014

General Discussion To obtain converging evidence on the role of the SC in MI we run two experiments in which different methods were adopted for isolating the SC-effectiveness of red (SC-effective) vs. blue (SC-ineffective) targets: the random luminance modulation technique in Experiment 1 and the tritanopic technique in Experiment 2. The results of the two experiments demonstrated the color selectivity of the spatial congruency effect and provided us with correlational evidence that the integration of multisensory information made available to participants was mediated by the SC. The widely adopted approach in the study of the MRE is the redundant signal paradigm (Anastasio, Patton, & Belkacem-Boussaid, 2000) in which participants are instructed to respond to a set of stimuli from different modalities. Here, we asked participants to perform a simple detection task within a focused attention paradigm, in which responses should only be made to a stimulus from a predefined target modality while stimuli from non-target modalities should be ignored. To summarize, Experiments 1 and 2 provided us with two results: the detection of SC-effective (red) but not of SC-ineffective (blue) targets displayed a spatial congruency effect (i.e., a larger MRE when visual and auditory stimuli originated from the same position than when they originated from different positions): in the same-position condition the MRE was larger for SC-effective (red) than SCineffective (blue) targets. Our experiments report additional behavioral evidence consistent with the crucial role of the SC in MI and highlight the automaticity of the integration regardless of the paradigm adopted (focused attention vs. redundant target). Although attention was not directly manipulated in our study, the fact that a similar pattern of results is obtained regardless of the direction of attention is in accordance with the idea that MI takes place in an early pre-attentive stage and can drive attention (Vroomen et al., 2001b). In fact, the role of attention in MI is controversial (McDonald et al., 2003; Shore & Simic, 2005; Talsma, Senkowski, Soto-Faraco, & Woldorff, 2010). In general, MI is thought as an automatic process. Other authors, however, argue that MI is not automatic and can be more or less effective when tasks are performed in multiple modalities (Alsius, Navarra, Campbell, & Soto-Faraco, 2005). Nowadays, we know that attention can influence target selection both through bottom-up (depending on target salience) and top-down (depending on target relevance for the task and voluntary intention of the observer) mechanisms. Given this

Downloaded by [New York University] at 12:53 07 October 2014

Makovac & Gerbino

31

distinction, the literature provides evidence of a bidirectional influence between MI and attention. For example, spatially and temporally coincident stimuli are more likely to get detected and further processed that the unisensory components alone. This is consistent with the idea that bottom-up processes can capture attention and orient it more easily towards an input source possessing multisensory properties (Driver, 1996; Spence & Driver, 2004; Van der Burg, Olivers, Bronkhorst, & Theeuwes, 2008; Vroomen, Bertelson, & de Gelder, 2001a). On the other hand, spatial attention has been shown to affect the earliest multisensory components of the event-related potential (Talsma, Doty, & Woldorff, 2007), which at least suggests that attention is involved in MI processes. Although our study does not disentangle the different aspects of the complex interplay between attention and MI, it demonstrates that the distribution of attention across auditory and visual modalities is not necessary to obtain a stronger MRE for spatially coincident stimuli, and offers converging evidence of the SC involvement in MI. NOTE 1. Mean RTs are presented in the text and figures as values in ms, although all statistical analyses were run on transformed scores (x = 1/RT). Analogously, MRE in the text and figures are values derived from raw RTs in ms, while MRE used in statistical analyses were based on transformed scores.

AUTHOR NOTES Elena Makovac earned her PhD in experimental psychology and cognitive neuroscience jointly at the University of Edinburgh (UK) and University of Trieste (IT). She is currently a post-doc at the Neuroimaging Laboratory, IRCCS Santa Lucia Foundation (Rome, Italy). Walter Gerbino is a professor in the Department of Life Sciences - Psychology Unit “Gaetano Kanizsa” at University of Trieste (IT). REFERENCES Alais, D., Newell, F. N., & Mamassian, P. (2010). Multisensory processing in review: From physiology to behaviour. Seeing and Perceiving, 23, 3–38. Alsius, A., Navarra, J., Campbell, R., & Soto-Faraco, S. (2005). Audiovisual integration of speech falters under high attention demands. Current Biology, 15, 839–843. Anastasio, T. J., Patton, P. E., & Belkacem-Boussaid, K. (2000). Using Bayes rule to model multisensory enhancement in the superior colliculus. Neural Computation, 12, 1165–1187. Anstis, S., & Cavanagh, P. (1983). A minimum motion technique for judging equiluminance. In J. D. Mollon, & L. T. Sharpe (Eds.), Colour vision (pp. 155–166). London, UK: Academic Press. Barbur, J. L. (2004). «Double-blindsight» revealed through the processing of color and luminance contrast defined motion signals. Progress in Brain Research, 144, 243–259. Barbur, J. L., Harlow, A. J., & Plant, G. T. (1994). Insights into the different exploits of colour in the visual cortex. Proceeding of the Royal Society B, 258, 327–334.

Downloaded by [New York University] at 12:53 07 October 2014

32

The Journal of General Psychology

Bertelson, P., & Radeau, M. (1981). Cross-modal bias and perceptual fusion with auditoryvisual spatial discordance. Perception and Psychophysics, 29, 578–584. Bertelson, P., Vroomen, J., de Gelder, B., & Driver, J. (2000). The ventriloquist effect does not depend on the direction of deliberate visual attention. Perception and Psychophysics, 62, 321–332. Birch, J., Barbur, J. L., & Harlow, A. J. (1992). New method based on random luminance masking for measuring isochromatic zones using high resolution color displays. Ophthalmic and Physiological Optics, 12, 133–136. Bolognini, N., Frassinetti, F., Serino, A., & Ladavas, E. (2005). “Acoustical vision” of below threshold stimuli: Interaction among spatially converging audiovisual inputs. Experimental Brain Research, 160, 273–282. Bolognini, N., & L`adavas, E. (2005). Visual localization of sounds. Neuropsychologia, 43, 1655–1661. Brockmole, J. R., & Franconeri, S. L. (2009). Introduction to the special issue on binding. Visual Cognition, 17, 1–9. Burnett, L. R., Stein, B. E., Perrault, J. T. Jr., & Wallace, M. T. (2007). Exitotoxic lesions of the superior colliculus preferentially impact multisensory integration. Experimental Brain Research, 179, 325–338. Calkins, D. J. (2001). Seeing with S cones. Progress in Retinal and Eye Research, 20, 255–287. Calvert, G. A., & Thesen, T. (2004). Multisensory integration: Methodological approaches and emerging principles in the human brain. Journal of Physiology - Paris, 98, 191–202. Cavanagh, P., MacLeod, D. I. A., & Anstis, S. M. (1987). Equiluminance: Spatial and temporal factors and the contribution of blue-sensitive cones. Journal of the Optical Society of America, 4, 1428–1438. Colonius, H., & Diederich, A. (2012a). Focused attention vs. crossmodal signals paradigm: Deriving predictions from the time-window-of-integration model. Frontiers in Integrative Neuroscience, 6(62). doi:10.3389/fnint.2012.00062 Colonius, H., & Diederich, A. (2012b). Models of the time window of integration. In B. E. Stein (Ed.), The new handbook of multisensory processes (pp. 545–555). Cambridge, MA: MIT Press. Corballis, M. C. (1998). Interhemispheric neural summation in the absence of the corpus callosum. Brain, 121, 1795–1807. Dacey, D. M., & Lee, B. B. (1994). Th blue on opponent pathway in primate retina originates from distinct bistratified ganglion cell type. Nature, 367, 731–735. de Monasterio, F. M. (1978). Properties of ganglion cells with atypical receptive-field organisation in the retina of macaques. Journal of Neurophysiology, 41, 1435–1449. Diederich, A., & Colonius, H. (2004). Modeling the time course of multisensory interaction in manual and saccadic responses. In G. A. Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processes (pp. 395–408). Cambridge, MA: MIT Press. Driver, J. (1996). Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading. Nature, 381, 66–68. Drum, B. (1983). Short-wavelength cones contribute to achromatic sensitivity. Vision Research, 23, 1433–1439. Eisner, A., & MacLeod, D. I. A. (1980). Blue-sensitive cones do not contribute to luminance. Journal of the Optical Society of America, 70, 121–123. Hendry, S. H., & Reid, R. C. (2000). The koniocellular pathway in primate vision. Annual Review of Neuroscience, 23, 127–153. Holmes, N. P. (2007). The law of inverse effectiveness in neurons and behaviour: Multisensory integration versus normal variability. Neuropsychologia, 45, 3340–3345. Holmes, N. P. (2009). The principle of inverse effectiveness in multisensory integration: Some statistical considerations. Brain Topography, 21, 168–176.

Downloaded by [New York University] at 12:53 07 October 2014

Makovac & Gerbino

33

Huerta, M. F., & Harting, J. K. (1984). The mammalian superior colliculus: Studies of its morphology and connections. In H. Vanegas (Ed.), Comparative Neurology of the Optic Tectum (pp. 687–773). New York, NY: Plenum Press. Kadunce, D. C., Vaughan, J. W., Wallace, M. T., Benedek, G., & Stein, B. E. (1997). Mechanisms of within- and cross-modality suppression in the superior colliculus. Journal of Neurophysiology, 78, 2834–2847. Leo, F., Bertini, C., di Pellegrino, G., & L`adavas, E. (2008). Multisensory integration for orienting responses in humans requires the activation of the superior colliculus. Experimental Brain Research, 186, 67–77. Lee, J., & Stromeyer, C. F. III (1989). Contribution of human short-wave cones to luminance and motion detection. The Journal of Physiology, 413, 563–593. Maravita, A., Bolognini, N., Bricolo, E., Marzi, C. A., & Savazzi, S. (2008). Is audio-visual integration subserved by the superior colliculus in humans? Clues from the redundant signal effect. Neuroreport, 19(3), 271–275. Marrocco, R. T., & Li, R. H. (1977). Monkey superior colliculus: properties of single cells and their afferent inputs. Journal of Neurophysiology, 40, 844–860. McDonald, J. J., Teder-S¨alej¨arvi, W. A., Di Russo, F., & Hillyard, S. A. (2003). Neural substrates of perceptual enhancement by crossmodal spatial attention. Journal of Cognitive Neuroscience, 15, 10–19. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748. Meredith, M. A., & Stein, B. E. (1996). Spatial determinants of multisensory integration in cat superior colliculus neurons. Journal of Neurophysiology, 75, 1843–1857. Mollon, J. D. (1982). Color vision. Annual Review of Psychology, 33, 41–85. Mozolic, J. L., Joyner, D., Hugenschmidt, C. E., Peiffer, A. M., Kraft, R. A., Maldjian, J. A., & Laurienti, P. J. (2008). Cross-modal deactivations during modality-specific selective attention. BMC Neurology, 8, 35. Ottes, F. P., Van Gisbergen, J. A., & Eggermont, J. J. (1987). Collicular involvement in a saccadic colour discrimination task. Experimental Brain Research, 66, 465–478. Peck, C. K. (1987). Saccade-related burst neurons in cat superior colliculus. Brain Research, 408, 329–333. P´erez-Bellido, A., Soto-Faraco, S., & L´opez-Moliner, J. (2013). Sound-driven enhancement of vision: Disentangling detection from decisional level contributions. Journal of Neurophysiology, 109, 1065–1077. Perrault, T. J. Jr., Vaughan, J. W., Stein, B. E., & Wallace, M.T. (2003). Neuron-specific response characteristics predict the magnitude of multisensory integration. Journal of Neurophysiology, 90, 4022–4026. Schiller, P. H., & Malpeli, J. G. (1977). Properties and tectal projections of monkey retinal ganglion cells. Journal of Neurophysiology, 40, 428–445. Shore, D. I., & Simic, N. (2005). Integration of visual and tactile stimuli: Top–down influences require time. Experimental Brain Research, 166, 509–517. Spence, C., & Driver, J. (1997). Audiovisual links in exogenous covert spatial orienting. Perception and Psychophysics, 59, 1–22. Spence, C., & Driver, J. (Eds.). (2004). Crossmodal space and crossmodal attention. Oxford, UK: Oxford University Press. Spence, C., & McDonald, J. J. (2004). The crossmodal consequences of the exogenous spatial orienting of attention. In G. Calvert, C. Spence, & B. Stein (Eds.), The handbook of multisensory processes (pp. 3–25). Cambridge, MA: MIT Press. Stein, B. E. (Ed.). (2012). The new handbook of multisensory processing. Cambridge, MA: The MIT Press.

Downloaded by [New York University] at 12:53 07 October 2014

34

The Journal of General Psychology

Stein, B. E., Huneycutt, W. S., & Meredith, M. A. (1988). Neurons and behavior: The same rules of multisensory integration apply. Brain Research, 448, 355–358. Stein, B. E., & Meredith, M. A. (1993). Merging the senses. Cambridge, MA: MIT Press. Stein, B. E., Meredith, M. A., Huneycutt, W. S., & McDade, L. (1989). Behavioral indices of multisensory integration: Orientation of visual cues is affected by auditory stimuli. Journal of Cognitive Neuroscience, 1, 12–24. Stiles, W. S. (1959). Color vision: The approach through increment tresholds. Proceedings of the National Academy of Sciences, USA, 45, 100–114. Stockman, A. S., MacLeod, D. I. A., & DePriest, D. D. (1991). The temporal proprerties of the human short-wave photoreceptors and their associated pathways. Vision Research, 31, 189–208. Sumner, P. (2006). Inhibition versus attentional momentum in cortical and collicular mechanisms of IOR. Cognitive Neuropsychology, 23, 1035–1048. Sumner, P., Adamjee, T., & Mollon, J. D. (2002). Signals invisible to the collicular and magnocellular pathways can capture visual attention. Current Biology, 12, 1312–1316. Sumner, P., Nachev, P., Vora, N., Husain, M., & Kennard, C. (2004). Distinct cortical and collicular mechanisms of inhibition of return revealed with S cone stimuli. Current Biology, 14, 2259–2263. Talsma, D., Doty, T. J., & Woldorff, M. G. (2007). Selective attention and audiovisual integration: Is attending to both modalities a prerequisite for early integration? Cerebral Cortex, 17, 679–690. Talsma, D., Senkowski, D., Soto-Faraco, S., & Woldorff, M. G. (2010). The multifaceted interplay between attention and multisensory integration. Trends in Cognitive Science, 14, 400–410. Talsma, D., & Woldorff, M. G. (2005). Selective attention and multisensory integration: Multiple phases of effects on the evoked brain activity. Journal of Cognitive Neuroscience, 17, 1098–1114. Tardif, E., Delacuisine, B., Probst, A., & Clarke, S. (2005). Intrinsic connectivity of human superior colliculus. Experimental Brain Research, 166, 316–324. Teder-S¨alej¨arvi, W. A., Di Russo F., McDonald J. J., & Hillyard, S. A. (2005). Effects of spatial congruity on audio-visual multimodal integration. Journal of Cognitive Neuroscience, 17, 1396–1409. Todd, J. W. (1912). Reaction to multiple stimuli. NewYork, NY: Science Press. Van der Burg, E., Olivers, C. N., Bronkhorst, A. W., & Theeuwes, J. (2008). Pip and pop: Nonspatial auditory signals improve spatial visual search. Journal of Experimental Psychology: Human Perception and Performance, 34, 1053–1065. Vroomen, J., Bertelson, P., & de Gelder, B. (2001a). Directing spatial attention towards the illusory location of a ventriloquized sound. Acta Psychologica, 108, 21–33. Vroomen, J., Bertelson, P., & de Gelder, B. (2001b). The ventriloquist effect does not depend on the direction of automatic visual attention. Perception and Psychophysics, 63, 651–659. Wald, G. (1966). The receptors of human vision. Science, 145, 1007–1017. Welch, R. B., & Warren, D. H. (1986). Intersensory interactions. In K. R. Boff, L. Kaufman, & J. P. Thomas (Eds.), Handbook of perception and human performance, Vol. 1 (pp. 1–36). New York: Wiley.

Original manuscript received March 22, 2013 Final version accepted August 15, 2013

Color selectivity of the spatial congruency effect: evidence from the focused attention paradigm.

The multisensory response enhancement (MRE), occurring when the response to a visual target integrated with a spatially congruent sound is stronger th...
280KB Sizes 0 Downloads 3 Views