Acta Psychologica 153 (2014) 139–146

Contents lists available at ScienceDirect

Acta Psychologica journal homepage: www.elsevier.com/ locate/actpsy

Crossmodal attention switching: Auditory dominance in temporal discrimination tasks☆ Sarah Lukas a,b,⁎, Andrea M. Philipp a, Iring Koch a a b

Institute of Psychology, RWTH Aachen University, Aachen, Germany Institute of Psychology and Education, Ulm University, Ulm, Germany

a r t i c l e

i n f o

Article history: Received 16 December 2013 Received in revised form 28 August 2014 Accepted 11 October 2014 Available online xxxx PsycINFO classification: 2323 2326 2340 Keywords: Crossmodal interference Task switching Temporal judgment Modality appropriateness

a b s t r a c t Visual stimuli are often processed more efficiently than accompanying stimuli in another modality. In line with this “visual dominance”, earlier studies on attentional switching showed a clear benefit for visual stimuli in a bimodal visual–auditory modality-switch paradigm that required spatial stimulus localization in the relevant modality. The present study aimed to examine the generality of this visual dominance effect. The modality appropriateness hypothesis proposes that stimuli in different modalities are differentially effectively processed depending on the task dimension, so that processing of visual stimuli is favored in the dimension of space, whereas processing auditory stimuli is favored in the dimension of time. In the present study, we examined this proposition by using a temporal duration judgment in a bimodal visual–auditory switching paradigm. Two experiments demonstrated that crossmodal interference (i.e., temporal stimulus congruence) was larger for visual stimuli than for auditory stimuli, suggesting auditory dominance when performing temporal judgment tasks. However, attention switch costs were larger for the auditory modality than for visual modality, indicating a dissociation of the mechanisms underlying crossmodal competition in stimulus processing and modality-specific biasing of attentional set. © 2014 Elsevier B.V. All rights reserved.

1. Introduction In browsing the research literature about how humans deal with the variety of different sensory impressions, one comes quickly upon the phenomenon of intersensory bias. Intersensory bias is the degree to which stimulus processing in one modality is changed if it occurs simultaneously with a stimulus presented in another modality compared to if it occurs alone. In a variety of studies, a very strong and robust intersensory bias for vision over audition (and also over proprioception) can be found. For example, Colavita (1974) let his participants react as soon as they detected a visual or an auditory stimulus with separate key presses for each modality. In most of the trials, only one stimulus was presented (unimodal trials), but some bimodal trials were occasionally interspersed. Surprisingly, participants almost always responded only to the visual stimulus. After the experiment, some participants stated that

☆ This research was funded by the Deutsche Forschungsgemeinschaft Grant KO 2045/ 04-03 to Iring Koch and Andrea M. Philipp and was conducted at the Institute of Psychology, Aachen, Germany. We would like to thank Nadine Wahl for conducting the experiments as well as Ben Dyson and Micah Murray for their helpful comments on an earlier version of this article. ⁎ Corresponding author at: Institute of Psychology and Education, General Psychology, Ulm University, Albert-Einstein Allee 47, 89081 Ulm, Germany. E-mail address: [email protected] (S. Lukas).

http://dx.doi.org/10.1016/j.actpsy.2014.10.003 0001-6918/© 2014 Elsevier B.V. All rights reserved.

they did not even realize that bimodal trials had occurred. Colavita (1974) termed this effect “visual dominance” and suggested an attentional approach, assuming that information processing of two simultaneously presented stimuli in different modalities was capacity-limited and that visual processing receives attentional priority. On the basis of such findings, Posner, Nissen, and Klein (1976) developed the theory of directed attention, which also assumes a differential, modality-specific allocation of attention. Specifically, this theory states that vision is not as automatically attention-capturing as audition. To compensate for the ensuing relative disadvantage in crossmodal situations, visual stimuli are processed with attentional priority, which then results in the visual dominance effect. Findings reported by Egeth and Sager (1977) and Sinnett, Spence, and Soto-Faraco (2007) supported the notion that the visual dominance effect is an attentional effect. The relative dominance of vision over audition could be changed if attention was manipulated. Guiding attention to auditory stimuli either by decreasing the probability of visual stimuli (and increasing the number of bimodal trials) or by instructing participants to respond only to auditory stimuli reduced the visual dominance effect. Likewise, the effect was increased if more attention was allocated to the visual stimulus. These findings suggest that visual dominance is essentially an attention phenomenon. Ragot, Cave, and Fano (1988) also assumed attentional processes as functional basis of the visual-dominance effect, but they used a different experimental approach. Instead of measuring RT in a stimulus detection

140

S. Lukas et al. / Acta Psychologica 153 (2014) 139–146

task, they provided their participants with a spatial localization task. The task was to decide if either a visual stimulus or an auditory stimulus was presented to the left or the right side. Visual and auditory stimuli were simultaneously presented either on the same side (congruent) or on opposite sides (incongruent). On congruent trials, the same response is required for both stimuli; in incongruent trials, different responses are required for each stimulus. The task-relevant stimulus modality was swapped only after the first half of the experiment. That is, participants had to guide their attention only to stimuli in one modality and did not have to be prepared to respond to upcoming stimuli in the other modality. Ragot et al. (1988) found a general congruence effect, but this congruence effect was not larger for the auditory modality. That is, visual distracters did not elicit a stronger crossmodal interference effect than auditory distracters. To explain this symmetric crossmodal congruence effect, Ragot et al. (1988) assumed that attentional focusing on only one of the two modalities led to the disappearance of visual dominance and hypothesized that it might re-appear if attention was less focused (and thus more flexible) because a switch in the task-relevant modality was possible. Lukas, Philipp, and Koch (2010b) tested this hypothesis with a crossmodal attention-switching paradigm using lateralized visual and auditory stimuli in each trial. Critically, employing task-switching methodology (see, e.g., Kiesel et al., 2010, for a review of task switching research), participants were explicitly instructed to switch attention between modalities, as indicated by an explicit cue at the beginning of each trial. If the cue was visual (an asterisk in the center of the monitor), participants had to decide if the visual stimulus was presented left or right by pressing a left vs. right response key. If the cue was auditory, participants had to respond to the location of the auditory stimulus. The authors found indeed strong evidence for a visual dominance effect. RTs were generally shorter for visual stimuli than for auditory stimuli, and the congruence effect was much larger for auditory target stimuli than for visual target stimuli. That is, visual distracters induced more crossmodal interference while processing auditory stimuli than vice versa. Notably though, the attention switch costs that were found in modality switches relative to modality repetitions across trials were similar in both modalities, even though a subsequent study suggested differentially increased auditory switch costs if the cuing interval was very short (Lukas, Philipp, & Koch, 2010a). These findings are in line with the theory of directed attention. However, the study of Ragot et al. (1988) as well as Lukas et al. (2010b) used a spatial task, which is, according to the modality appropriateness hypothesis, the more “appropriate” dimension for vision (e.g., Welch, DuttonHurt, & Warren, 1986; Welch & Warren, 1980). In most cases, the stimulus input is complex and consists of a variety of dimensions (e.g., space, and time) and modalities (e.g., vision, audition, and touch). According to Freides (1974), each modality is specified to process information about its “appropriate” dimension. For example, vision is especially accurate in the dimension of space, whereas audition performs better in the dimension of time. Empirical evidence for the modality appropriateness hypothesis comes particularly from findings revealing auditory bias on vision (e.g., Aschersleben & Bertelson, 2003; Shams, Kamitani, & Shimojo, 2000; Walker & Scott, 1981; Welch et al., 1986). For example, Aschersleben and Bertelson (2003) used a sensorimotor synchronization task, in which the participants should reproduce a sequence of light flashes with tapping movements while ignoring simultaneously presented auditory distracters. In a second experiment, the converse task was assigned, that is, reproducing an auditory sequence while ignoring the visual distracters. A much stronger bias towards the auditory distracters could be found in Experiment 1 than towards the visual distracters in Experiment 2 (though this bias was significant, too). In a recent study, Sandhu and Dyson (2012) found first hints of a relative advantage for processing auditory stimuli in a modality-switch study that was quite similar to the studies by Lukas et al. (2010a,b). In Sandhu and Dyson's (2012) study, subjects were required to process

auditory stimuli in a temporal duration judgment task, whereas visual stimuli required a spatial localization task. These authors found increased RT switch costs for auditory stimuli on incongruent trials, but the error rates showed exactly the opposite pattern, with increased error switch costs for visual stimuli and thus evidence for auditory dominance. However, given the opposing patterns in RT and error rates, it is not easy to interpret these findings as clear evidence for auditory dominance in crossmodal attention switching (i.e., as opposed to the findings of Lukas et al., 2010a,b), but what is clear though is that the dominance relations between vision and audition may indeed depend on the specific processing requirements of the task. Based on Sandhu and Dyson's (2012) findings and previous findings reported by Lukas et al. (2010a,b), it is obviously important to examine whether the dominance relation in visual–auditory crossmodal attention tasks is crucially mediated by task demands. Tasks requiring spatial processing might lead to visual dominance (which has been already shown by Lukas et al., 2010a,b), but tasks requiring temporal processing might result in auditory dominance. As Sandhu and Dyson (2012) used two different tasks for auditory and visual stimuli, a design in which the same task for both stimulus modality is used (in this case a temporal task) is needed to close the gap. The aim of the present study was to demonstrate auditory dominance in crossmodal attention switching when using a temporal processing task. Specifically, we presented visual and auditory stimuli simultaneously and let the participants decide if the stimulus in the relevant modality was presented for a short vs. long duration. Across trials, the relevant stimulus modality switched unpredictably, as indicated by an explicit instructional cue in the same modality prior to target-stimulus onset. We report two experiments. In Experiment 1, we used an intermediate cuing interval of 600 ms. In Experiment 2, we manipulated the duration of the cuing interval to examine the influence of cue-based preparation for crossmodal attention switches with respect to temporal discrimination judgments. 2. Experiment 1 2.1. Method 2.1.1. Participants 18 participants took part in the experiment (12 female, 6 male between 19 and 28 years; M = 22.6, SD = 2.4). All reported normal or corrected-to-normal vision and audition. They received partial course credit or 6 €. 2.1.2. Stimuli and tasks The visual stimulus was a white diamond (1.5 cm × 1.5 cm) on a black background presented at the center of the monitor. Participants were seated about 50 cm from the monitor, resulting in a visual angle of 1.72° of the visual stimulus. The auditory stimulus was a tone of 400 Hz binaurally presented via earphones. Both stimuli were presented simultaneously and randomly for either 100 ms (short presentation time) or for 500 ms (long presentation time). On congruent trials, both stimuli were presented for either the short or long presentation time. On incongruent trials, one stimulus was presented for the short presentation time, and the other for the long presentation time (see Fig. 1). Before stimulus presentation, explicit cues were shown, indicating the task-relevant stimulus modality. An asterisk, presented at the center of the monitor for 200 ms, indicated that the visual stimulus was task-relevant. A 600 Hz tone, also presented for 200 ms, indicated the auditory stimulus as relevant. The task was to decide if the stimulus in the relevant modality was presented for a short or long time interval. Response keys were the left and right ALT keys on a German QWERTZ keyboard. The stimulus-response mapping was counterbalanced across participants. The experiment was programmed with ERTS (Version 33.33e, BeriSoft Cooperation, Frankfurt am Main, Germany).

S. Lukas et al. / Acta Psychologica 153 (2014) 139–146

Fig. 1. Experimental design. If an auditory cue was presented, participants should decide if the auditory stimulus was presented for a short or a long time period, ignoring the simultaneously presented visual stimulus (the length of the stimulus modality presentation is pictured with short vs. long arrows under the stimulus-modality symbols). If a visual cue was presented, participants should decide if the visual stimulus was presented for a short or a long time period, ignoring the auditory stimulus.

2.1.3. Procedure Instructions were given visually on the monitor and orally if further explanation was needed. Each trial started with a cue. After the cue was presented for 200 ms, the screen was black for another 400 ms. Then, the visual stimulus and the auditory stimulus were presented simultaneously for either 100 ms or 500 ms. The screen turned black again until a response was made or 3000 ms elapsed. Visual feedback was given for 400 ms immediately after the response or after the maximum of 3000 ms elapsed. It stated “Richtig!” (correct) if the response was correct, “Falsch!” (wrong) if the answer was wrong, and “Bitte schneller antworten!” (please respond faster) if no response was recorded within the 3000 ms response window. After 600 ms, the next cue appeared. Hence, the response–cue interval (RCI) was 1000 ms, the response– stimulus interval (RSI) was 1600 ms. The experiment started with one practice block of 24 trials, followed by six blocks of 66 trials each. Each block was preceded by two nonrecorded warm-up trials. That is, 64 valid trials in which each of the eight experimental conditions was equally distributed went into the analysis. Altogether, the experiment lasted about 35 min. The experiment complied with the ethical standards established by the Declaration of Helsinki. 2.1.4. Design The independent variables were modality (visual vs. auditory), modality transition (switch vs. repetition), and congruence (incongruent vs. congruent). The dependent variables were RT and error rates. Significance was tested at α = .05. 2.2. Results and discussion Besides the two warm-up trials, the first (recorded) trial of each block was discarded from analysis because it could not be determined whether it was a switch trial or a repetition trial. Altogether, there were 49.2% of repetition and 50.7% of switch trials. There were no systematic frequency differences between the experimental conditions in repetition and switch trials, χ2 (3) = 7.11, p N .05. Trials with a RT of less than 200 ms or higher than three SD from the mean of each subject were excluded from RT analysis. Additionally, all error trials and trials following an error were excluded. Altogether, the error rate was 14.2%. For one subject, the error rate on visual-relevant incongruent trials approached 100%, suggesting that the task was not correctly understood, so that the data of this subject were excluded from further analysis, resulting in a final sample of n = 17.

141

Fig. 2. RT in Experiment 1 as a function of stimulus modality, stimulus-modality transition and congruence. Error bars indicate +/− one standard error.

2.2.1. RT We conducted a three-way analysis of variance (ANOVA) on RT using the independent variables modality, modality transition, and congruence (see Fig. 2). The main effect of congruence was significant, F(1, 16) = 1011.7, p b .001, ηp2 = .98. RT for incongruent trials was longer (934 ms) than for congruent trials (634 ms). The main effect of modality was not significant, F(1, 16) = 2.8, p = .113, ηp2 = .15, but the interaction of modality and congruence was significant, F(1, 16) = 16.2, p b .001, ηp2 = .5. The congruence effect was larger for the visual-relevant trials (340 ms) than for the auditory-relevant trials (261 ms), suggesting auditory dominance. The main effect of modality transition was significant, too, F(1, 16) =31.9, p b .001, ηp2 = .67. Participants responded faster on repetition trials (753 ms) than on switch trials (815 ms). Also, the interaction between modality and modality transition was significant, F(1, 16) =5.8, p = .028, ηp2 = .27. Switch costs were higher for auditory trials (88 ms) than for visual trials (35 ms), suggesting that switching is more difficult from vision to audition than vice versa. No further interaction was significant, Fs b 1.1

2.2.2. Error rates The same three-way ANOVA as for the RT was conducted for the error rate (see Fig. 3). The main effect of congruence was significant, F(1, 16) = 25.4, p b .001, ηp2 = .61. Incongruent trials produced more errors (19.7%) than congruent trials (6.5%). Unlike in RT, the main effect of modality was significant, too, F(1, 16) = 21.0, p b .001, ηp2 = .57. Visual stimuli elicited by far more errors (17.5%) than auditory stimuli (8.6%). Like in RT, the interaction of modality and congruence was significant, F(1, 16) = 11.03, p = .004, ηp2 = .41, indicating that the congruence effect was larger for the visual trials (19.3%) than for the auditory trials (7.1%). The main effect of modality transition failed to reach statistical significance, F(1, 16) = 4.4, p = .053, ηp2 = .21, but the interaction of modality transition and modality was significant, F(1, 16) = 5.8, p = .029, ηp2 = .27, which revealed that error switch costs were larger for auditory trials (3.6%; F [1, 16] = 9.2. p = .008, ηp2 = .36), than for 1 To gain further insight of the underlying processes of the effects found in Experiment 1, we conducted another ANOVA including the variable duration of the relevant stimulus, resulting in a four-way ANOVA (with the variables modality, modality transition, congruence, and duration). It showed that the congruence effect was especially large for trials in which the auditory long stimulus was relevant (475 ms) and in which the visual short stimulus was relevant (584 ms), F(1, 16) = 731.77, p b .001, ηp2 = .98. That is, the stimulus combination of a short visual stimulus and a long auditory stimulus elicited especially high RTs, no matter which stimulus was relevant for the task. The three-way ANOVA with the variables modality, modality transition, and duration was as well significant, F (1, 16) = 19.65, p b .001, ηp2 = .55. It showed that switch costs were especially high for auditory short stimuli (140 ms) of comparable size for auditory long stimuli (62 ms) and visual long stimuli (54 ms) and barely present for visual short stimuli (6 ms).

142

S. Lukas et al. / Acta Psychologica 153 (2014) 139–146

Fig. 3. Error rate in Experiment 1 as a function of stimulus modality, stimulus-modality transition, and congruence. Error bars indicate +/− one standard error.

visual trials (which did not result in significant switch costs; 0.7%; F b 1). This interaction was further qualified by a significant three-way interaction, F(1, 16) = 4.7, p = .046, ηp2 = .23. Switch costs were especially large for auditory incongruent trials (10.7%) and comparably low for the other conditions. The interaction of modality transition and congruence was not significant, F(1, 16) = 1.5, p N .05, ηp2 = .09.2 In summary, both RT and error rates revealed auditory dominance in the crossmodal congruence effect. That is, in the present temporal discrimination task, the auditory stimulus disturbs processing of the visual stimulus more than vice versa. These results are in striking contrast to the results of Lukas et al. (2010a,b), who used a spatial discrimination task in an otherwise almost identical paradigm and found a visual dominance effect. This contrast indicates that the relative crossmodal dominance patterns indeed depend on the processing requirements of the task. Spatial discrimination tasks favor visual dominance (Lukas et al., 2010a,b), whereas temporal discrimination tasks favor auditory dominance, as shown in the present study under very comparable experimental conditions (see also Sandhu & Dyson, 2012). Taken together, the findings speak in favor of the modality appropriateness hypothesis. If the nature of the task is adjusted to the respective stimulus modality, the relative dominance of processing in one modality over the other modality is changed, which in turn affects (or even reverses) the pattern of modality-specific congruence effects. The analysis of stimulus presentation duration gave further insight into this effect. It showed that the large congruence effect was mainly due to the combination of auditory long presented and visually short presented stimuli. Regarding time perception, the Scalar Expectancy Theory assumes that three components interact and result to the time percept: a pacemaker, a mode switch, and an accumulator (e.g. Chen & Yeh, 2009; Droit-Volet & Meck, 2007). The pacemaker sends pulses, the mode switch serves as a gate that controls when the pulses are started and ended to be stored in the accumulator. Time perception can be affected be each of the three components. It is assumed that for auditory stimuli, the pacemaker is faster than for visual stimuli, resulting in the effect that auditory stimuli are usually perceived as being longer than visual stimuli (Chen & Yeh, 2009). In the present study, the stimulus combination of auditory long and visual short stimuli yields a high RT. It is possible that the auditory long stimulus influences the perception of the visual stimulus, prolonging its perceived duration, thus making it more difficult to differentiate. However, this effect is not reflected in the error rates.

2 Like for the RT, we additionally conducted a four-way ANOVA including the variable stimulus duration. The three-way interaction of stimulus modality, congruence and duration showed that the congruence effect for visual long stimuli was especially high (29.6%), whereas it was of comparable size for the other trial types (auditory long: 7.5%, auditory short: 6.5%, visual short: 8.8%). This result stands in contrast to the RT, for which visual short and auditory long trials elicited high RT.

The contrasting stimulus combination of a long visual and a short auditory stimulus yielded the highest error rate; which indicates a speedaccuracy trade-off. Nonetheless, this trade-off can be interpreted with respect to an auditory dominance effect in a temporal task. Subjects might tend to react to the auditory stimulus. If the auditory stimulus is long, the RT is prolonged, maybe because the two stimuli are more difficult to differentiate, but the long processing time prevents from a hastily decision, leading to less errors. In contrast, with a short auditory stimulus and a long visual stimulus, attention is drawn to the auditory stimulus, leading to a fast reaction, but to a higher error rate. Please note though that we found larger attention switch costs for auditory trials than for visual trials. We address this modality-specific switch-cost asymmetry in Experiment 2. The aim of Experiment 2 was to replicate the auditory dominance effect, but, in addition, we examined whether the increased attention switch costs for auditory trials are the result of an involuntary carryover of a strong bias for visual processing. It is assumed that for tasks with only visual stimuli, a strong bias for the previous task is attenuated with a longer preparation time for the upcoming task (e. g., Meiran, 2000; Meiran, Kessler, & Adi-Japha, 2008). It was shown that preparation also helped subject to direct attention to a specific stimulus modality (e. g., Murray, De Santis, Thut, & Wylie, 2009; Spence & Driver, 1997) leading to better performance. In a bimodal task switching paradigm, we could show that modality switch costs were significantly reduced by preparation using a spatial task (Lukas et al., 2010a). If there is indeed a stronger attention bias for visual processing, the switch-cost asymmetry observed in Experiment 1 should be reduced or eliminated if there is sufficient time for preparing an attention switch to processing in the auditory modality. To this end, we varied the cuing interval (CTI) in Experiment 2. To our knowledge, modality-specific preparation effects with respect to a temporal discrimination task have not been examined so far. In Experiment 2, we specifically examined whether the modalityspecific asymmetries in the congruence effect and the switch costs were influenced by cue-based, advance preparation. 3. Experiment 2 3.1. Method 3.1.1. Participants 14 participants took part in the experiment (7 female, 7 male between 19 and 29 years; M = 22.4, SD = 2.8). All reported normal or corrected-to-normal vision and audition. They received partial course credit or 6 €. 3.1.2. Stimuli, tasks, procedure, and design The experimental conditions were like in Experiment 1 except for the duration of the CTI. In Experiment 1, the RCI was 1000 ms and the CTI was 600 ms. In Experiment 2, the long CTI was 1200 ms and the short CTI was 400 ms, so that the RSI varied inversely (i.e., 2200 ms vs. 1400 ms) to keep the overall RCI constant and equal to that used in Experiment 1 (i.e. 1000 ms). There were 48.3% of repetition and 51.7% of switch trials in Experiment 2. There were no systematic differences between the experimental conditions in repetition and switch trials, χ2 (7) = 7.22, p N .05. 3.2. Results and discussion 3.2.1. RT A four-way ANOVA with modality, modality transition, congruence, and CTI as independent variables revealed a main effect of congruence, F(1, 13) = 1,034.8, p b .001, ηp2 = .99 (see Fig. 4). RT on congruent trials was shorter (648 ms) than on incongruent trials (953 ms). The main effect of modality was not significant, F(1, 13) = 2.3, p N .05, ηp2 = .15, but the two-way interaction of modality and congruence was significant, F(1, 13) = 10.2, p = .007, ηp2 = .44. Like in Experiment 1, we

S. Lukas et al. / Acta Psychologica 153 (2014) 139–146

143

Fig. 4. RT in Experiment 2 as a function of stimulus modality, stimulus-modality transition, congruence and cue–target interval. Error bars indicate +/− one standard error.

found that the congruence effect was larger for visual stimuli (329 ms) than for auditory stimuli (282 ms). The main effect of modality transition was significant, F(1, 13) = 44.45, p b .001, ηp2 = .77, indicating that RT was longer on modalityswitch trials than on modality-repetition trials (840 ms vs. 761 ms), revealing switch costs of 79 ms. Likewise, the main effect of CTI was significant, F(1, 13) = 22.8, p b .001, ηp2 = .64, indicating that RT was shorter with long CTI than with short CTI (769 ms vs. 832 ms) and thus revealing a general cue-based preparation effect of 63 ms. Moreover, these two variables interacted, F(1, 13) = 16.8, p = .001, ηp2 = .56, showing that the modality switch costs were indeed reduced with long CTI relative to short CTI (51 ms vs. 107 ms). Like in Experiment 1, the switch costs were larger for auditory trials than for visual trials (96 ms vs. 62 ms), but the corresponding interaction of modality and modality transition was not significant, F(1, 13) = 2.72, p = .12, ηp2 = .17, and this interaction was not further modulated by duration, F(1, 12) = 0.1, p = .74, ηp2 = .001, nor by CTI, F(1, 13) = 1.6, p N .22, ηp2 = .11. CTI also did not modulate the interaction of modality and congruence (see above), F(1, 13) = 3.45, p = .086, ηp2 = .21. All other effect were non-significant, too, Fs b 1. 3.2.2. Error rate Congruent trials were less error prone (6.1%) than incongruent trials (16.1%), F(1, 13) = 34.7, p b .001, ηp2 = .73. There was a main effect of modality, F(1, 13) = 11.4, p = .005, ηp2 = .47. Visual trials elicited a higher error rate (13.9%) than auditory trials (8.3%). The interaction of modality and congruence was significant, too, F(1, 13) = 18.6, p b .001, ηp2 = .59. As for the RT, the congruence effect was larger for visual trials (13.7%) than for auditory trials (6.3%) (see Fig. 5). The main effect of modality transition was not significant, F b 1, but there was a main effect of CTI, F(1, 13) = 17.0, p b .001, ηp2 = .57. With a short CTI, participants produced 13.4% of errors, with a long CTI 8.8%. Moreover, modality transition interacted with CTI, F(1, 13) = 9.96, p = .008, ηp2 = .43, indicating that error switch costs were indeed smaller (even eliminated) with long CTI (− 0.9%) than with short CTI (2.1%). Modality transition also interacted with congruence, F(1, 13) = 5.18, p b .05, ηp2 = .29, indicating that the congruence effect was larger on switch trials (12.0%) than on repetition trials (8.0%). In addition, CTI interacted with congruence, F(1, 13) = 15.4, p = .002, ηp2 = .54, showing that a long CTI reduced the congruence effect from 12.5% to 7.4%. The three-way interaction of CTI, modality transition, and congruence was significant, too, F(1, 13) = 15.8, p = .002, ηp2 = .55, showing that the congruence effect was not much affected by CTI on modality repetitions (7.8% vs. 8.1%), but on modality switches, the congruence effect was reduced from 17.3% with short CTI to 6.8% with long CTI. No further interaction was significant, Fs b 2.6, ps N .13. In summary, we replicated the auditory dominance effect by showing a larger congruence effect for the visual than for the auditory modality.

Fig. 5. Error rate in Experiment 2 as a function of stimulus modality, stimulus-modality transition, congruence, and cue–target interval. Error bars indicate +/− one standard error.

Like in Experiment 1, switch costs were higher for the auditory modality (i.e., 96 ms vs. 62 ms), but this effect was not significant, which might be due to reduced statistical power. Importantly, a long CTI served to improve performance generally and reduce switch costs specifically, but it did not seem to alter the pattern of crossmodal interference. 4. General discussion The present study was aimed at examining the relative dominance patterns in visual–auditory crossmodal attention. Specifically, previous studies found clear visual dominance when employing a crossmodal attention-switching paradigm using task-switching methodology (Lukas et al., 2010a,b). These studies used a crossmodal spatial discrimination (i.e., localization) task, but a more recent study (Sandhu & Dyson, 2012), under quite similar experimental switching conditions, found more mixed evidence, with no clear visual dominance but even some hints at auditory dominance when visual stimuli were spatially processed but auditory stimuli required temporal processing. The present study tested the hypothesis that auditory dominance would prevail if the task requires temporal processing in both modalities. To this end, we used a temporal duration judgment tasks for both the visual and auditory modality. 4.1. Synopsis of results Two experiments revealed clear evidence for crossmodal interference. First, we found a strong crossmodal congruence effect, and, second, we found attentional modality switch costs. Notably, the congruence effect was markedly asymmetric. Both experiments show that the congruence effect was larger for the visual modality than for the auditory modality, and this was true both for RT and error rate. This highly consistent effect indicates that the auditory distractor interferes more strongly with processing visual targets than vice versa, which clearly suggests auditory dominance. A more detailed analysis revealed that the trial combination of an auditory long and a visual short stimulus yields an especially long RT. Concerning attention switching, larger switch costs for the auditory modality were shown which were significant in Experiment 1 and numerically present, though not statistically significant, in Experiment 2. Finally, Experiment 2 revealed that cue-based preparation (i.e., with long CTI) is generally beneficial and serves to reduce the attention switch costs specifically. However, this preparation effects was not markedly modality-specific, suggesting that cue processing refers to more generic processes.

144

S. Lukas et al. / Acta Psychologica 153 (2014) 139–146

In the next subsections, we first discuss the crossmodal congruence effect, then the attention switch costs, and finally the apparent dissociation of crossmodal congruence effects and attentional switch costs. 4.2. Auditory dominance in crossmodal congruence in temporal discrimination tasks The present pattern of a clear auditory dominance with respect to the crossmodal congruence effects (i.e., larger congruence effects when processing visual targets) in a temporal task differs from previous studies using spatial tasks. For example, Ragot et al. (1988) found no asymmetrical congruence effects in spatial tasks when the modality of the target stimuli were constant, whereas Lukas et al. (2010a,b) found clear visual dominance when the target modality could switch from trial to trial. Finally, Sandhu and Dyson (2012) found somewhat mixed evidence, with a different pattern in RT and error rate, but the error rates suggested more clearly auditory dominance. Notably, Sandhu and Dyson (2012) used a temporal task for the auditory stimuli. In the present experiments, we required temporal processing (i.e., duration judgment) for both visual and auditory stimuli and found auditory dominance. This finding identifies the processing requirements of the task as critical determinant of the relative dominance pattern in crossmodal attention. Chen and Zhou (2013) used a categorization task with three different congruence levels: visual and auditory stimuli were faces and spoken names of politicians and movie stars. If the name (auditory) and the face (visual) were representing exactly the same, the condition was called congruent. The two further conditions were visual and auditory stimuli requiring the same response i.e., both, auditory and visual stimulus were a politician or a movie star (incongruent at preresponse level) and visual and auditory stimuli requiring different responses—one stimulus was a politician, one was a movie star (incongruent at response level). Their results showed that visual dominance was seen at the preresponse level, but auditory dominance at the response level. This study only used a categorization task (politician vs. movie star). There are no hints of how the results would look like if a temporal or a spatial task was used with respect of the three congruence levels. But the results are an indication that modality domincance is highly task-specific. The trial combination of an auditory long and a visual short stimulus yields an especially long RT, suggesting that the auditory stimulus prolongs the perceived duration of the visual stimulus as was found previously (Chen & Yeh, 2009). This perceived prolongation makes it more difficult to differentiate between the two stimuli—but prevents from a hasty response. In contrast, the trial combination of a short auditory and a long visual stimulus directs attention to the auditory stimulus, yielding a fast response, but a high error rate. This finding corroborates an auditory dominance effect. With respect to the task that we chose for this study – a temporal task – one can add that the issue of time perception is complex. Many different factors influence the perceived duration of a stimulus. For example the location of stimuli (Kliegl & Huckauf, 2014), the presence of a second task (Brown, 2008), the allocated attention to the stimulus (Macar, Grondin, & Casini, 1994; Tse, Intriligator, Rivest, & Cavanagh, 2004), affective factors (Angrilli, Cherubini, Pavese, & Manfredini, 1997), or the temporal task itself (Gil & Droit-Volet, 2011). It is beyond the scope of this study to discuss all these factors, but clearly more research is needed to fully understand the interactions of time perception with crossmodal auditory–visual stimuli. As Ben Dyson, one of our reviewers, pointed out, the large congruence effect might arise because the different stimulus duration manipulated artificially the difficulty of the task (hence, both RT and error). Participants might wait for the long stimulus to end and thus, performance on the congruent short trials will lead to responding earlier (i.e. about 400 ms earlier, because the difference between short and long duration is 400 ms) than the congruent long trials. Congruent short trials were answered with a mean RT of about 600 ms, congruent

long trials were answered with a mean RT of about 672 ms. Although the difference is significant, it is far from the assumed 400 ms difference. Concerning the error rate: it is not entirely clear, why a “waiting for the longer stimulus duration” – which could be seen in the RT—should be carried over to the error rates. A longer RT should rather yield fewer errors if a speed-accuracy trade-off is existent. However, consistent with the RT, and the suggestion that longer stimuli are more “difficult”, longer stimuli reveal more errors. But congruent trials do not show a difference in error rate with respect of the duration: congruent short trials lead to an error rate of 6%, congruent long trials reveal an error rate of 6.9%. Here as well, the difference between short and long presented congruent trials is not large. A post-hoc test revealed that is was not significant. That is, a carry-over process from RT to error rate of a possible processing facilitation as a result of reliably shorter stimulus presentation in half of the congruent trials is not probable. Regarding the stimulus modality, the error rate reveals the same pattern of results like the RT: a stronger congruence effect for visual (19%) than for auditory trials (7%). Even if there is a constant adding to the congruence effect, this constant should be the same for auditory as for visual trials. However, we find a difference in visual and auditory trials for both – RT and error rate – and this is the important effect we were focusing on. So, even if there was a difference of task difficulty regarding the stimulus durations, it is different for visual and auditory trials—and we claim that this due to some cognitive processes that are modalityspecific. Taken together with previous findings, the present study clearly supports previous suggestions that temporal tasks favor the auditory domain, which is in line with a recent proposal by Sandhu and Dyson (2012). These authors combined temporal tasks and spatial tasks, which might have led to a less pronounced dominance pattern and so in more mixed findings (i.e., different patterns in RT and error rate). The present results confirm their proposal by showing that using temporal tasks for both modalities results in unambiguous evidence for auditory dominance. The present findings qualify the “directed attention” view (Posner et al., 1976) by demonstrating that visual attention does not necessarily dominate but that the modality dominance is also determined by task characteristics (e.g., whether it requires spatial or temporal processing). This does not exclude the idea that visual processing is intrinsically less attention-capturing than, for example, auditory processing, but it specifies this idea by adding the crucial contribution of task requirements, as proposed by the notion of “modality appropriateness” of processing when explaining patterns of crossmodal interference (e.g., Welch & Warren, 1980). 4.3. Crossmodal attention switch costs Some previous studies on modality switching used unimodal stimuli in detection tasks and found a relative cost when the present stimulus was not in the modality that was primed by the preceding stimulus (e.g., Hunt & Kingstone, 2004; Murray et al., 2009; Sinnett et al., 2007; Spence & Driver, 1997, Stephan & Koch, 2010). Hunt and Kingstone (2004) as well as Murray et al. (2009) did not find different switch costs for auditory and visual stimuli. Note though that using unimodal stimuli does not require selective attention because the stimuli are unambiguous with respect to the relevant task and/or modality. In contrast, using bimodal stimulation combined with cued task-switching methodology allowed us to examine both crossmodal congruence (see above) and crossmodal selective attention switching. We found clear attention switch costs in both experiments. Note that our cue was unimodal and the feedback given to the participants was only visual. Although we cannot completely rule out influences of this methodological specificity, we do not believe that it had a relevant impact on our findings. The cue-stimulus-modality mapping was investigated in previous studies (Lukas et al., 2010a,b) and did not reveal fundamental changes in the result pattern. If the visual feedback had an effect on guiding

S. Lukas et al. / Acta Psychologica 153 (2014) 139–146

attention, one would expect that it had guided attention to the visual domain. That is, if our findings were impacted, then rather in the direction of weakening the auditory dominance effect than enforcing it. Finding attention switch costs confirms previous results obtained in crossmodal attention switching studies (Lukas et al., 2010a,b; Sandhu & Dyson, 2012). Similar attention switch effects have also been found in switching between visual dimensions or features (e.g., Logan, 2005; Müller, Reimann, & Krummenacher, 2003) and between auditory dimensions and features (e.g., Koch, Lawo, Fels, & Vorländer, 2011), suggesting that the mechanisms underlying attention switches contribute to performance both across and within modalities. However, the present study shows larger attention switch costs for auditory stimuli in Experiment 1 (and a non-significant trend in Experiment 2). Hence, auditory dominance is not present in the attention switch costs, and if there is a relative dominance pattern, then it rather favors the visual domain (i.e., switching is more “costly” from the visual modality to the auditory modality than vice versa). This pattern is not very consistent across studies, though. For example, in Lukas et al. (2010b), switch costs were significantly larger for the auditory modality in Experiment 1, but this pattern was not replicated in their Experiment 2. Likewise, in Lukas et al. (2010a), there was no significant switch-cost asymmetry across four attention switching experiments (this study additionally included a non-switching baseline experiment), even though switch costs were numerically (but not significantly) higher for the auditory modality in some experiments. Sandhu and Dyson's (2012) study also showed no clear pattern of switch-cost asymmetry. Therefore, empirically, this pattern of visual dominance in attention switch costs is much less robust than that found in the crossmodal congruence effect. However, we can definitely conclude that the crossmodal congruence effect was completely reversed when using a temporal duration judgment task relative to a spatial localization task, whereas the pattern of somewhat larger switch costs for the auditory modality did not seem to be affected by the type of task required in crossmodal processing. This suggests that the processes underlying the crossmodal congruence effect and the attention switch costs are dissociable. Before we discuss this dissociation in the next subsection, it should be noted that this preliminary conclusion of a dissociation of the processes underlying the crossmodal congruence effect and the attention switch costs is also supported by the finding that cue-based preparation did not have a marked modality-specific influence, but it reduced switch costs consistently in both RT and error rates, whereas it did not affect the congruence effect very consistently (note though that the error rates showed a three-way interaction, suggesting that preparation reduced the congruence effect, but only on modality-switch trials). This pattern of preparation effect is consistent with previous findings using a crossmodal spatial attention switching task (Lukas et al., 2010a) and suggests an attentional biasing or weighting process (see next subsection). That is, across studies we found strong evidence for a dissociation of the processes underlying congruence effect and switch costs with respect to the interaction of modality (auditory vs. visual) and task requirements (spatial vs. temporal processing), and within the present study, we found somewhat less strong evidence for this dissociation with respect to the differential influence of cue-based preparation on the congruence effect and on the attention switch costs. 4.4. Dissociation of crossmodal congruence and crossmodal attention switching To account for the cue-based preparation found in Experiment 2, we assume that the cue triggers an attentional control process that changes processing weights for specific stimulus aspects. Such a competitive biasing mechanism has been assumed as a basic mechanism of visual selective attention (e.g., Desimone & Duncan, 1995) and also in research on visual attention switching (e.g., Logan, 2005) and visual task switching (e.g., Meiran et al., 2008). With respect to attentional processes in visual

145

search paradigms, Müller and colleagues (e.g., Müller et al., 2003; Rangelov, Müller, & Zehetleitner, 2012) developed a multiple weighting system account for attentional switch costs within vision. For audition, Dyson and Quinlan (2002) discuss a similar weighting system account and conclude that although vision and audition differ in early input stages, they seem to be governed by similar higher level monitoring processes. Based on the present set of findings, we believe that such a competitive weighting mechanism also applies to modality-specific processing biases. Specifically, we assume that attention switch costs between modalities can be explained both by persisting weights in favor of the irrelevant modality (i.e., proactive interference based on persisting activation of previous attentional set; see also Koch, Gade, Schuch, & Philipp, 2010) and by the time that it takes to change the modality-specific attentional processing weights in line with the current task demands. The cuing effect could then be readily explained by assuming that this re-weighting can be achieved at least partly prior to target presentation (e.g., Kiesel et al., 2010; for a review of theoretical accounts of task-switch costs). Hence, the attentional switch costs represent the action of transient control processes that change processing weights on a trial-to-trial basis in the sense of flexible attentional gating. Hints at a modality-specific asymmetry in attentional switch costs (i.e., smaller costs for the visual modality) could then be explained by assuming that the attentional dynamic is slower for auditory processing and thus shows some signs of relatively stronger attentional inertia (Koch & Lawo, 2014). In contrast, the robust finding of auditory dominance in the crossmodal congruence effect in the present study using a temporal task, and the finding of a robust visual dominance in previous studies using a visual task, suggests a more sustained mechanism of attentional, modality-specific biasing that depends on task demands (spatial vs. temporal processing). One way to think about this influence is that sustained attentional biasing generally shifts the relative processing weights in favor of one modality over the competing modality. In line with the multiple weighting account (Rangelov et al., 2012), the present data suggest that the sustained modality-specific biasing component is largely independent from the transient weighting component. However, at this stage, the present consideration remain speculative, and further work, possibly including formal modeling, is required to solidify our specific version of a modality-weighting account that combines sustained and transient attentional control aspects in crossmodal attention switching.

5. Conclusions The present study demonstrates that previous evidence for visual dominance in visual–auditory crossmodal attention switching can be turned into auditory dominance if the task requires temporal processing instead of spatial processing. Hence, task demands impose constraints on crossmodal attentional processing that are in line with the idea of modality-appropriate processing dimensions. Furthermore, our data demonstrate that more sustained attentional processes of giving one modality general priority over another modality (which accounts for modality-specific asymmetries in crossmodal congruence effects) can be dissociated from more transient mechanisms of flexible, trial-totrial biasing of processing to enable intentional (i.e., explicitly cued) modality switches in processing. This interplay of modality-specific, sustained processes and more transient, flexible attentional control can be accounted for in a multiple weighting systems framework.

References Angrilli, A., Cherubini, P., Pavese, A., & Manfredini, S. (1997). The influence of affective factors on time perception. Perception & Psychophysics, 59, 972–982. Aschersleben, G., & Bertelson, P. (2003). Temporal ventriloquism: Crossmodal interaction on the time dimension: 2. Evidence from sensorimotor synchronization. International Journal of Psychophysiology, 50, 157–163.

146

S. Lukas et al. / Acta Psychologica 153 (2014) 139–146

Brown, S.W. (2008). Time and attention: Review of the literature. In S. Grondin (Ed.), Psychology of time (pp. 111–138). Bingley, U. K.: Emerald Group. Chen, K. -M., & Yeh, S. -L. (2009). Asymmetric cross-modal effects in time perception. Acta Psychologica, 130, 225–234. Chen, Q., & Zhou, X. (2013). Vision dominates at their preresponse level and audition dominates at the response level in cross-modal interaction: behavioral and neural evidence. The Journal of Neuroscience, 33, 7109–7121. Colavita, F.B. (1974). Human sensory dominance. Perception & Psychophysics, 16, 409–412. Desimone, R., & Duncan, J. (1995). Neural mechanisms of selective visual attention. Annual Review of Neuroscience, 18, 193–222. Droit-Volet, S., & Meck, W.H. (2007). How emotions colour our perception of time. Trends in Cognitive Science, 11, 504–513. Dyson, B.J., & Quinlan, P.T. (2002). Within- and between-dimensional processing in the auditory modality. Journal of Experimental Psychology: Human Perception and Performance, 28, 1483–1498. Egeth, H.E., & Sager, L.C. (1977). On the locus of visual dominance. Perception & Psychophysics, 22, 77–86. Freides, D. (1974). Human information processing and sensory modality: Cross-modal functions, information complexity, memory, and deficit. Psychological Bulletin, 81, 284–310. Gil, S., & Droit-Volet, S. (2011). “Time flies in the presence of angry faces”… depending on the temporal task used! Acta Psychologica, 136, 354–362. Hunt, A.R., & Kingstone, A. (2004). Multisensory executive functioning. Brain and Cognition, 55, 325–327. Kiesel, A., Steinhauser, M., Wendt, M., Falkenstein, M., Jost, K., Philipp, A.M., & Koch, I. (2010). Control and interference in task switching—A review. Psychological Bulletin, 136, 849–874. Kliegl, K., & Huckauf, A. (2014). Perceived duration decreases with increasing eccentricity. Acta Psychologica, 150, 136–145. Koch, I., Gade, M., Schuch, S., & Philipp, A.M. (2010). The role of inhibition in task switching: A review. Psychonomic Bulletin & Review, 17, 1–14. Koch, I., & Lawo, V. (2014). Exploring temporal dissipation of attention settings in auditory task switching. Attention, Perception, & Psychophysics, 76, 73–80. http://dx.doi.org/ 10.3758/s13414-013-0571-5. Koch, I., Lawo, V., Fels, J., & Vorländer, M. (2011). Switching in the cocktail party— Exploring intentional control of auditory selective attention. Journal of Experimental Psychology: Human Perception and Performance, 37, 1140–1147. http://dx.doi.org/10. 1037/a0022189. Logan, G.D. (2005). The time it takes to switch attention. Psychonomic Bulletin & Review, 12, 647–653. http://dx.doi.org/10.3758/BF03196753. Lukas, S., Philipp, A.M., & Koch, I. (2010a). The role of preparation and cue-modality in crossmodal task switching. Acta Psychologica, 134, 318–322.

Lukas, S., Philipp, A.M., & Koch, I. (2010b). Switching attention between modalities: Further evidence for visual dominance. Psychological Research, 74, 255–267. Macar, F., Grondin, S., & Casini, L. (1994). Controlled attention sharing influences time estimation. Memory & Cognition, 22, 673–686. Meiran, N. (2000). Reconfiguration of stimulus task sets and response task sets during task switching. In S. Monsell, & J. Driver (Eds.), Attention and performance XVIII: Control of cognitive processes (pp. 377–400). Cambridge: MA: MIT Press. Meiran, N., Kessler, Y., & Adi-Japha, E. (2008). Control by Action Representation and Input Selection (CARIS): A theoretical framework for task switching. Psychological Research, 72, 473–500. Müller, H.J., Reimann, B., & Krummenacher, J. (2003). Visual search for singleton feature targets across dimensions: Stimulus- and expectancy-driven effects in dimensional weighting. Journal of Experimental Psychology: Human Perception and Performance, 29, 1021–1035. Murray, M.M., De Santis, L., Thut, G., & Wylie, G.R. (2009). The costs of crossing paths and switching tasks between audition and vision. Brain and Cognition, 69, 47–55. Posner, M.I., Nissen, M.J., & Klein, R.M. (1976). Visual dominance: an informationprocessing account of its origins and significance. Psychological Review, 83, 157–171. Ragot, R., Cave, C., & Fano, M. (1988). Reciprocal effects of visual and auditory stimuli in a spatial compatibility situation. Bulletin of the Psychonomic Society, 26, 350–352. Rangelov, D., Müller, H.J., & Zehetleitner, M. (2012). The multiple-weighting-systems hypothesis: Theory and empirical support. Attention, Perception, & Psychophysics, 74, 540–552. Sandhu, R., & Dyson, B.J. (2012). Re-evaluating visual and auditory dominance through modality switching costs and congruency analyses. Acta Psychologica, 140, 111–118. Shams, L., Kamitani, Y., & Shimojo, S. (2000). What you see is what you hear. Nature, 408, 788. Sinnett, S., Spence, C., & Soto-Faraco, S. (2007). Visual dominance and attention: The Colavita effect revisited. Perception & Psychophysics, 69, 673–686. Spence, C., & Driver, J. (1997). On measuring selective attention to an expected sensory modality. Perception & Psychophysics, 59, 389–403. Stephan, D., & Koch, I. (2010). Central cross-talk in task switching: Evidence from manipulating input–output modality compatibility. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36, 1075–1081. Tse, P.U., Intriligator, J., Rivest, J., & Cavanagh, P. (2004). Attention and the subjective expansion of time. Perception & Psychophysics, 66, 1171–1189. Walker, J.T., & Scott, K.J. (1981). Auditory–visual conflicts in the perceived duration of lights, tones, and gaps. Journal of Experimental Psychology: Human Perception and Performance, 7, 1327–1339. Welch, R.B., DuttonHurt, L.D., & Warren, D.H. (1986). Contributions of audition and vision to temporal rate perception. Perception & Psychophysics, 39, 294–300. Welch, R.B., & Warren, D.H. (1980). Immediate perceptual response to intersensory discrepancy. Psychological Bulletin, 88, 638–667.