Oscillatory Responses to Semantic and Syntactic Violations Aneta Kielar1, Jed A. Meltzer1,2, Sylvain Moreno1,2, Claude Alain1,2, and Ellen Bialystok1,3
Abstract ■ EEG studies employing time–frequency analysis have revealed changes in theta and alpha power in a variety of language and memory tasks. Semantic and syntactic violations embedded in sentences evoke well-known ERPs, but little is known about the oscillatory responses to these violations. We investigated oscillatory responses to both kinds of violations, while monolingual and bilingual participants performed an acceptability judgment task. Both violations elicited power decreases (eventrelated desynchronization, ERD) in the 8–30 Hz frequency range, but with different scalp topographies. In addition, semantic anomalies elicited power increases (event-related synchronization, ERS) in the 1–5 Hz frequency band. The 1–5 Hz ERS was
INTRODUCTION Electroencephalography (EEG) is a popular technique for investigating the neural correlates of language processing. Most commonly, EEG data are analyzed using the ERP technique, in which electrical signals are averaged across trials time-locked to an event such as stimulus presentation or response onset. For instance, semantically unexpected words (e.g., He likes his coffee with cream and dog) elicit the N400 response, a negative-going waveform peaking about 400 msec after word onset, maximal over midline central-parietal electrodes (Kutas & Federmeier, 2000; Osterhout & Nicol, 1999; Kutas & Hillyard, 1980). Syntactic processing has been associated with the early left anterior negativity (about 150–200 msec poststimulus) and the left anterior negativity (about 400 msec poststimulus). The early left anterior negativity has been observed to phrase structure violations or word category errors (Hahne & Friederici, 1999; Friederici, Pfeifer, & Hahne, 1993), whereas left anterior negativity has been observed to a variety of syntactic violations, including morphological agreement violations (Coulson, King, & Kutas, 1998; Münte, Heinze, & Mangun, 1993). In addition, syntactic anomalies such as inappropriately inflected verbs (e.g., The cat will eating the food) elicit
1
Rotman Research Institute, Toronto, Ontario, Canada, 2University of Toronto, 3York University, Toronto, Ontario, Canada © 2014 Massachusetts Institute of Technology
strongly phase-locked to stimulus onset and highly correlated with time domain averages, whereas the 8–30 Hz ERD response varied independently of these. In addition, the results showed that language expertise modulated 8–30 Hz ERD for syntactic violations as a function of the executive demands of the task. When the executive function demands were increased using a grammaticality judgment task, bilinguals but not monolinguals demonstrated reduced 8–30 Hz ERD for syntactic violations. These findings suggest a putative role of the 8–30 Hz ERD response as a marker of linguistic processing that likely represents a separate neural process from those underlying ERPs. ■
the P600, a positive-going waveform with a longer latency than the N400 (∼600 msec) and a slightly more posterior central-parietal distribution (Osterhout & Mobley, 1995; Osterhout & Holcomb, 1992, 1993). The P600 has also been observed for syntactically complex (Kaan, Harris, Gibson, & Holcomb, 2000) or ambiguous structures (Osterhout, Holcomb, & Swinney, 1994) and may index reanalysis and repair of the syntactic structure after a violation has been detected (Friederici, Hahne, & Saddy, 2002; Osterhout & Holcomb, 1992). In addition, the amplitude and latency of these ERPs have been shown to be modulated by subject-specific attributes such as language proficiency (Newman, Tremblay, Nichols, Neville, & Ullman, 2012; Pakulak & Neville, 2010) and age of acquisition (Sanders, Weber-Fox, & Neville, 2008; Weber-Fox & Neville, 1999). Although the ERP method has been extensively utilized in neurolinguistic research, ERPs are only sensitive to neural activity that is phase-locked to the event onset and ignore non-phase-locked activity that is cancelled out by the averaging procedure. As ERP signals depend on precise time-locking to a stimulus, neural events that are delayed or simply more variable in latency generate a smaller signal in the time domain average (Mouraux & Iannetti, 2008) because the peaks and troughs of voltage perturbations do not line up precisely across trials. Therefore, in cases where one group (such as a clinical population) exhibits a decreased ERP response (Kielar, Meltzer-Assher, & Journal of Cognitive Neuroscience 26:12, pp. 2840–2862 doi:10.1162/jocn_a_00670
Thompson, 2012; Kawohl et al., 2010; Swaab, Brown, & Hagoort, 1997), it is difficult to distinguish between reduced neural activity and greater temporal variability using this technique. An attractive alternative approach is time–frequency analysis of the EEG signal. This technique reveals changes in the amplitude of ongoing oscillations, induced by behavioral events (Mouraux & Iannetti, 2008; Pfurtscheller & Lopes da Silva, 1999). Each individual trial is subjected to a moving window spectral analysis technique, such as the short-time Fourier transform or the wavelet transform, resulting in a two-dimensional data matrix on each trial, consisting of spectral power across time points and frequencies. These power values are then averaged across trials and usually normalized as deviations from power in a prestimulus baseline period. One version of this approach, adopted in this study, is known as the eventrelated spectral perturbation (ERSP), a term that encompasses both increases and decreases in power, termed event-related synchronization (ERS) and desynchronization (ERD), respectively. Although ERSP responses are induced by a stimulus in a particular time period (and frequency range), they do not depend on precise phaselocking of the induced amplitude change across trials and may therefore be more sensitive to signal changes that exhibit more temporal variability across trials and participants. Thus, the basic characterization of the ERSP responses to semantic and syntactic violations is an important first step in applying ERSP analysis to the study of the differences in linguistic processing between groups. Thus far, only a handful of ERP studies have examined time–frequency modulations in linguistic violation paradigms. A rather heterogeneous pattern of results has emerged, presumably owing to considerable differences in experimental design across studies. For semantic anomalies that evoke an N400 response, studies have reported both ERS and ERD in the 4–8 Hz theta band (ERS: Davidson & Indefrey, 2007; Hald, Bastiaansen, & Hagoort, 2006; ERD: Allefeld, Frisch, & Schlesewsky, 2005) as well as ERD in the 8–12 Hz alpha band (Willems, Oostenveld, & Hagoort, 2008). For syntactic anomalies, studies have reported theta power ERS (Roehm, Schlesewsky, Bornkessel, Frisch, & Haider, 2004; Bastiaansen, van Berkum, & Hagoort, 2002a) and alpha and beta ERD (Bastiaansen, Magyari, & Hagoort, 2009; Davidson & Indefrey, 2007). Few studies have examined both semantic and syntactic violations within the same paradigm (cf., Davidson & Indefrey, 2007), and still fewer have used the ERSP technique to study differences in linguistic processing across tasks and groups, despite its potential advantages. To investigate the similarities and differences in oscillatory responses related to processing semantic and syntactic information, we applied time–frequency analysis to a multifactorial EEG data set from a previous ERP study (Moreno, Bialystok, Wodniecka, & Alain, 2010), which examined responses to sentence-embedded semantic and syntactic anomalies. In addition, the effects of these
linguistic anomalies were examined in two different tasks and two groups of participants, namely bilinguals and monolinguals. Differences in linguistic processing between these two groups are the subject of a rich behavioral literature and therefore present a good test case to evaluate the potential of the ERSP technique to reveal differences in linguistic processing across groups. For example, bilinguals often show an advantage on tasks requiring control of attention and cognitive flexibility (Moreno, Rodrigues-Fornells, & Laine, 2008; Bialystok, Craik, & Ryan, 2006; Bialystok, Klein, Craik, & Viswanathan, 2004; Bialystok & Martin, 2004; Jackson, Swainson, Cunnington, & Jackson, 2001). Furthermore, electrophysiological studies have reported differences related to bilingualism in ERP patterns for processing of semantic and syntactic information (e.g., Moreno & Kutas, 2005; Neville & Weber-Fox, 1996). In some studies, the N400 effect has been found to be delayed in bilinguals for their second, less dominant language (Moreno & Kutas, 2005; Hahne, 2001; Neville & Weber-Fox, 1996; Ardal, Donald, Meuter, Muldrew, & Luce, 1990). Other studies reported either a delayed or significantly reduced P600 effect in bilinguals (Hahne, 2001; Hahne & Friederici, 2001; Neville & Weber-Fox, 1996). However, Osterhout and colleagues (Osterhout et al., 2008; Kotz, Holcomb, & Osterhout, 2007) found similar P600 responses elicited by syntactic violations in monolinguals and bilinguals. In this study, we reanalyzed the EEG data from the Moreno et al. (2010) study using a time–frequency analysis. There were two main goals of this study. The first was to characterize the nature of the ERSP responses in this paradigm. This experiment featured both semantic and syntactic anomalies occurring in the same task, the same part of speech, and the same position in the sentence. Given the heterogeneous nature of the findings in previous studies of ERSP in violation paradigms, this data set offers a good opportunity to compare the brainʼs response to both kinds of anomalies under similar conditions. We sought to establish the frequencies that are modulated by linguistic violations, the timing of the responses, the directionality (power increase or decrease), and the scalp topography. In addition, we directly investigated the relationship between oscillatory activity and time-locked responses present in the ERP signal evoked by the same semantic and syntactic anomalies. The second goal was to compare the strength of the ERSP response across tasks and groups to see if the previous findings with the ERP technique are replicated using ERSP analysis. Moreno et al. (2010) found equivalent responses to semantic and syntactic anomalies for bilinguals and monolinguals on the acceptability judgment task, but different responses on the grammaticality judgment task, consistent with the behavioral advantage for bilinguals on the latter task. We tested whether a similar dissociation was present in the ERSP data and whether the directionality of the effects matched that seen in the ERP data. By comparing ERSP responses across groups and tasks on Kielar et al.
2841
an established paradigm that distinguishes these groups, we hope to validate the use of ERSP as a routine tool for assessing differences in linguistic processing across other groups of participants, particularly in clinical populations.
METHODS Participants EEG data were obtained from 14 healthy English monolingual (mean age = 23.06 ± 3.1 years, seven men) and 14 bilingual (mean age = 23.5 ± 4.5 years, two men) university students. Monolingual participants were all born and raised in either Canada or the United States. Bilingual participants were born in Canada (4), Russia (1), Romania (1), or Israel (8). Twelve of the 14 bilingual participants learned their second language (L2) before the age of 12 (mean = 6), and two participants learned English later in life, one at 15 years old and the second at 14 years old. On average, bilinguals spoke English at home 26% of the time and at work 85% of the time. Similarly, bilinguals heard English at home 33% of the time and at work 89% of the time. One bilingual reported to have English as first language, and six bilinguals considered English as their dominant language. Their average proficiency self-ratings on a 0–100% scale (100% corresponding to native proficiency) was 97.1% for dominant and 81.9% for nondominant language. All participants reported normal vision and provided informed consent before the start of the study, in line with the ethical guidelines outlined by the University of Toronto and Baycrest Centre for Geriatric Care. More detailed demographic and behavioral profiles for each group are presented in Moreno et al. (2010). Materials The materials are described in detail in Moreno et al. (2010), and examples are presented in Table 1. One hundred twenty sentence frames, originally used in Osterhout and Nicol (1999), were used to construct sentences that were correct, grammatically incorrect but semantically acceptable, and semantically anomalous but grammatically correct. A fourth condition containing words that were both semantically and syntactically anomalous was included, but treated as fillers and not analyzed, because the semantic and syntactic effects could not be unambiguously isolated. The responses to semantically or syntactically anomalous words were compared
with nonanomalous words in the same sentence position. The semantically anomalous sentences were created by introducing a mismatch in animacy between the verb and the agent of the sentence (e.g., computer–paint), creating a selectional restriction violation. The syntactically incorrect structures were formed by introducing verb tense violations (e.g., will lasting). The correct and semantically inconsistent critical words were matched on frequency (semantically acceptable: mean = 96, semantically anomalous: mean = 70, p > .2; Kučera & Francis, 1967) and length (semantically acceptable: mean letters = 4.94, semantically anomalous: mean letters = 4.52, p > .03). These experimental materials were used to create three stimulus lists. Each list contained 30 exemplars of each of the four experimental sentence types. Items were counterbalanced between lists such that only one version of each sentence was presented on a given list. Each participant was presented with one list of 120 sentences for the grammaticality task and a separate list of 120 sentences for the acceptability task. Sentence frames were repeated between the two tasks, but appeared in different conditions, and lists were counterbalanced across participants.
Procedure Cognitive Assessment Before EEG recording, participants completed a battery of psychological tests including the Language and Social Background Questionnaire, Peabody Picture Vocabulary Test (PPVT)-III (Dunn & Dunn, 1997), Cattell Culture Fair Intelligence (Cattell, 1957), and Corsi Block tests (used after Fischer, 2001). The purpose of these tests was to assess the similarity of the two participant groups on vocabulary (PPVT), intelligence (Cattell), and working memory (Corsi Block test). The cognitive assessment session lasted approximately 45 min. EEG Recording during Sentence Judgment Tasks The EEG recording was conducted in a soundproof room. On each trial, participants were presented with a 500-msec fixation cross, followed by a sentence. Sentences were presented word by word, with each word appearing in the center of the screen for 300 msec. Words were separated by a blank screen interval of 350 msec. A 1450-msec blank screen interval was inserted at the end of each sentence, after which a response prompt appeared. In this study, the processing demands were manipulated by
Table 1. Examples of Sentences Used in the Acceptability and Grammaticality Tasks Semantically Correct
Semantically Incorrect
Syntax correct
A new computer will last for many years
A new computer will paint for many years
Syntax incorrect
A new computer will lasting for many years
A new computer will painting for many years
2842
Journal of Cognitive Neuroscience
Volume 26, Number 12
varying task instructions. Participants completed both a standard acceptability judgment task and the grammaticality judgment paradigm previously used by Bialystok and colleagues (Bialystok & Majumder, 1998; Bialystok, 1986). In the acceptability task, the participants were instructed to answer “yes” if they judged the sentence to be acceptable and “no” if the sentence was either semantically or grammatically incorrect. In the grammaticality task, participants responded “yes” if the sentence was grammatically correct and “no” if it was grammatically incorrect, irrespective of meaning. Thus, in the grammaticality task, sentences such as “A new computer will paint for many years” would be judged as correct, whereas the same sentence will be judged as incorrect in the acceptability task. These manipulations created a higher level of attentional control in the grammaticality task, because it was necessary to attend to only one dimension in spite of conflicting information from the other dimension. On the basis of prior behavioral studies (Bialystok & Majumder, 1998; Bialystok, 1986), in the grammaticality task, a higher degree of executive control is required to attend to syntax when the meaning is inconsistent. It is more difficult to approve the grammatically correct sentences and ignore the conflicting semantic information. Participants used two different hands to respond. They indicated their responses by pressing one button for “yes” and another one for “no,” with the left- and right-hand assignments counterbalanced across participants. For all participants, the grammaticality task was performed first. Participants had a brief break ( .05). The two language groups differed on the PPVT scores, F(1, 27) = 5.36, p < .005, with monolinguals obtaining higher scores than bilinguals. Accuracy rates on the grammaticality and acceptability judgment tasks (performed during EEG acquisition) were analyzed with repeated-measures ANOVA with Condition as a within-subject factor and Group as a between-subject factor. Before the ANOVA, the proportion correct for each participant and condition was subjected to the arcsinesquare-root transformation (Sokal & Rohlf, 1995), with values of 1 replaced by [1 − (1/(4n)], with n equal to the number of trials in each condition (30). The untransformed accuracy scores and standard deviations for the two groups are presented in Table 3. In the acceptability task, there was a significant main effect of group, indicating that monolinguals were significantly more accurate than bilinguals, F(1, 26) = 4.66, p < .05. The main effect of Condition was also significant, F(2, 52) = 14.98, p < 0 .001, reflecting overall higher accuracy for correct and syntactically incorrect sentences than for semantically anomalous structures (both ps < .001). There were no significant dif-
ferences between correct and syntactically anomalous sentences ( p > .05), and the Group × Condition interaction was not significant, F < 1. In the grammaticality task, there were no significant main effects of Group, F(1, 26) = 1.755, p > .05, or condition, F < 1. The Group × Condition interaction was not significant (F < 1), indicating that in the grammaticality task monolinguals and bilinguals did not differ in their ability to judge the syntax of the sentences.
Oscillatory Responses: ERSP Illustration of ERSP Characteristics The primary statistical analysis of ERSP responses in this experiment concerns the difference in the responses between anomalous and correct words in the same sentence position. However, as time–frequency analysis is somewhat less familiar than the ERP technique to most neuroscientists, we chose to illustrate additional characteristics of the responses measured in this study, although they are not the direct subject of statistical testing. We wish to demonstrate to the reader in a qualitative fashion how ongoing oscillations are modulated in a rapid serial visual presentation sentence comprehension task, both in correct sentences and in the presence of violations. Averaged time–frequency decompositions for one representative electrode, P5, are presented in Figure 2A. These are averaged across all 28 participants (both monolinguals and bilinguals) on data from the acceptability task. Power in the alpha and beta bands (approximately 6–25 Hz) is modulated by the periodic presentation of words in the sentence. Additionally, there is a general trend for power to decrease from the beginning to the end of the sentence. Superimposed on these two effects, linguistic
Table 3. The Mean Accuracy (Standard Deviations) Judgment Scores in Percent Correct for Each Language Group in the Acceptability and Grammaticality Tasks Language Group Monolinguals
Bilinguals
Condition
Acceptability Mean (SD)
Grammaticality Mean (SD)
Correct
95 (2)
96 (1)
Semantic errors
87 (3)
95 (2)
Syntactic errors
95 (2)
96 (2)
Correct
90 (2)
95 (1)
Semantic errors
81 (3)
93 (2)
Syntactic errors
93 (2)
93 (2)
Kielar et al.
2845
Figure 2. Example representations of oscillatory reactivity (ERD and ERS) in the acceptability task, combined across monolinguals and bilinguals. (A) Time–frequency maps of ERSP signal time-locked to control, semantic violations, and syntactic violations shown at one representative electrode, P5. (B) Time courses of 8–30 Hz power decreases at site P4, time-locked to the critical word onset. Peaks and troughs of power before and after the critical word are because of visual presentation of other words in the sentence. (C) Time courses of 8–30 Hz power decreases at site P5, time-locked to critical word onset. (D) Time courses of 1–5 Hz power increases at site Pz, time-locked to critical word onset.
violations induce a sharp drop in power that is maximal approximately 500–1500 msec after the onset of the critical word. The drop is larger for semantic violations than syntactic ones. For semantic violations, we also observe an increase in power in the delta and theta bands (approximately 1–5 Hz), which is consistent with some prior reports (Davidson & Indefrey, 2007; Hald et al., 2006). To avoid potential overlap with power increase in the theta range, we chose a lower limit of 8 Hz for our ANOVAs on differences in the alpha-beta power decrease across tasks and groups. Similarly, for delta-theta power increases, a 1–5 Hz frequency range was used for computing ANOVAs. Note that the 1–5 Hz ERS response has a slightly earlier 2846
Journal of Cognitive Neuroscience
onset and peak than the 8–30 Hz ERD on most electrodes. However, the statistical analyses (see below) indicated that significant differences in ERS between the conditions are predominantly observed in the same range as the ERD, 500–1500 msec after the onset of the critical word. Figures 2B and 2C displays the time course of power fluctuations in each condition, averaged over the 8–30 Hz range, at two representative electrodes (P4 and P5). These time course plots allow the reader to easily discern the three patterns of modulation present in the 8–30 Hz range: periodic rise-and-fall driven by the stimulus presentation, an overall decreasing trend across the sentence, and a specific decrease induced by linguistic violations. In this Volume 26, Number 12
example, P4 (right parietal) responds mainly to semantic violations, whereas P5 (left parietal) responds to both semantic and syntactic violations. Similarly, Figure 2D displays the time course of power fluctuations in the 1–5 Hz frequency band for semantic and syntactic violations at one of the central electrodes (Pz) that showed the strongest responses in this frequency range. The 1–5 Hz response is specifically modulated by semantic violations but is not periodically driven by the visual word presentation. Cluster Analysis Results For semantic violations versus control words, we observed two significant clusters of ERSP differences. Semantically anomalous words induced a power decrease (ERD), occur-
ring between 500 and 1500 msec, in the frequency range of 6–30 Hz (Figure 3A). In addition, semantic violations elicited a power increase (ERS) in the time range of 500– 1500 msec, in the frequency range of 1–5 Hz (Figure 3B). To simplify the display of this cluster across the dimensions of time, frequency, and scalp topography, we computed a measure that we refer to as T-sum. This is the sum of t values for all data points (time, frequency, and electrode) that are part of the cluster (Meltzer & Braun, 2011). By summing across different dimensions of the cluster, different aspects of the cluster can be highlighted. Note that this procedure of summing t values across individual dimensions is done solely for purposes of summarizing the clusterʼs spatiotemporal characteristics in a format convenient for display and does not play a role in the
Figure 3. Significant clusters of EEG oscillatory reactivity in the acceptability task, combined across monolinguals and bilinguals. (A) t values of 8–30 Hz ERD for semantic violations versus control words, summed across all channels at each time and frequency point and the scalp topography of the one significant cluster of ERD for semantic violations versus control words, represented as sums of t values across time and frequency points at each electrode. (B) t values of 1–5 Hz ERS for semantic violations versus control words and the scalp topography of the one significant cluster of ERS for semantic violations versus control words. (C) t values of 8–30 Hz ERD for syntactic violations versus control words and the scalp topography of the one significant cluster of ERD for syntactic violations versus control words.
Kielar et al.
2847
assessment of cluster significance as a whole. Figure 3A and B shows the T-sum for the clusters of ERD and ERS in response to semantic violations versus control words. At each time–frequency point, significant t values are summed across electrodes. A given point can achieve a higher t score either by having more significant electrodes or by having higher t values at the individual electrodes (generally both). Thus, the region of maximal response can be easily discerned. For the comparison of syntactic violations versus control words, we obtained one significant cluster, in the same time range of 500–1500 msec, in the frequency range of 6–22 Hz (see Figure 3C). Thus, both kinds of violations induced ERD in largely overlapping time and frequency ranges, although the response to semantic violations was larger and extended to higher frequencies. The observed frequency range includes the commonly defined alpha band (8–12 Hz) as well as the beta band (15–30 Hz). In addition, semantic violations induced ERS in the delta and theta bands (1–5 Hz). To illustrate the topography of these clusters, we computed T-sum values at each electrode, summing over both time and frequency dimensions. The resulting topographies of ERD and ERS for semantic violations are shown in Figure 3A and B. ERD induced by semantic violations was maximal over bilateral parietal electrodes, whereas ERS was maximal at the frontal and central sites, with additional increases observed at the temporal sites. The topography of ERD for syntactic violations is shown in Figure 3C. The distribution of the syntactic response appeared to be somewhat more left-lateralized and also stronger at bilateral fronto-central electrodes. However, the statistical significance of differences in topography cannot be tested directly from the cluster analysis, which operates on each electrode separately. Because the cluster analysis identified consistent time–frequency windows in which ERSP responses to linguistic anomalies occurred, we then conducted repeated-measures ANOVAs on ERSP values averaged within the relevant windows to formally assess differences in scalp topography and effects of bilingualism within each task. Relationship between ERP and ERSP Measures Because the cluster analysis revealed significant time– frequency responses in the alpha-beta (8–30 Hz) and delta-theta (1–5 Hz) frequency bands, we investigated the extent to which these ERSP responses may be driven by time-locked signals. To do that, we performed a time– frequency analysis on the ERP signal using the same procedure as was applied to compute ERD and ERS responses but applied to the signal after averaging across trials within each participant. This “ERSP of ERPs” analysis was computed for the acceptability task across all 28 participants. The analysis revealed power increases for both semantic and syntactic anomalies from 1 to 5 Hz spanning 500– 1500 msec time window (see Figure 4A). These effects 2848
Journal of Cognitive Neuroscience
were the strongest over the central electrode sites and were greater for semantic than syntactic violations (see Figure 4A and B). Correlations were computed between the ERSP measures (8–30 Hz ERD and 1–5 Hz ERS) and the ERSP of ERPs effects in the 1–5 Hz frequency range and within the 500–1500 msec time window. These analyses were computed for the acceptability task, separately for semantic and syntactic violations on the difference scores (semanticcontrol; syntactic-control) across the 28 participants. The correlations were computed on the average response combined across all electrodes and also separately within each of the previously defined 11 electrode clusters. The results are presented in Table 4. The analysis revealed significant correlations between the 1–5 Hz ERS and the “ERSP of ERPs” signals. For semantic violations, significant positive correlations were present when computed across all electrodes, as well as separately at the central, parietal, and frontal regions. These positive correlations reflect correspondence between power increases in 1–5 Hz frequency range for the total time–frequency response (ERS) and the portion of it that is attributable to phase-locked signals (ERSP of ERPs). For syntactic violations, significant correlations were also found at the left central parietal, left frontal, and parietal regions, but these correlations were somewhat weaker than those seen for semantic violations and did not meet the Bonferroni-corrected significance threshold for multiple tests done in 11 electrode regions. Also, correlation in the average signal across all electrodes was not significant for syntactic violations. The same analysis was performed for 8–30 Hz ERD. However, the analysis revealed that there were no significant correlations between the 8–30 Hz ERD and the ERSP of ERPs measures. These results indicate that 1–5 Hz ERS, but not 8–30 Hz ERD, correlated with power increases in the 1–5 Hz frequency range present in the ERP signals. Comparison of Figure 4B and C illustrates the similarity between 1–5 Hz ERS power computed over single trials (Figure 4C) and the time–frequency characteristics of the ERP signal averaged across trials (Figure 4B). Topographical maps of the correlations at each electrode are presented in Figure 4D for the two comparisons. Electrodes showing significant correlations are highlighted at both an uncorrected level ( p < .05) and a level Bonferronicorrected for multiple comparisons across individual electrodes ( p < .0008). We also computed correlations between 8–30 Hz ERD and 1–5 Hz ERS. This analysis revealed one significant positive correlation for semantic violation limited to the left central-parietal region, but this correlation did not survive Bonferroni correction for multiple comparisons over the 11 electrode regions. No other correlations were significant, and there were no significant correlations for the syntactic anomalies. These results indicate that participants who exhibit a strong 1–5 Hz ERS response in single trials also tend to exhibit a strong response in the time domain average or Volume 26, Number 12
Figure 4. Representation of oscillatory activity obtained from the ERP signal using time–frequency analysis in the acceptability task, combined across monolinguals and bilinguals, compared with 1–5 Hz ERS. (A) Time–frequency maps of ERSP of ERP signals time-locked to control, semantic violations, and syntactic violations shown at one representative electrode, CPz. (B) Time courses of 1–5 Hz ERSP of ERPs at central site CPz, time-locked to the critical word onset, and the scalp topography for semantic violations versus control in 1–5 Hz band at 500–1500 msec time window. (C) Time courses of 1–5 Hz ERS at central site CPz, time-locked to the critical word onset, and the scalp topographies for semantic violations versus control at 500–1500 msec time window. (D) The topographical maps of correlations computed for semantic violation versus controls between ERSP measures (1–5 Hz ERS and 8–30 Hz ERD) and the ERSP of ERPs effects, in the 1–5 Hz frequency range. The electrodes showing significant correlations are marked with symbols (black diamonds: p < .05, uncorrected; blue circles: p < .0008, Bonferroni-corrected).
Kielar et al.
2849
Table 4. Correlations between ERSP of ERPs and 1–5 Hz ERS, 8–30 Hz ERD and between 8–30 Hz ERD and 1–5 Hz ERS Computed for Semantic and Syntactic Anomalies in Acceptability Task, Combined across Monolinguals and Bilinguals ERSP of ERPs vs. 1–5 Hz ERS Condition
Region
Semantic
Average CP PP
Significance (p)
Region
0.434
.021a
Average
0.113
ns
0.454
Correlation (r)
8–30 Hz ERD vs. 1–5 Hz ERS
Significance (p)
Region
0.001
ns
Average
0.226
ns
Correlation
Significance (p)
CP
−0.120
ns
CP
0.344
ns
.015
a
PP
0.169
ns
PP
0.117
ns
a
RCP
0.126
ns
RCP
0.170
ns
LCP
0.090
ns
LCP
0.454
0.015a
RCP
0.379
.047
LCP
0.190
ns
FC
0.550
.002bc
FC
−0.005
ns
FC
−0.238
ns
LF
0.568
.002
bc
LF
−0.104
ns
LF
0.138
ns
RF
0.353
ns
RF
0.214
ns
RF
0.005
ns
RFC
−0.287
ns
RFC
−0.230
ns
RFC
0.384
.044
LFC
0.345
ns
RP
Syntactic
Correlation (r)
ERSP of ERPs vs. 8–30 Hz ERD
0.467
a
LFC
−0.231
ns
LFC
0.252
ns
.012
a
RP
−0.097
ns
RP
0.060
ns
a
LP
0.169
ns
LP
0.340
ns
LP
0.469
.012
Average
0.116
ns
Average
0.010
ns
Average
0.179
ns
CP
0.074
ns
CP
0.124
ns
CP
0.254
ns
PP
0.291
ns
PP
0.081
ns
PP
0.058
ns
RCP
0.144
ns
RCP
0.071
ns
RCP
0.169
ns
LCP
0.128
ns
LCP
0.269
ns
FC
−0.268
ns
FC
0.223
ns
LF
−0.221
ns
LF
−0.066
ns
−0.020
ns
LCP FC
0.451 −0.062
.016
a
ns a
LF
0.432
.022
RF
0.061
ns
RF
−0.006
ns
RF
RFC
0.162
ns
RFC
−0.123
ns
RFC
0.108
ns
LFC
0.427
.023a
LFC
0.086
ns
LFC
0.227
ns
RP
0.175
ns
RP
−0.107
ns
RP
0.036
ns
LP
0.048
ns
LP
0.000
ns
LP
0.413
.029
a
The analyses were computed on the mean amplitude of the difference scores as averaged across all electrode regions and separately at each of the 11 electrode regions. The significant correlations are marked in bold font. The electrodes showing significant correlations are marked (a: p < .05, uncorrected; bc: p < .004, Bonferroni corrected).
ERP, whereas no such relationship was apparent with the 8–30 Hz ERD. This pattern is consistent with a common neural process driving the expression of 1–5 Hz ERS and ERPs, distinct from that underlying the 8–30 Hz ERD. These 1–5 Hz power increases represent phase-locked signals evoked by the violations but collapse across polarity, as both negative and positive components contain power. Thus, it is possible to observe significant ERS in the delta and theta ranges, even if ERP responses are highly variable across participants. The relationship between ERP responses to semantic violations, 1–5 Hz ERSP of ERPs, and 1–5 Hz ERS are presented in Figure 5. Figure 5A illustrates the topographical distribution of ERP 2850
Journal of Cognitive Neuroscience
responses to semantic violations versus control words. In this experiment, semantic violations evoked both negative (N400) and positive (P600) waveforms at different electrode sites, with a biphasic waveform at central sites. Figure 5B shows that 1–5 Hz ERSP of ERPs picks up signals of either polarity, both positive and negative. The strong correlations observed across participants (Figure 4) suggest that ERP components of either polarity contribute to the 1–5 Hz ERS seen in the time–frequency analyses. For a representative electrode C3, we display the time courses of ERP, 1–5 Hz ERSP of ERPs, and 1–5 Hz ERS responses in Figure 5C, D, and E, respectively. These graphs illustrate that the violation induced both negative and Volume 26, Number 12
positive ERP components, both of which are reflected in 1–5 Hz ERS. Statistical Analysis of 8–30 Hz ERD and 1–5 Hz ERS To investigate the scalp distributions of ERD and ERS effects and to test for significant interactions between violation conditions and language groups in the two tasks, we chose to analyze ERD in the time window of 500–
1500 msec and frequency range of 8–30 Hz. In addition, ERS was analyzed in the 1–5 Hz frequency range and 500–1500 msec time window. This range comprises the bulk of the significant clusters (Figure 3) and is consistent with previous studies reporting modulations of brain oscillations in this range by language tasks (Willems et al., 2008; Davidson & Indefrey, 2007). Although significant ERD does extend below 8 Hz at some electrodes, we also observed ERS up to about 6 Hz in semantic condition at
Figure 5. The relationship between ERP responses to semantic violations, 1–5 ERSP of ERPs and 1–5 Hz ERS. (A) The topographical distribution of ERP responses to semantic violations versus control words. Semantic violations evoked both negative (N400) and positive (P600) waveforms at different electrode sites, with a biphasic waveform at central sites. (B) The topographical distribution of 1–5 Hz ERSP of ERPs responses to semantic violations versus control words. 1–5 Hz ERSP of ERPs picks up signals of either polarity, both positive and negative. (C) Time courses of ERPs at C3. (D) Time courses of 1–5 Hz ERSP of ERPs at C3. (E) Time courses of 1–5 Hz ERS at C3.
Kielar et al.
2851
Table 5. Statistical Results in the 500–1500 msec Time Window A. Effects of Violation Condition and Region in Acceptability Task Frequency Range 8–30 Hz ERD
1–5 Hz ERS
df
F
p
Violation condition (semantic vs. syntactic)
(1,27)
9.73
.004
Region
(10,270)
2.48
.019
Condition × Region
(10,270)
6.65
Abstract ■ EEG studies employing time–frequency analysis have revealed changes in theta and alpha power in a variety of language and memory tasks. Semantic and syntactic violations embedded in sentences evoke well-known ERPs, but little is known about the oscillatory responses to these violations. We investigated oscillatory responses to both kinds of violations, while monolingual and bilingual participants performed an acceptability judgment task. Both violations elicited power decreases (eventrelated desynchronization, ERD) in the 8–30 Hz frequency range, but with different scalp topographies. In addition, semantic anomalies elicited power increases (event-related synchronization, ERS) in the 1–5 Hz frequency band. The 1–5 Hz ERS was
INTRODUCTION Electroencephalography (EEG) is a popular technique for investigating the neural correlates of language processing. Most commonly, EEG data are analyzed using the ERP technique, in which electrical signals are averaged across trials time-locked to an event such as stimulus presentation or response onset. For instance, semantically unexpected words (e.g., He likes his coffee with cream and dog) elicit the N400 response, a negative-going waveform peaking about 400 msec after word onset, maximal over midline central-parietal electrodes (Kutas & Federmeier, 2000; Osterhout & Nicol, 1999; Kutas & Hillyard, 1980). Syntactic processing has been associated with the early left anterior negativity (about 150–200 msec poststimulus) and the left anterior negativity (about 400 msec poststimulus). The early left anterior negativity has been observed to phrase structure violations or word category errors (Hahne & Friederici, 1999; Friederici, Pfeifer, & Hahne, 1993), whereas left anterior negativity has been observed to a variety of syntactic violations, including morphological agreement violations (Coulson, King, & Kutas, 1998; Münte, Heinze, & Mangun, 1993). In addition, syntactic anomalies such as inappropriately inflected verbs (e.g., The cat will eating the food) elicit
1
Rotman Research Institute, Toronto, Ontario, Canada, 2University of Toronto, 3York University, Toronto, Ontario, Canada © 2014 Massachusetts Institute of Technology
strongly phase-locked to stimulus onset and highly correlated with time domain averages, whereas the 8–30 Hz ERD response varied independently of these. In addition, the results showed that language expertise modulated 8–30 Hz ERD for syntactic violations as a function of the executive demands of the task. When the executive function demands were increased using a grammaticality judgment task, bilinguals but not monolinguals demonstrated reduced 8–30 Hz ERD for syntactic violations. These findings suggest a putative role of the 8–30 Hz ERD response as a marker of linguistic processing that likely represents a separate neural process from those underlying ERPs. ■
the P600, a positive-going waveform with a longer latency than the N400 (∼600 msec) and a slightly more posterior central-parietal distribution (Osterhout & Mobley, 1995; Osterhout & Holcomb, 1992, 1993). The P600 has also been observed for syntactically complex (Kaan, Harris, Gibson, & Holcomb, 2000) or ambiguous structures (Osterhout, Holcomb, & Swinney, 1994) and may index reanalysis and repair of the syntactic structure after a violation has been detected (Friederici, Hahne, & Saddy, 2002; Osterhout & Holcomb, 1992). In addition, the amplitude and latency of these ERPs have been shown to be modulated by subject-specific attributes such as language proficiency (Newman, Tremblay, Nichols, Neville, & Ullman, 2012; Pakulak & Neville, 2010) and age of acquisition (Sanders, Weber-Fox, & Neville, 2008; Weber-Fox & Neville, 1999). Although the ERP method has been extensively utilized in neurolinguistic research, ERPs are only sensitive to neural activity that is phase-locked to the event onset and ignore non-phase-locked activity that is cancelled out by the averaging procedure. As ERP signals depend on precise time-locking to a stimulus, neural events that are delayed or simply more variable in latency generate a smaller signal in the time domain average (Mouraux & Iannetti, 2008) because the peaks and troughs of voltage perturbations do not line up precisely across trials. Therefore, in cases where one group (such as a clinical population) exhibits a decreased ERP response (Kielar, Meltzer-Assher, & Journal of Cognitive Neuroscience 26:12, pp. 2840–2862 doi:10.1162/jocn_a_00670
Thompson, 2012; Kawohl et al., 2010; Swaab, Brown, & Hagoort, 1997), it is difficult to distinguish between reduced neural activity and greater temporal variability using this technique. An attractive alternative approach is time–frequency analysis of the EEG signal. This technique reveals changes in the amplitude of ongoing oscillations, induced by behavioral events (Mouraux & Iannetti, 2008; Pfurtscheller & Lopes da Silva, 1999). Each individual trial is subjected to a moving window spectral analysis technique, such as the short-time Fourier transform or the wavelet transform, resulting in a two-dimensional data matrix on each trial, consisting of spectral power across time points and frequencies. These power values are then averaged across trials and usually normalized as deviations from power in a prestimulus baseline period. One version of this approach, adopted in this study, is known as the eventrelated spectral perturbation (ERSP), a term that encompasses both increases and decreases in power, termed event-related synchronization (ERS) and desynchronization (ERD), respectively. Although ERSP responses are induced by a stimulus in a particular time period (and frequency range), they do not depend on precise phaselocking of the induced amplitude change across trials and may therefore be more sensitive to signal changes that exhibit more temporal variability across trials and participants. Thus, the basic characterization of the ERSP responses to semantic and syntactic violations is an important first step in applying ERSP analysis to the study of the differences in linguistic processing between groups. Thus far, only a handful of ERP studies have examined time–frequency modulations in linguistic violation paradigms. A rather heterogeneous pattern of results has emerged, presumably owing to considerable differences in experimental design across studies. For semantic anomalies that evoke an N400 response, studies have reported both ERS and ERD in the 4–8 Hz theta band (ERS: Davidson & Indefrey, 2007; Hald, Bastiaansen, & Hagoort, 2006; ERD: Allefeld, Frisch, & Schlesewsky, 2005) as well as ERD in the 8–12 Hz alpha band (Willems, Oostenveld, & Hagoort, 2008). For syntactic anomalies, studies have reported theta power ERS (Roehm, Schlesewsky, Bornkessel, Frisch, & Haider, 2004; Bastiaansen, van Berkum, & Hagoort, 2002a) and alpha and beta ERD (Bastiaansen, Magyari, & Hagoort, 2009; Davidson & Indefrey, 2007). Few studies have examined both semantic and syntactic violations within the same paradigm (cf., Davidson & Indefrey, 2007), and still fewer have used the ERSP technique to study differences in linguistic processing across tasks and groups, despite its potential advantages. To investigate the similarities and differences in oscillatory responses related to processing semantic and syntactic information, we applied time–frequency analysis to a multifactorial EEG data set from a previous ERP study (Moreno, Bialystok, Wodniecka, & Alain, 2010), which examined responses to sentence-embedded semantic and syntactic anomalies. In addition, the effects of these
linguistic anomalies were examined in two different tasks and two groups of participants, namely bilinguals and monolinguals. Differences in linguistic processing between these two groups are the subject of a rich behavioral literature and therefore present a good test case to evaluate the potential of the ERSP technique to reveal differences in linguistic processing across groups. For example, bilinguals often show an advantage on tasks requiring control of attention and cognitive flexibility (Moreno, Rodrigues-Fornells, & Laine, 2008; Bialystok, Craik, & Ryan, 2006; Bialystok, Klein, Craik, & Viswanathan, 2004; Bialystok & Martin, 2004; Jackson, Swainson, Cunnington, & Jackson, 2001). Furthermore, electrophysiological studies have reported differences related to bilingualism in ERP patterns for processing of semantic and syntactic information (e.g., Moreno & Kutas, 2005; Neville & Weber-Fox, 1996). In some studies, the N400 effect has been found to be delayed in bilinguals for their second, less dominant language (Moreno & Kutas, 2005; Hahne, 2001; Neville & Weber-Fox, 1996; Ardal, Donald, Meuter, Muldrew, & Luce, 1990). Other studies reported either a delayed or significantly reduced P600 effect in bilinguals (Hahne, 2001; Hahne & Friederici, 2001; Neville & Weber-Fox, 1996). However, Osterhout and colleagues (Osterhout et al., 2008; Kotz, Holcomb, & Osterhout, 2007) found similar P600 responses elicited by syntactic violations in monolinguals and bilinguals. In this study, we reanalyzed the EEG data from the Moreno et al. (2010) study using a time–frequency analysis. There were two main goals of this study. The first was to characterize the nature of the ERSP responses in this paradigm. This experiment featured both semantic and syntactic anomalies occurring in the same task, the same part of speech, and the same position in the sentence. Given the heterogeneous nature of the findings in previous studies of ERSP in violation paradigms, this data set offers a good opportunity to compare the brainʼs response to both kinds of anomalies under similar conditions. We sought to establish the frequencies that are modulated by linguistic violations, the timing of the responses, the directionality (power increase or decrease), and the scalp topography. In addition, we directly investigated the relationship between oscillatory activity and time-locked responses present in the ERP signal evoked by the same semantic and syntactic anomalies. The second goal was to compare the strength of the ERSP response across tasks and groups to see if the previous findings with the ERP technique are replicated using ERSP analysis. Moreno et al. (2010) found equivalent responses to semantic and syntactic anomalies for bilinguals and monolinguals on the acceptability judgment task, but different responses on the grammaticality judgment task, consistent with the behavioral advantage for bilinguals on the latter task. We tested whether a similar dissociation was present in the ERSP data and whether the directionality of the effects matched that seen in the ERP data. By comparing ERSP responses across groups and tasks on Kielar et al.
2841
an established paradigm that distinguishes these groups, we hope to validate the use of ERSP as a routine tool for assessing differences in linguistic processing across other groups of participants, particularly in clinical populations.
METHODS Participants EEG data were obtained from 14 healthy English monolingual (mean age = 23.06 ± 3.1 years, seven men) and 14 bilingual (mean age = 23.5 ± 4.5 years, two men) university students. Monolingual participants were all born and raised in either Canada or the United States. Bilingual participants were born in Canada (4), Russia (1), Romania (1), or Israel (8). Twelve of the 14 bilingual participants learned their second language (L2) before the age of 12 (mean = 6), and two participants learned English later in life, one at 15 years old and the second at 14 years old. On average, bilinguals spoke English at home 26% of the time and at work 85% of the time. Similarly, bilinguals heard English at home 33% of the time and at work 89% of the time. One bilingual reported to have English as first language, and six bilinguals considered English as their dominant language. Their average proficiency self-ratings on a 0–100% scale (100% corresponding to native proficiency) was 97.1% for dominant and 81.9% for nondominant language. All participants reported normal vision and provided informed consent before the start of the study, in line with the ethical guidelines outlined by the University of Toronto and Baycrest Centre for Geriatric Care. More detailed demographic and behavioral profiles for each group are presented in Moreno et al. (2010). Materials The materials are described in detail in Moreno et al. (2010), and examples are presented in Table 1. One hundred twenty sentence frames, originally used in Osterhout and Nicol (1999), were used to construct sentences that were correct, grammatically incorrect but semantically acceptable, and semantically anomalous but grammatically correct. A fourth condition containing words that were both semantically and syntactically anomalous was included, but treated as fillers and not analyzed, because the semantic and syntactic effects could not be unambiguously isolated. The responses to semantically or syntactically anomalous words were compared
with nonanomalous words in the same sentence position. The semantically anomalous sentences were created by introducing a mismatch in animacy between the verb and the agent of the sentence (e.g., computer–paint), creating a selectional restriction violation. The syntactically incorrect structures were formed by introducing verb tense violations (e.g., will lasting). The correct and semantically inconsistent critical words were matched on frequency (semantically acceptable: mean = 96, semantically anomalous: mean = 70, p > .2; Kučera & Francis, 1967) and length (semantically acceptable: mean letters = 4.94, semantically anomalous: mean letters = 4.52, p > .03). These experimental materials were used to create three stimulus lists. Each list contained 30 exemplars of each of the four experimental sentence types. Items were counterbalanced between lists such that only one version of each sentence was presented on a given list. Each participant was presented with one list of 120 sentences for the grammaticality task and a separate list of 120 sentences for the acceptability task. Sentence frames were repeated between the two tasks, but appeared in different conditions, and lists were counterbalanced across participants.
Procedure Cognitive Assessment Before EEG recording, participants completed a battery of psychological tests including the Language and Social Background Questionnaire, Peabody Picture Vocabulary Test (PPVT)-III (Dunn & Dunn, 1997), Cattell Culture Fair Intelligence (Cattell, 1957), and Corsi Block tests (used after Fischer, 2001). The purpose of these tests was to assess the similarity of the two participant groups on vocabulary (PPVT), intelligence (Cattell), and working memory (Corsi Block test). The cognitive assessment session lasted approximately 45 min. EEG Recording during Sentence Judgment Tasks The EEG recording was conducted in a soundproof room. On each trial, participants were presented with a 500-msec fixation cross, followed by a sentence. Sentences were presented word by word, with each word appearing in the center of the screen for 300 msec. Words were separated by a blank screen interval of 350 msec. A 1450-msec blank screen interval was inserted at the end of each sentence, after which a response prompt appeared. In this study, the processing demands were manipulated by
Table 1. Examples of Sentences Used in the Acceptability and Grammaticality Tasks Semantically Correct
Semantically Incorrect
Syntax correct
A new computer will last for many years
A new computer will paint for many years
Syntax incorrect
A new computer will lasting for many years
A new computer will painting for many years
2842
Journal of Cognitive Neuroscience
Volume 26, Number 12
varying task instructions. Participants completed both a standard acceptability judgment task and the grammaticality judgment paradigm previously used by Bialystok and colleagues (Bialystok & Majumder, 1998; Bialystok, 1986). In the acceptability task, the participants were instructed to answer “yes” if they judged the sentence to be acceptable and “no” if the sentence was either semantically or grammatically incorrect. In the grammaticality task, participants responded “yes” if the sentence was grammatically correct and “no” if it was grammatically incorrect, irrespective of meaning. Thus, in the grammaticality task, sentences such as “A new computer will paint for many years” would be judged as correct, whereas the same sentence will be judged as incorrect in the acceptability task. These manipulations created a higher level of attentional control in the grammaticality task, because it was necessary to attend to only one dimension in spite of conflicting information from the other dimension. On the basis of prior behavioral studies (Bialystok & Majumder, 1998; Bialystok, 1986), in the grammaticality task, a higher degree of executive control is required to attend to syntax when the meaning is inconsistent. It is more difficult to approve the grammatically correct sentences and ignore the conflicting semantic information. Participants used two different hands to respond. They indicated their responses by pressing one button for “yes” and another one for “no,” with the left- and right-hand assignments counterbalanced across participants. For all participants, the grammaticality task was performed first. Participants had a brief break ( .05). The two language groups differed on the PPVT scores, F(1, 27) = 5.36, p < .005, with monolinguals obtaining higher scores than bilinguals. Accuracy rates on the grammaticality and acceptability judgment tasks (performed during EEG acquisition) were analyzed with repeated-measures ANOVA with Condition as a within-subject factor and Group as a between-subject factor. Before the ANOVA, the proportion correct for each participant and condition was subjected to the arcsinesquare-root transformation (Sokal & Rohlf, 1995), with values of 1 replaced by [1 − (1/(4n)], with n equal to the number of trials in each condition (30). The untransformed accuracy scores and standard deviations for the two groups are presented in Table 3. In the acceptability task, there was a significant main effect of group, indicating that monolinguals were significantly more accurate than bilinguals, F(1, 26) = 4.66, p < .05. The main effect of Condition was also significant, F(2, 52) = 14.98, p < 0 .001, reflecting overall higher accuracy for correct and syntactically incorrect sentences than for semantically anomalous structures (both ps < .001). There were no significant dif-
ferences between correct and syntactically anomalous sentences ( p > .05), and the Group × Condition interaction was not significant, F < 1. In the grammaticality task, there were no significant main effects of Group, F(1, 26) = 1.755, p > .05, or condition, F < 1. The Group × Condition interaction was not significant (F < 1), indicating that in the grammaticality task monolinguals and bilinguals did not differ in their ability to judge the syntax of the sentences.
Oscillatory Responses: ERSP Illustration of ERSP Characteristics The primary statistical analysis of ERSP responses in this experiment concerns the difference in the responses between anomalous and correct words in the same sentence position. However, as time–frequency analysis is somewhat less familiar than the ERP technique to most neuroscientists, we chose to illustrate additional characteristics of the responses measured in this study, although they are not the direct subject of statistical testing. We wish to demonstrate to the reader in a qualitative fashion how ongoing oscillations are modulated in a rapid serial visual presentation sentence comprehension task, both in correct sentences and in the presence of violations. Averaged time–frequency decompositions for one representative electrode, P5, are presented in Figure 2A. These are averaged across all 28 participants (both monolinguals and bilinguals) on data from the acceptability task. Power in the alpha and beta bands (approximately 6–25 Hz) is modulated by the periodic presentation of words in the sentence. Additionally, there is a general trend for power to decrease from the beginning to the end of the sentence. Superimposed on these two effects, linguistic
Table 3. The Mean Accuracy (Standard Deviations) Judgment Scores in Percent Correct for Each Language Group in the Acceptability and Grammaticality Tasks Language Group Monolinguals
Bilinguals
Condition
Acceptability Mean (SD)
Grammaticality Mean (SD)
Correct
95 (2)
96 (1)
Semantic errors
87 (3)
95 (2)
Syntactic errors
95 (2)
96 (2)
Correct
90 (2)
95 (1)
Semantic errors
81 (3)
93 (2)
Syntactic errors
93 (2)
93 (2)
Kielar et al.
2845
Figure 2. Example representations of oscillatory reactivity (ERD and ERS) in the acceptability task, combined across monolinguals and bilinguals. (A) Time–frequency maps of ERSP signal time-locked to control, semantic violations, and syntactic violations shown at one representative electrode, P5. (B) Time courses of 8–30 Hz power decreases at site P4, time-locked to the critical word onset. Peaks and troughs of power before and after the critical word are because of visual presentation of other words in the sentence. (C) Time courses of 8–30 Hz power decreases at site P5, time-locked to critical word onset. (D) Time courses of 1–5 Hz power increases at site Pz, time-locked to critical word onset.
violations induce a sharp drop in power that is maximal approximately 500–1500 msec after the onset of the critical word. The drop is larger for semantic violations than syntactic ones. For semantic violations, we also observe an increase in power in the delta and theta bands (approximately 1–5 Hz), which is consistent with some prior reports (Davidson & Indefrey, 2007; Hald et al., 2006). To avoid potential overlap with power increase in the theta range, we chose a lower limit of 8 Hz for our ANOVAs on differences in the alpha-beta power decrease across tasks and groups. Similarly, for delta-theta power increases, a 1–5 Hz frequency range was used for computing ANOVAs. Note that the 1–5 Hz ERS response has a slightly earlier 2846
Journal of Cognitive Neuroscience
onset and peak than the 8–30 Hz ERD on most electrodes. However, the statistical analyses (see below) indicated that significant differences in ERS between the conditions are predominantly observed in the same range as the ERD, 500–1500 msec after the onset of the critical word. Figures 2B and 2C displays the time course of power fluctuations in each condition, averaged over the 8–30 Hz range, at two representative electrodes (P4 and P5). These time course plots allow the reader to easily discern the three patterns of modulation present in the 8–30 Hz range: periodic rise-and-fall driven by the stimulus presentation, an overall decreasing trend across the sentence, and a specific decrease induced by linguistic violations. In this Volume 26, Number 12
example, P4 (right parietal) responds mainly to semantic violations, whereas P5 (left parietal) responds to both semantic and syntactic violations. Similarly, Figure 2D displays the time course of power fluctuations in the 1–5 Hz frequency band for semantic and syntactic violations at one of the central electrodes (Pz) that showed the strongest responses in this frequency range. The 1–5 Hz response is specifically modulated by semantic violations but is not periodically driven by the visual word presentation. Cluster Analysis Results For semantic violations versus control words, we observed two significant clusters of ERSP differences. Semantically anomalous words induced a power decrease (ERD), occur-
ring between 500 and 1500 msec, in the frequency range of 6–30 Hz (Figure 3A). In addition, semantic violations elicited a power increase (ERS) in the time range of 500– 1500 msec, in the frequency range of 1–5 Hz (Figure 3B). To simplify the display of this cluster across the dimensions of time, frequency, and scalp topography, we computed a measure that we refer to as T-sum. This is the sum of t values for all data points (time, frequency, and electrode) that are part of the cluster (Meltzer & Braun, 2011). By summing across different dimensions of the cluster, different aspects of the cluster can be highlighted. Note that this procedure of summing t values across individual dimensions is done solely for purposes of summarizing the clusterʼs spatiotemporal characteristics in a format convenient for display and does not play a role in the
Figure 3. Significant clusters of EEG oscillatory reactivity in the acceptability task, combined across monolinguals and bilinguals. (A) t values of 8–30 Hz ERD for semantic violations versus control words, summed across all channels at each time and frequency point and the scalp topography of the one significant cluster of ERD for semantic violations versus control words, represented as sums of t values across time and frequency points at each electrode. (B) t values of 1–5 Hz ERS for semantic violations versus control words and the scalp topography of the one significant cluster of ERS for semantic violations versus control words. (C) t values of 8–30 Hz ERD for syntactic violations versus control words and the scalp topography of the one significant cluster of ERD for syntactic violations versus control words.
Kielar et al.
2847
assessment of cluster significance as a whole. Figure 3A and B shows the T-sum for the clusters of ERD and ERS in response to semantic violations versus control words. At each time–frequency point, significant t values are summed across electrodes. A given point can achieve a higher t score either by having more significant electrodes or by having higher t values at the individual electrodes (generally both). Thus, the region of maximal response can be easily discerned. For the comparison of syntactic violations versus control words, we obtained one significant cluster, in the same time range of 500–1500 msec, in the frequency range of 6–22 Hz (see Figure 3C). Thus, both kinds of violations induced ERD in largely overlapping time and frequency ranges, although the response to semantic violations was larger and extended to higher frequencies. The observed frequency range includes the commonly defined alpha band (8–12 Hz) as well as the beta band (15–30 Hz). In addition, semantic violations induced ERS in the delta and theta bands (1–5 Hz). To illustrate the topography of these clusters, we computed T-sum values at each electrode, summing over both time and frequency dimensions. The resulting topographies of ERD and ERS for semantic violations are shown in Figure 3A and B. ERD induced by semantic violations was maximal over bilateral parietal electrodes, whereas ERS was maximal at the frontal and central sites, with additional increases observed at the temporal sites. The topography of ERD for syntactic violations is shown in Figure 3C. The distribution of the syntactic response appeared to be somewhat more left-lateralized and also stronger at bilateral fronto-central electrodes. However, the statistical significance of differences in topography cannot be tested directly from the cluster analysis, which operates on each electrode separately. Because the cluster analysis identified consistent time–frequency windows in which ERSP responses to linguistic anomalies occurred, we then conducted repeated-measures ANOVAs on ERSP values averaged within the relevant windows to formally assess differences in scalp topography and effects of bilingualism within each task. Relationship between ERP and ERSP Measures Because the cluster analysis revealed significant time– frequency responses in the alpha-beta (8–30 Hz) and delta-theta (1–5 Hz) frequency bands, we investigated the extent to which these ERSP responses may be driven by time-locked signals. To do that, we performed a time– frequency analysis on the ERP signal using the same procedure as was applied to compute ERD and ERS responses but applied to the signal after averaging across trials within each participant. This “ERSP of ERPs” analysis was computed for the acceptability task across all 28 participants. The analysis revealed power increases for both semantic and syntactic anomalies from 1 to 5 Hz spanning 500– 1500 msec time window (see Figure 4A). These effects 2848
Journal of Cognitive Neuroscience
were the strongest over the central electrode sites and were greater for semantic than syntactic violations (see Figure 4A and B). Correlations were computed between the ERSP measures (8–30 Hz ERD and 1–5 Hz ERS) and the ERSP of ERPs effects in the 1–5 Hz frequency range and within the 500–1500 msec time window. These analyses were computed for the acceptability task, separately for semantic and syntactic violations on the difference scores (semanticcontrol; syntactic-control) across the 28 participants. The correlations were computed on the average response combined across all electrodes and also separately within each of the previously defined 11 electrode clusters. The results are presented in Table 4. The analysis revealed significant correlations between the 1–5 Hz ERS and the “ERSP of ERPs” signals. For semantic violations, significant positive correlations were present when computed across all electrodes, as well as separately at the central, parietal, and frontal regions. These positive correlations reflect correspondence between power increases in 1–5 Hz frequency range for the total time–frequency response (ERS) and the portion of it that is attributable to phase-locked signals (ERSP of ERPs). For syntactic violations, significant correlations were also found at the left central parietal, left frontal, and parietal regions, but these correlations were somewhat weaker than those seen for semantic violations and did not meet the Bonferroni-corrected significance threshold for multiple tests done in 11 electrode regions. Also, correlation in the average signal across all electrodes was not significant for syntactic violations. The same analysis was performed for 8–30 Hz ERD. However, the analysis revealed that there were no significant correlations between the 8–30 Hz ERD and the ERSP of ERPs measures. These results indicate that 1–5 Hz ERS, but not 8–30 Hz ERD, correlated with power increases in the 1–5 Hz frequency range present in the ERP signals. Comparison of Figure 4B and C illustrates the similarity between 1–5 Hz ERS power computed over single trials (Figure 4C) and the time–frequency characteristics of the ERP signal averaged across trials (Figure 4B). Topographical maps of the correlations at each electrode are presented in Figure 4D for the two comparisons. Electrodes showing significant correlations are highlighted at both an uncorrected level ( p < .05) and a level Bonferronicorrected for multiple comparisons across individual electrodes ( p < .0008). We also computed correlations between 8–30 Hz ERD and 1–5 Hz ERS. This analysis revealed one significant positive correlation for semantic violation limited to the left central-parietal region, but this correlation did not survive Bonferroni correction for multiple comparisons over the 11 electrode regions. No other correlations were significant, and there were no significant correlations for the syntactic anomalies. These results indicate that participants who exhibit a strong 1–5 Hz ERS response in single trials also tend to exhibit a strong response in the time domain average or Volume 26, Number 12
Figure 4. Representation of oscillatory activity obtained from the ERP signal using time–frequency analysis in the acceptability task, combined across monolinguals and bilinguals, compared with 1–5 Hz ERS. (A) Time–frequency maps of ERSP of ERP signals time-locked to control, semantic violations, and syntactic violations shown at one representative electrode, CPz. (B) Time courses of 1–5 Hz ERSP of ERPs at central site CPz, time-locked to the critical word onset, and the scalp topography for semantic violations versus control in 1–5 Hz band at 500–1500 msec time window. (C) Time courses of 1–5 Hz ERS at central site CPz, time-locked to the critical word onset, and the scalp topographies for semantic violations versus control at 500–1500 msec time window. (D) The topographical maps of correlations computed for semantic violation versus controls between ERSP measures (1–5 Hz ERS and 8–30 Hz ERD) and the ERSP of ERPs effects, in the 1–5 Hz frequency range. The electrodes showing significant correlations are marked with symbols (black diamonds: p < .05, uncorrected; blue circles: p < .0008, Bonferroni-corrected).
Kielar et al.
2849
Table 4. Correlations between ERSP of ERPs and 1–5 Hz ERS, 8–30 Hz ERD and between 8–30 Hz ERD and 1–5 Hz ERS Computed for Semantic and Syntactic Anomalies in Acceptability Task, Combined across Monolinguals and Bilinguals ERSP of ERPs vs. 1–5 Hz ERS Condition
Region
Semantic
Average CP PP
Significance (p)
Region
0.434
.021a
Average
0.113
ns
0.454
Correlation (r)
8–30 Hz ERD vs. 1–5 Hz ERS
Significance (p)
Region
0.001
ns
Average
0.226
ns
Correlation
Significance (p)
CP
−0.120
ns
CP
0.344
ns
.015
a
PP
0.169
ns
PP
0.117
ns
a
RCP
0.126
ns
RCP
0.170
ns
LCP
0.090
ns
LCP
0.454
0.015a
RCP
0.379
.047
LCP
0.190
ns
FC
0.550
.002bc
FC
−0.005
ns
FC
−0.238
ns
LF
0.568
.002
bc
LF
−0.104
ns
LF
0.138
ns
RF
0.353
ns
RF
0.214
ns
RF
0.005
ns
RFC
−0.287
ns
RFC
−0.230
ns
RFC
0.384
.044
LFC
0.345
ns
RP
Syntactic
Correlation (r)
ERSP of ERPs vs. 8–30 Hz ERD
0.467
a
LFC
−0.231
ns
LFC
0.252
ns
.012
a
RP
−0.097
ns
RP
0.060
ns
a
LP
0.169
ns
LP
0.340
ns
LP
0.469
.012
Average
0.116
ns
Average
0.010
ns
Average
0.179
ns
CP
0.074
ns
CP
0.124
ns
CP
0.254
ns
PP
0.291
ns
PP
0.081
ns
PP
0.058
ns
RCP
0.144
ns
RCP
0.071
ns
RCP
0.169
ns
LCP
0.128
ns
LCP
0.269
ns
FC
−0.268
ns
FC
0.223
ns
LF
−0.221
ns
LF
−0.066
ns
−0.020
ns
LCP FC
0.451 −0.062
.016
a
ns a
LF
0.432
.022
RF
0.061
ns
RF
−0.006
ns
RF
RFC
0.162
ns
RFC
−0.123
ns
RFC
0.108
ns
LFC
0.427
.023a
LFC
0.086
ns
LFC
0.227
ns
RP
0.175
ns
RP
−0.107
ns
RP
0.036
ns
LP
0.048
ns
LP
0.000
ns
LP
0.413
.029
a
The analyses were computed on the mean amplitude of the difference scores as averaged across all electrode regions and separately at each of the 11 electrode regions. The significant correlations are marked in bold font. The electrodes showing significant correlations are marked (a: p < .05, uncorrected; bc: p < .004, Bonferroni corrected).
ERP, whereas no such relationship was apparent with the 8–30 Hz ERD. This pattern is consistent with a common neural process driving the expression of 1–5 Hz ERS and ERPs, distinct from that underlying the 8–30 Hz ERD. These 1–5 Hz power increases represent phase-locked signals evoked by the violations but collapse across polarity, as both negative and positive components contain power. Thus, it is possible to observe significant ERS in the delta and theta ranges, even if ERP responses are highly variable across participants. The relationship between ERP responses to semantic violations, 1–5 Hz ERSP of ERPs, and 1–5 Hz ERS are presented in Figure 5. Figure 5A illustrates the topographical distribution of ERP 2850
Journal of Cognitive Neuroscience
responses to semantic violations versus control words. In this experiment, semantic violations evoked both negative (N400) and positive (P600) waveforms at different electrode sites, with a biphasic waveform at central sites. Figure 5B shows that 1–5 Hz ERSP of ERPs picks up signals of either polarity, both positive and negative. The strong correlations observed across participants (Figure 4) suggest that ERP components of either polarity contribute to the 1–5 Hz ERS seen in the time–frequency analyses. For a representative electrode C3, we display the time courses of ERP, 1–5 Hz ERSP of ERPs, and 1–5 Hz ERS responses in Figure 5C, D, and E, respectively. These graphs illustrate that the violation induced both negative and Volume 26, Number 12
positive ERP components, both of which are reflected in 1–5 Hz ERS. Statistical Analysis of 8–30 Hz ERD and 1–5 Hz ERS To investigate the scalp distributions of ERD and ERS effects and to test for significant interactions between violation conditions and language groups in the two tasks, we chose to analyze ERD in the time window of 500–
1500 msec and frequency range of 8–30 Hz. In addition, ERS was analyzed in the 1–5 Hz frequency range and 500–1500 msec time window. This range comprises the bulk of the significant clusters (Figure 3) and is consistent with previous studies reporting modulations of brain oscillations in this range by language tasks (Willems et al., 2008; Davidson & Indefrey, 2007). Although significant ERD does extend below 8 Hz at some electrodes, we also observed ERS up to about 6 Hz in semantic condition at
Figure 5. The relationship between ERP responses to semantic violations, 1–5 ERSP of ERPs and 1–5 Hz ERS. (A) The topographical distribution of ERP responses to semantic violations versus control words. Semantic violations evoked both negative (N400) and positive (P600) waveforms at different electrode sites, with a biphasic waveform at central sites. (B) The topographical distribution of 1–5 Hz ERSP of ERPs responses to semantic violations versus control words. 1–5 Hz ERSP of ERPs picks up signals of either polarity, both positive and negative. (C) Time courses of ERPs at C3. (D) Time courses of 1–5 Hz ERSP of ERPs at C3. (E) Time courses of 1–5 Hz ERS at C3.
Kielar et al.
2851
Table 5. Statistical Results in the 500–1500 msec Time Window A. Effects of Violation Condition and Region in Acceptability Task Frequency Range 8–30 Hz ERD
1–5 Hz ERS
df
F
p
Violation condition (semantic vs. syntactic)
(1,27)
9.73
.004
Region
(10,270)
2.48
.019
Condition × Region
(10,270)
6.65
