Journal of Neuroscience Methods 241 (2015) 1–9

Contents lists available at ScienceDirect

Journal of Neuroscience Methods journal homepage: www.elsevier.com/locate/jneumeth

Basic Neuroscience

Frequency-modulated steady-state visual evoked potentials: A new stimulation method for brain–computer interfaces Alexander M. Dreyer a , Christoph S. Herrmann a,b,∗ a Experimental Psychology Lab, Department of Psychology, Center for Excellence ‘Hearing4all’, European Medical School, University of Oldenburg, 26111 Oldenburg, Germany b Research Center Neurosensory Science, University of Oldenburg, 26111 Oldenburg, Germany

h i g h l i g h t s • We were able to evoke SSVEPs with a frequency-modulated flickering LED. • FM SSVEP’s amplitude does not differ from sinusoidally evoked SSVEP amplitude. • Subjective flicker perceptibility decreases with increasing carrier frequency.

a r t i c l e

i n f o

Article history: Received 4 April 2014 Received in revised form 4 December 2014 Accepted 6 December 2014 Available online 15 December 2014 Keywords: Steady-state visual evoked potential (SSVEP) Frequency modulation Brain–computer interface (BCI) Amplitude modulation Light-emitting diode (LED) EEG

a b s t r a c t Background: Steady-state visual evoked potentials (SSVEPs) are widely used for brain–computer interfaces. However, users experience fatigue due to exposure to flickering stimuli. High-frequency stimulation has been proposed to reduce this problem. We adapt frequency-modulated (FM) stimulation from the auditory domain, where it is commonly used to evoke steady-state responses, and compare the EEG as well as behavioral flicker perceptibility ratings. New method: We evoke SSVEPs with a green light-emitting diode (LED) driven by FM signals. Results: FM-SSVEPs with different carrier and modulation frequencies can reliably be evoked with spectral peaks at the lower FM sideband. Subjective perceptibility ratings decrease with increasing FM carrier frequencies, while the peak amplitude and signal-to-noise ratio (SNR) remain the same. Comparison with existing method: There are neither amplitude nor SNR differences between SSVEPs evoked rectangularly, sinusoidally or via FM. Perceptibility ratings were lower for FM-SSVEPs with carrier frequencies of 20 Hz and above than for sinusoidally evoked SSVEPs. Conclusions: FM-SSVEPs seem to be beneficial for BCI usage. Reduced flicker perceptibility in FM-SSVEPs suggests reduced fatigue, which leads to an enhanced user experience and performance. © 2014 Elsevier B.V. All rights reserved.

1. Introduction Repetitive presentation of a stimulus leads to a synchronization of neural firing in sensory cortex neurons at the respective

Abbreviations: EEG, electroencephalogram; SSVEP, steady-state visual evoked potential; BCI, brain––computer interface; LED, light-emitting diode; FM, frequency-modulation; AM, amplitude-modulation; SNR, signal-to-noise ratio; ITR, information transfer rate; FFT, fast Fourier transform; ERP, event-related potential; IAF, individual alpha frequency; ASSR, auditory steady-state response; CFF, critical flicker fusion frequency; FDR, false discovery rate. ∗ Corresponding author at: Experimental Psychology Lab, Department of Psychology, University of Oldenburg, 26111 Oldenburg, Germany. Tel.: +49 441 798 4936; fax: +49 441 798 3865. E-mail address: [email protected] (C.S. Herrmann). http://dx.doi.org/10.1016/j.jneumeth.2014.12.004 0165-0270/© 2014 Elsevier B.V. All rights reserved.

stimulus-presentation frequency. For different sensory modalities, this phenomenon can be measured as distinct oscillations in the electroencephalogram (EEG). In the auditory domain such oscillations are known as so called auditory steady-state responses (ASSRs; Ross et al., 2000) and in the visual domain they are known as steady-state visual evoked potentials (SSVEPs; Silberstein, 1995). Steady state responses can be evoked with a wide range of stimulation frequencies in both domains, but especially strong responses to particular resonance frequencies (Herrmann, 2001; Zaehle et al., 2010) as well as to attended stimuli (Müller et al., 1998; Ross et al., 2004) have been shown. SSVEPs will be frequency- and phaselocked to the stimulus (Regan, 1989) and can be evoked using many different flickering stimuli (e.g. Halbleib et al., 2012; Martens & Hübner, 2013). Due to their high signal-to-noise ratio (SNR) in the frequency domain, SSVEPs can be reliably recorded within a few

2

A.M. Dreyer, C.S. Herrmann / Journal of Neuroscience Methods 241 (2015) 1–9

seconds of stimulation. Therefore they are very promising stimuli for brain–computer-interfaces (BCIs) and many studies have reported the successful integration of flickering stimuli into their BCI systems (see Vialatte et al., 2010 for a review). Presenting stimuli on a computer screen limits the number of available stimulation frequencies because they depend on the respective refresh-rate of the screen used, but one can overcome such limitations with the use of light emitting diodes (LEDs) as stimuli. Moreover LED stimulation devices can easily and cheaply get specifically designed for their respective task without any spatial limitations. Hwang et al. (2012) for example, built a keyboard out of thirty individually flickering LEDs that was successfully implemented as a mental spelling system with a high information transfer rate (ITR), which is a commonly used performance measure for BCI systems. The high ITR in combination with the 30 simultaneous commands shows the progress of SSVEP-BCI system during the last years, since Vialatte et al. (2010) reviewed SSVEP-based systems with ‘only’ up to 13 simultaneous commands (Cheng et al., 2002). While stimulation parameters such as frequency, stimulus proximity and number of stimuli were tested and analyzed in detail (e.g. Ng et al., 2012) because they influence SSVEP power and SNR which are crucial factors when planning BCI systems, we believe, that potential BCI systems, especially for clinical application, should also further integrate the users perspective. On the one hand, more simultaneous commands broaden the range of possible applications, on the other hand, BCI users might be overwhelmed by the attentional demand of many independently flickering stimuli and experience fatigue during the stimulation process. Increasing levels of fatigue after SSVEP stimulation have been shown with subjective and physiological measures (Cao et al., 2014). One way to cope with such a problem might be to reduce the perceptibility of each individual stimulus so that it is easier for the user to ignore certain stimuli which are not important for a certain command. Low frequency flickering stimuli have been shown to be more perceptible and more annoying than faster flickering stimuli (Lin et al., 2012). This is consistent with the fact that subjects perceive a constant dim light instead of a flickering LED when the stimulation rate exerts a critical flicker fusion frequency (CFF; e.g. 30 Hz in Herrmann, 2001), which depends on parameters like stimulus brightness, color, size, its position in the visual field and others (Levin et al., 2011). SSVEPs can still be recorded at higher frequencies, but with increasing stimulation frequency their respective power decreases (Ku´s et al., 2013). Consequently, BCI accuracy drops with increasing flicker frequency (Volosyak et al., 2011). Recently, Chang et al. (2014) proposed amplitude-modulated (AM) stimulation for the visual domain in order to reduce eye fatigue and the risk of epileptic seizures. Their study revealed similar BCI performance of low constant-frequency SSVEPs and higher frequency AM-SSVEPs while subjects reported less eye fatigue and a reduced sense of flickering with AM stimulation. This suggests that also frequency-modulated (FM) stimulation, which is frequently used in ASSR-research (Picton et al., 2003), might be another compromise between stimulation based on high carrier frequencies and recording low frequency spectra, because FM signals have sidebands at the distance of modulation and carrier frequency in their spectral decomposition. In contrast to AM signals, FM signals have a constant envelope, i.e. the amplitude of the flicker stays constant. A comparison of AM and FM signals and spectra is shown in Fig. 1. This suggests that also FM stimulation is a feasible protocol for BCI and other SSVEP studies. Apart from one study that used frequency shift keying for modulating the stimulation signal (Kimura et al., 2013), which works differently than the FM approach we introduce in Section 2.2, we are not aware of any study that evoked SSVEPs with FM signals to date.

The goal of our study therefore was to test whether we would be able to evoke SSVEPs at certain sidebands with an LED that is driven with FM signals and to analyze behavioral subjective reports on the perceptibility of different carrier/modulation frequency pairs in comparison to a constant-frequency stimulation. 2. Materials and methods 2.1. Participants Twelve students (9 female) from the University of Oldenburg with a mean age of 23.1 years (range from 19 to 25), were paid for participating in the EEG session of this study. Thirteen additional students (9 female) with a mean age of 25.7 years (range from 24 to 30), voluntarily reported the flicker perceptibility of the different stimulation frequencies in a short experimental session without EEG recording. All participants had normal or corrected-to-normal vision and were informed about the risk of seizures in epileptics due to flicker stimulation. They reported not to have ever suffered from epilepsy and gave their written informed consent. The study was approved by the local ethics committee. 2.2. Stimuli A green LED (diameter 0.5 cm) was mounted in a distance of 1 m from the participants’ nasion, thereby covering 0.286◦ of the visual field. The LED was mounted on a tripod and in front of a black wall. A digital-to-analog converter (NI USB-6229 BNC, National Instruments, Austin, Texas, USA) was used to drive the LED at different frequencies, namely at 10 Hz (sinusoidally and rectangularly) and at nine additional modulated frequencies. The modulated signals were generated with MATLAB (The MathWorks Inc., Natick, MA, USA) using the following formula: signal = A + FV ∗ sin (2 ∗ pi ∗ Fc ∗ t + (M ∗ sin (2 ∗ pi ∗ Fm ∗ t))) . A is the DC bias (1.9 V) at which the LED was driven. FV is the flicker voltage span (0.04 V), Fc represents the carrier frequencies (20, 30, 40, 50, 60, 70, 80, 90, 100 Hz) and Fm the corresponding modulation frequencies (10, 20, 30, 40, 50, 60, 70, 80, 90 Hz). M is the modulation index (0.5) and t is the time vector. The parenthesized values were used for the experiment. In order to make sure that the LED flickered sinusoidally between a light glimmer and its maximal possible brightness, we tested different stimulation parameters and analyzed the flicker using a photodiode connected to our EEG amplifier which lead to the given average and span values as well as to the modulation Index of 0.5 and a sampling rate of 10 kHz. The carrier/modulation frequency pairs all had a difference of 10 Hz which equals the lower sideband of the modulated signal in its frequency domain. Refer to Fig. 1A, B and D to see the types of stimulation we used to evoke SSVEPs in this study. For illustrative purposes the amplitude values used for this figure deviate from the experimental values. 2.3. Data acquisition and experimental procedure The experiment was carried out in an electrically shielded, dark recording chamber, using Brain Vision Recorder (Brain Products GmbH, Gilching, Germany) for EEG acquisition. Sampling rate was set to 500 Hz while the amplifiers frequency passband ranged from 0.1 Hz to 250 Hz. Thirty-two electrodes, including one vertical EOG electrode below the right eye, were placed on size-appropriate EEG caps according to the international 10–20 system. One further electrode was placed on the nose as online reference. A twominute baseline recording was done first which gave the subjects a chance to adapt to the darkness. After that, we started with

A.M. Dreyer, C.S. Herrmann / Journal of Neuroscience Methods 241 (2015) 1–9

Sine wave

Square wave

B

1

1

0

0

−1

Amplitude

Amplitude

A

−1

0

time in s

1

1

0

10

20

30 frequency in Hz

40

0

50

AM wave

time in s

10

20

1

30 frequency in Hz

40

50

FM wave

D

1

1 0

−1

Amplitude

0 Amplitude

0 1

C

−1

0

time in s

1

1

0

3

0

time in s

1

1

10

20

30 frequency in Hz

40

50

0

10

20

30 frequency in Hz

40

50

Fig. 1. Exemplary time series stimulation data and respective frequency spectra of (A) a 10 Hz sine wave, (B) a 10 Hz square wave, (C) a 30 Hz carrier sine wave, amplitudemodulated by a 5 Hz sine wave and (D) a 30 Hz carrier sine wave, frequency-modulated by a 20 Hz sine wave. Note that the spectral peaks represent (A) the stimulation frequency only, (B) the stimulation frequency and odd harmonic frequencies, (C) and (D) the carrier frequency and sidebands at carrier frequency +/− modulation frequency. Note that FM wave spectra strongly depend on the modulation index, which was 0.5 in (D).

the 10 Hz flicker signals in pseudo-randomized order, followed by the frequency modulated signals in randomized order. The participants were instructed to centrally fixate the LED during the whole stimulation period, because this was expected to create the strongest SSVEPs (Lin et al., 2012). Each stimulation block lasted 100 s. After each block the participants decided when to start the next stimulation block which in most cases was within 2 min. We used the same procedure and setup for the perceptibility report, but we did not record EEG and reduced the stimulation time to 10 s. We further left out the 10 Hz rectangular stimulation and completely randomized the order of the ten remaining stimuli. After each stimulation, subjects had to rate the intensity of their flicker perception on a scale from 1 (no flicker/constant light perception) to 5 (very strong flicker perception). Note, that we explicitly instructed them to ignore parameters like flicker frequency and to really concentrate on their flicker perception only.

2.4. EEG data analysis Analysis was done using MATLAB, the EEGLAB toolbox (Delorme & Makeig, 2004), R (R Core Team, 2013) and the R-matlab package (Bengtsson, 2014). We first subtracted the common-average reference. For further analysis we only included electrodes from occipital scalp locations, namely P7, P3, Pz, P4, P8, O1 and O2. The data was high-pass filtered with a cutoff at 1 Hz. Power line noise was reduced using a notch filter with a stop-band from 48 to 52 Hz. To create event-related potentials (ERPs), i.e. SSVEPs, and to spectrally analyze them (cf. Fig. 4 for exemplary results), we included all subjects and every 100 ms we created largely overlapping 1 s epochs for further analysis (cf. Fig. 2E). These were corrected by their respective mean values, so that each epoch had a mean of zero. Ocular and other artifacts were rejected in a semiautomatic procedure, using individually adjusted threshold values for the automatic EEGLAB rejection function as well as a point-topoint threshold artifact rejection function. Individual inspection

of all datasets completed this process. In three out of the 132 (subjects × frequencies) datasets we classified a total of 5 out of 924 (subjects × frequencies × electrodes) complete channels as too noisy, which we than extrapolated, using the mean values of the remaining occipital channels. The SSVEPs were then calculated by averaging the epochs as well as the occipital channels to an ERP for each condition, which was decomposed into its frequency components using the fast Fourier transform (FFT). For the baseline recording, we generated analogous epochs which were preprocessed the same way as the experimental data. Baseline spectra were calculated by computing the fast Fourier transform of each epoch and averaging these for each individual subject. The highest peaks between 8 and 12 Hz in these spectra were considered the individual alpha frequency (IAF). A higher frequency resolution of 0.1 Hz was achieved by adding 9 s of zeros to the end of each epoch. We tested whether this approach would make a difference for the experimental data as well, and decided to show results with 1 Hz frequency resolution only, because the SSVEP peaks at 10 Hz were not affected. For computation of Fig. 5, we analyzed two datasets from one exemplary subject using the following four different averaging approaches:

• total raw spectra: After preprocessing the data, the total spectra of raw EEG were calculated over the complete stimulation period, according to the following pseudocode (corresponds to Fig. 2B & Fig. 5A):

abs(fft(data))

• total averaged spectra The total spectra averaged across 1 s epochs relate to 1 s epochs which do not overlap. Artifactual epochs were rejected visually. The remaining epochs were first analyzed spectrally with an FFT

4

A.M. Dreyer, C.S. Herrmann / Journal of Neuroscience Methods 241 (2015) 1–9

Fig. 2. Relation between raw EEG and different epoching approaches. (A) Shows the raw EEG during 10 Hz sine stimulation of an exemplary subject. (B) Shows 1 × 100 s epoch for calculating the total spectra of raw EEG. (C) Shows 100 × 1 s epochs for averaged total spectra. (D) Shows same epochs as (C), which were also used for calculating non-overlapping evoked spectra. (E) Shows strongly overlapping 1 s epochs for overlapping evoked spectra.

and the results were then averaged (corresponds to Fig. 2C and Fig. 5B): mean(abs(fft(non − overlapping epochs))) • non-overlapping evoked spectra The same epochs were also averaged to yield an ERP first and then analyzed with an FFT to create the non-overlapping evoked spectra (corresponds to Fig. 2D and Fig. 5C): abs(fft(mean(non − overlapping epochs))) • overlapping evoked spectra Overlapping evoked spectra were calculated with the same approach as for Fig. 4 but with data from the respective exemplary subject only (corresponds to Fig. 2 E and Fig. 5 D):

certain stimulation conditions. This suggests that their CFF for these stimulation conditions was reached. A single within-subject factor design with repeated measures (13 subjects × 10 stimulation conditions) was used. Analysis of variance revealed highly significant differences in the mean perceptibility scores across the 10 conditions (F(9,108) = 55.83, p < .0001). Pairwise t-tests with false discovery rate (FDR) adjusted p-values between all possible pairs of stimulation conditions revealed four different perceptibility levels with significant differences to all conditions in the other levels. As shown in Fig. 3, the strongest flicker was perceived in the 10 Hz sinusoidal condition. All FM-conditions were perceived at a significantly lower level. Among these, we see significant perceptibility decrease from the 20 Hz carrier frequency to the 30 Hz carrier frequency and then from the 30 Hz carrier frequency to the 40 Hz carrier frequency. Perceptibility ratings from carrier frequencies 40 Hz up to 100 Hz all clustered around a rating of 2 which relates to ‘slight flicker’ perception in the questionnaire.

abs(fft(mean(overlapping epochs))) 3.2. SSVEP Fig. 2 shows the relation of raw EEG and the different epoching approaches. Each consecutive overlapping epoch starts with the onset of a 10 Hz cycle of the visual stimulation, i.e. 100 ms after the previous epoch started. Note the importance of choosing analysis windows that exactly contain an integer number of stimulation cycles, when generating epochs (Bach & Meigen, 1999). 3. Results Some subjects reported illusionary movements of the LED, which occurred especially toward the end of each 100 s stimulation block. 3.1. Perceptibility report Subjects used the whole range of the report questionnaire, indicating that some subjects did not see any flicker under

We were able to evoke 10 Hz SSVEPs reliably in all frequency modulated conditions as well as in the 10 Hz conditions. Exemplary SSVEPs and frequency spectra can be seen in Fig. 4. We tested the significance using the SNR according to Meigen & Bach (1999). To find the SNR, the SSVEP amplitudes of the overlapping evoked spectra were divided by the average amplitude of the two neighboring frequencies. Table 1 provides SNR values for all subjects in all conditions and the respective amplitude values (in brackets). When using the two neighboring frequencies as noise estimators, SNR values above 4.55 relate to an SSVEP significance level of p < .01 (Meigen & Bach, 1999), hence all the recorded SSVEPs were significantly different from noise. Based on the overlapping evoked spectra (cf. Fig. 2E), we used a single within-subject factor design with repeated measures (12 subjects × 11 stimulation conditions). Analysis of variance (F(10,110) = 1.19, p = .31) of the respective SSVEP amplitudes at 10 Hz revealed that there was no significant difference among the

A.M. Dreyer, C.S. Herrmann / Journal of Neuroscience Methods 241 (2015) 1–9

5

Perceptibility Scores * 5 **

4

Mean Score

**

3

2

1

10 Hz sine Fc 20 Hz Fc 30 Hz Fc 40 Hz Fc 50 Hz Fc 60 Hz Fc 70 Hz Fc 80 Hz Fc 90 Hz Fc 100 Hz

Fig. 3. Mean perceptibility questionnaire scores for all conditions. Error bars show the respective standard errors of the means. The scores correspond to 1 = “no flicker”, 2 = “slight flicker”, 3 = no explicit description, 4 = “clear flicker” and 5 = “very strong flicker”. The indicated significant (* p < .05; ** p < .001) comparisons can be seen as transitions between the four levels (indicated by colors) with no differences within themselves but significant differences to all other conditions.

conditions, which was confirmed by pairwise t-tests with FDR adjusted p-values. All possible pairs of stimulation conditions were compared but none of them were significantly different from each other. We repeated this analysis with normalized SNR values (Ku´s et al., 2013) instead of the amplitudes. The results were similar. Analysis of variance results were the following: F(10,110) = 1.76, p = .08. FDR adjusted pairwise t-tests revealed that none of the possible frequency pairs significantly differed from each other. To assure that we were not just measuring internal alpha oscillations we checked whether the IAFs were at exactly 10 Hz. This was the case for only one subject. Fig. 5 shows the results from different exemplary analysis approaches for one subject and two stimulation conditions that further confirm the above results. It reveals certain properties of these different spectral analysis approaches. The total spectra of 1

μV

200

300

400

0.5

0 0

ms

5

10

Fc 20 Hz

200

300

20

25

30

Hz

25

30

Hz

25

30

Hz

25

30

Hz

Fc 20 Hz μV

100

15

400

0.5

0 0

ms

5

10

Fc 40 Hz

15

20

Fc 40 Hz 0.5 μV

0 −1 0 1

D

100

0 −1 0 1

C

10 Hz sine

0 −1 0 1

B

10 Hz sine

100

200

300

400

0 0

ms

5

10

Fc 90 Hz

15

20

Fc 90 Hz 0.5

0 −1 0

μV

A

raw EEG (top row) have a frequency resolution of 0.01 Hz. An overall amplitude increase in the alpha range can be seen with distinct SSVEP peaks at exactly 10 Hz (see black arrows), representing the stimulation frequency (top left) or the lower FM-sideband (top right). A second peak at 10.86 Hz (top left) or 10.91 Hz (top right) can be considered the IAF (see gray arrows). The averaged total spectra (second row) follow a similar pattern with a reduced frequency resolution of 1 Hz and the respective values at 10 Hz (SSVEP) and 11 Hz (IAF) merging into one broad peak. After computing the ERP, only signals that are locked to the stimulation, namely the 10 Hz peak, remain, as can be seen in the third and bottom row. Due to highly overlapping epochs that were averaged for the analysis in the bottom row, all other (e.g. IAF) than the stimulation frequency or FM-sideband and their harmonics get suppressed. On the one hand, this implies that strongly overlapping evoked spectra have the highest SNR which should

100

200

300

400

ms

0 0

5

10

15

20

Fig. 4. Recorded ERPs and their frequency decomposition for (A) the 10 Hz sinusoid stimulation, (B) the 20 Hz carrier/10 Hz modulation pair stimulation, (C) the 40 Hz carrier/30 Hz modulation pair stimulation and (D) the 90 Hz carrier/80 Hz modulation pair stimulation. The frequency resolution of the spectra is 1 Hz.

6

A.M. Dreyer, C.S. Herrmann / Journal of Neuroscience Methods 241 (2015) 1–9

Table 1 SSVEP-SNR calculated based on Meigen & Bach (1999) for each subject in each condition and the respective mean values for the different stimulation conditions. SSVEP amplitude values for each subject and each conditions are shown in brackets). Subject

10 Hz Rect

10 Hz Sine

Fc 20 Hz

Fc 30 Hz

Fc 40 Hz

Fc 50 Hz

Fc 60 Hz

Fc 70 Hz

Fc 80 Hz

Fc 90 Hz

Fc 100 Hz

1

35.99 (1.04) 45.83 (0.93) 220.85 (0.65) 32.12 (0.40) 40.06 (0.54) 98.16 (0.18) 70.18 (0.27) 56.64 (0.21) 40.57 (0.45) 38.07 (0.19) 11.25 (0.19) 86.13 (0.25) 64.65 (0.44)

26.89 (0.89) 131.14 (0.79) 33.04 (0.48) 41.62 (0.57) 40.26 (0.41) 94.43 (0.23) 36.68 (0.17) 27.74 (0.24) 36.04 (0.44) 39.70 (0.27) 19.60 (0.26) 54.11 (0.28) 48.44 (0.42)

14.36 (0.54) 15.01 (1.05) 21.23 (0.21) 18.59 (0.33) 46.91 (0.39) 130.63 (0.37) 27.88 (0.16) 43.24 (0.24) 25.83 (0.17) 30.28 (0.25) 8.87 (0.11) 15.42 (0.07) 33.19 (0.32)

23.28 (0.74) 41.66 (1.38) 13.23 (0.24) 30.50 (0.33) 18.60 (0.15) 96.41 (0.22) 38.23 (0.22) 37.84 (0.14) 19.19 (0.16) 16.51 (0.14) 18.11 (0.35) 48.52 (0.22) 33.51 (0.36)

41.59 (2.19) 13.12 (2.09) 30.85 (0.26) 15.32 (0.15) 73.96 (0.58) 121.05 (0.28) 63.13 (0.17) 39.46 (0.11) 66.05 (0.21) 18.93 (0.08) 16.51 (0.14) 74.94 (0.25) 47.91 (0.54)

18.38 (0.92) 73.85 (1.91) 46.91 (0.36) 24.67 (0.51) 64.28 (0.39) 29.95 (0.16) 61.05 (0.20) 35.86 (0.10) 25.32 (0.28) 31.96 (0.22) 19.57 (0.15) 93.43 (0.19) 43.77 (0.45)

15.05 (0.57) 19.06 (1.51) 25.32 (0.19) 13.88 (0.34) 38.20 (0.30) 13.81 (0.04) 28.63 (0.17) 60.30 (0.13) 58.20 (0.30) 79.64 (0.27) 4.74 (0.10) 27.42 (0.12) 32.02 (0.34)

18.60 (0.96) 10.56 (0.41) 93.16 (0.3) 41.15 (0.14) 122.41 (0.34) 105.12 (0.27) 51.83 (0.14) 21.87 (0.14) 221.65 (0.76) 14.79 (0.27) 14.55 (0.14) 77.15 (0.17) 66.07 (0.34)

19.81 (1.01) 33.00 (1.05) 34.97 (0.37) 27.48 (0.37) 11.81 (0.19) 36.07 (0.27) 15.20 (0.11) 19.31 (0.11) 88.87 (0.24) 34.11 (0.15) 15.88 (0.17) 77.49 (0.22) 34.50 (0.36)

71.17 (0.92) 18.63 (0.70) 37.24 (0.25) 14.14 (0.32) 82.71 (0.32) 79.32 (0.43) 9.77 (0.10) 12.24 (0.17) 43.18 (0.27) 10.83 (0.12) 16.96 (0.16) 17.36 (0.16) 34.46 (0.33)

24.59 (0.67) 19.84 (1.24) 17.16 (0.34) 56.46 (0.43) 19.63 (0.25) 133.93 (0.35) 41.15 (0.19) 87.63 (0.24) 26.43 (0.25) 44.58 (0.24) 15.18 (0.15) 30.01 (0.17) 43.05 (0.38)

2 3 4 5 6 7 8 9 10 11 12 Mean

be considered when using frequency spectra as BCI features. On the other hand, analysis strategies that try to investigate IAF or other aspects of the EEG should consider one of the two total spectra. 4. Discussion Our goal was to introduce FM flickering LEDs as a new stimulation method for SSVEP research and as a possible new tool for BCI

systems. With this study we made a first step toward that direction by showing that different FM signals with carrier frequencies up to 100 Hz can evoke SSVEPs reliably. Note, that SSVEPs were only recorded with frequencies of the lower sideband of the modulated signal, which was always at 10 Hz, and not at the carrier frequencies themselves, though the power in the stimulation data is distributed into the carrier frequency and an upper/lower sideband (cf. Fig. 1). Clear harmonic peaks can only be observed in spectra generated by strongly overlapping epochs. Interestingly, the

10 Hz sine

Fc 40 Hz Total spectrum of raw EEG

Total spectrum of raw EEG μV

0 0 3

B

0.5

10

20

30

40

0 0

Hz

3

Total spectrum averaged across 1 s epochs

2 1 0 0

10

20

30

40

0 0

Hz

30

40

Hz

Total spectrum averaged across 1 s epochs

10

20

30

40

Hz

40

Hz

40

Hz

Non−overlapping evoked spectrum 0.5 μV

0.5

0 0

10

20

30

40

0 0

Hz

Overlapping evoked spectrum

10

20

30

Overlapping evoked spectrum

0.5

0.5 μV

D

20

1

Non−overlapping evoked spectrum

C

10

2 μV

A

0.5

0 0

10

20

30

40

Hz

0 0

10

20

30

Fig. 5. Exemplary spectra from exemplary single subject analysis of two stimulation conditions. (A) Total spectra over the complete 100 s stimulation period. (B) Total averaged spectra generated from non-overlapping 1 s epochs. (C) Non-overlapping evoked spectra generated from non-overlapping 1 s epochs. (D) Overlapping evoked spectra generated from strongly (90%) overlapping 1 s epochs.

A.M. Dreyer, C.S. Herrmann / Journal of Neuroscience Methods 241 (2015) 1–9

amplitudes and SNRs of the respective SSVEPs neither differed between the FM conditions nor did they differ from the commonly used sinusoidal or rectangular flicker conditions. Consideration of the perceptibility scores reveals the benefits of FM stimulation. While the SSVEP amplitude at 10 Hz remains similar, the subjective flicker perceptibility strongly decreases with increasing carrier frequencies up to 40 Hz and remains on a very low level at higher frequencies. This implies that subjects only perceive a slight flicker or even no flicker at all with carrier frequencies above 30 Hz. In accordance with Lin et al. (2012), this suggests that higher FM frequency pairs lead to less perceptibility of the flicker and also to less subjective annoyance and possibly reduced eye fatigue. It is intriguing that we can use FM stimulation to evoke a 10 Hz SSVEP peak without a conscious perception of a 10 Hz flicker. Why is that the case? The visual system detects fluctuations of light intensity only if they are below CFF. Mathematically this is similar to computing an envelope of the visual signal after passing it through a low-pass filter with a cutoff frequency at CFF. The envelope of a 10 Hz sine wave – which is below CFF – is identical to the sine wave and thus the 10 Hz flicker can be perceived. The envelopes of the used FM signals with carrier frequencies above CFF are straight horizontal lines, i.e. the very brief drops in brightness are too quick for our visual system to be detected and thus cannot be perceived. In the auditory domain, it is common to use amplitude modulated signals in order to evoke ASSRs (Ross et al., 2000). For that purpose an oscillatory signal at a carrier frequency fc is modulated by a modulation frequency fm . As shown in Fig. 1, this results in spectral peaks of the auditory signal at fc and fc ± fm . Recently, Chang et al. (2014) used AM stimulation also in the visual domain to evoke SSVEPs. Interestingly, SSVEPs were evoked at the expected spectral side-bands fc ± fm . In contrast, AM stimulation in the auditory domain leads to ASSR peaks at fm – a frequency which is not contained in the spectrum of the auditory signal (cf. Fig. 1C). If a 1000 Hz sine tone is modulated with a frequency of 40 Hz, this results in spectral peaks at 960, 1000, and 1040 Hz but no peak at 40 Hz. Nevertheless, 40 Hz ASSRs can be evoked with such an AM sound. It has been assumed that neurons are able to detect the envelope of the oscillatory auditory signal which indeed oscillates at 40 Hz. In our study, SSVEPs revealed a prominent spectral peak at 10 Hz, i.e. at fc − fm (cf. Fig. 4) as would be expected from the spectrum of the visual stimulation (cf. Fig. 1). An interesting question relates to the neural mechanism that may be responsible for SSVEPs resulting from FM stimuli. Multiple possible mechanisms for the neural detection of FM signals in the auditory domain have been discussed and are described elsewhere (Picton et al., 1987; Luo et al., 2007). In principle, whenever multiple neurons reveal different tuning curves of their firing to certain temporal frequencies of the input, this can result in a response to FM signals. When the frequency of the input changes, different neurons are excited at different times to different degrees (Heil, 1997). This mechanism might also be at work in the visual system, since neurons in the visual system also reveal temporal frequency tuning (Foster et al., 1985). This may be the general mechanism underlying neural responses to FM stimulation. However, in our experiment we have used different FM stimulation protocols that all had their lower sidebands at 10 Hz. In this case, an additional mechanism comes into play. It has been shown that single neurons as well as neuronal assemblies show resonant behavior and can act as neural oscillators (Lampl & Yarom, 1997). EEG signals frequently reveal neural oscillations resulting in peaks in frequency spectra (Herrmann et al., 2014). Such neural oscillators are susceptible to visual input that oscillates at the corresponding resonance frequency. Visual input flickering at 10 Hz, for example, results in a strong resonance peak of the SSVEP spectrum around 10 Hz (Herrmann, 2001). In our experiment, each FM signal had a side-band at 10 Hz which coincides with this resonance peak.

7

Therefore, we argue that the SSVEPs observed in this study result from an excitation of the neuronal oscillator that is responsible for this 10 Hz resonance peak. It should be noted, however, that it is a matter of debate whether SSVEPs are a superposition of single responses to each of the repetitive light stimuli or whether an internal neural oscillator is driven by an external repetitive stimulus (Capilla et al., 2011). These two cases can only be differentiated by future experiments that use different stimulation intensities at multiple frequencies in order to demonstrate a synchronization of the neural oscillator to the external driving force (Pikovsky et al., 2003). In fact, the 10 Hz peak of our SSVEPs in response to FM stimulation at carrier frequencies of 100 Hz speaks in favor of the EEG synchronizing to the oscillatory visual input, since there are no discrete light flashes at 10 Hz but rather the unperceivable 100 Hz oscillation is modulated in its frequency at a rate of 10 Hz. It is also interesting to note that the amplitude of the spectral peak at 10 Hz is not significantly different between sinusoidal, square and FM driving (cf. Fig. 4, right column). This is notable, because the side band peak at 10 Hz for FM is actually lower than that for a pure 10 Hz sine wave (cf. Fig. 1 A vs. D). However, the EEG alpha oscillator, just like physical oscillators, is most probably susceptible not only to its resonance frequency but also to integer multiples – a phenomenon referred to in physics as sub-harmonic resonance (Othmer & Xie, 1999). For example, (Herrmann, 2001) were able to show that light flickering at 80 Hz evoked a 10 Hz response in the SSVEP, i.e. an 1:8 sub-harmonic resonance. The FM signals used in our study have peaks not only at 10 Hz (fc − fm ) but at two integer multiples fc and fc + fm . Therefore, the alpha oscillator is driven by three spectral peaks which might explain the unexpectedly large peak at 10 Hz. Another question addresses the ecological validity of FM signals in the visual domain. In the auditory domain, FM signals are present in our environment as for example in form of bird chirps and speech sounds. Therefore, it seems plausible that our auditory system has the ability to detect such FM signals. In the visual system, moving objects that change their velocity result in FM signals when their contours excite receptive fields. A striped object moving at constant velocity across the receptive field of a neuron in visual cortex is similar to a certain frequency of single light flashes, i.e. the spatial frequency of the objects is transformed into a temporal frequency of responses. A decrease of velocity is similar to a downward modulation of the frequency of light flashes. An increase is similar to an upward modulation of the frequency of light flashes. Therefore, it seems plausible to assume that temporal FM signals are detected by our visual system. The plateau of perceptibility scores for carrier frequencies above 30 Hz might partly be a result of our perceptibility questionnaire which does not differentiate anymore between ‘slight flicker’ perception and ‘no flicker’ perception. Furthermore, the choice of a green LED made it harder to reach the CFF which is known to be highest for green light because of the fast green cone pathways for CFF-near stimuli (Levin et al., 2011). The total darkness in our recording chamber might also play a role. It can be expected that more background light in the room would have made it harder to perceive the flicker because this would have decreased the perceived brightness of our stimuli (Kalloniatis & Luu, 2007). Some subjects orally reported that they think they only perceived the flicker because of the darkness. Real life applications would most likely not be used in a totally dark chamber, but note that while decreasing stimulus perceptibility, background light can also reduce BCI performance (Allison et al., 2010). If further experiments generate similar results regarding the perceptibility, one could modify the concept of the CFF for FM stimulation to reflect the carrier frequency value at which minimal perceptibility of a

8

A.M. Dreyer, C.S. Herrmann / Journal of Neuroscience Methods 241 (2015) 1–9

flickering stimulus is reached, which will not change anymore with higher carrier frequency values, as long as the distance between carrier and modulation frequency stays the same. Like the CFF for constant-frequency flickering stimuli, this value will strongly depend on stimulation parameters and experimental conditions, meaning that the value of 40 Hz we found with our experimental setup is not to be generalized. The reported illusory motion of the LED probably relates to the autokinetic illusion (Levy, 1972). If a spot of light is observed in an otherwise dark room, the spot appears to drift across visual space. This is thought to be due to the absence of visual reference points in the dark. It has been demonstrated that this illusion is even stronger for flickering spots of light (Elfner & Page, 1963). Therefore, we assume that some of our subjects observed an autokinetic illusion due to the dark EEG recording chamber. It is also important to mention that we instructed the subjects to fixate the LED at all times. In a realistic BCI application one would have more than one LED and some kind of feedback and the subjects would not constantly fixate one stimulus. The effect of an increasing number of stimuli and their constantly shifting position in the visual field on the perceptibility of FM stimuli has to be tested. SSVEPs and visual perceptibility/the CFF strongly depend on the visual stimuli used and we are aware that our experiment is limited regarding the frequency pairs we tested and also regarding the stimuli because we only tested one specific LED. With FM stimulation and visual flickering stimuli in general the possibilities are manifold. Therefore, we hope that other researchers will pick up the idea of FM stimulation as a new research tool and that they might even test them on existing BCI systems. A direct comparison of AM(Chang et al., 2014) and FM-SSVEPS would also be very helpful. Both were introduced with the aim of more user friendly BCIs and a comparison could reveal, which approach future research should focus on but this obviously needs testing both under identical conditions with identical stimulation devices. To follow up this study, we are currently trying to extend the FM-SSVEP frequency range with FM-sidebands in the beta range as well as the number of simultaneously flickering LEDs, which will then be tested on a BCI implementation. In relation to the aforementioned LED-keyboard (Hwang et al., 2012) we imagine that it is possible to reduce the perceptibility of the individually flickering LEDs strongly while still being able to reliably train a classifier on the SSVEPs. In the best case, such a keyboard would not look much different from commercially available backlight keyboards. To really be able to use FM stimulation for the development of such keyboards and other new BCI applications, an extensive analysis of different stimulation parameters like number of stimuli, stimulus size, stimulation frequencies and others (similar to Ng et al., 2012) is essential and should in our view be the next step of FM-SSVEP research. Acknowledgement The study was supported by the Deutsche Forschungsgemeinschaft (DFG), grants SFB/TRR 31 and SPP 1665 (C.S.H.). References Allison B, Luth T, Valbuena D, Teymourian A, Volosyak I, Graser A. BCI demographics: how many (and what kinds of) people can use an SSVEP BCI? IEEE Trans Neural Syst Rehabil Eng 2010;18(2):107–16, http://dx.doi.org/10.1109/TNSRE.2009.2039495. Bach M, Meigen T. Do’s and don’ts in Fourier analysis of steady-state potentials. Doc Ophtalmol 1999;99(1):69–82. Bengtsson H. R.matlab: Read and write of MAT files together with R-to-MATLAB connectivity; 2014, https://github.com/HenrikBengtsson/R.matlab/ Cao T, Wan F, Wong CM, da Cruz JN, Hu Y. Objective evaluation of fatigue by EEG spectral analysis in steady-state visual evoked potential-based

brain–computer interfaces. Biomed Eng Online 2014;13(1):28, http://dx.doi.org/10.1186/1475-925X-13-28. Capilla A, Pazo-Alvarez P, Darriba A, Campo P, Gross J. Steady-state visual evoked potentials can be explained by temporal superposition of transient event-related responses. PLOS ONE 2011;6(1):e14543. Chang MH, Baek HJ, Lee SM, Park KS. An amplitude-modulated visual stimulation for reducing eye fatigue in SSVEP-based brain–computer interfaces. Clin Neurophysiol 2014;125(7):1380–91, http://dx.doi.org/10.1016/j.clinph.2013.11.016. Cheng M, Gao X, Gao S, Xu D. Design and implementation of a brain–computer interface with high transfer rates. IEEE Trans Bio-Med Eng 2002;49(10):1181–6. Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods 2004;134(1):9–21, http://dx.doi.org/10.1016/j.jneumeth.2003.10.009. Elfner LF, Page HA. Autokinetic enhancement as a function of flicker. Percept Motor Skills 1963;17:299–301, http://dx.doi.org/10.2466/pms.1963.17.1.299. Foster KH, Gaska JP, Nagler M, Pollen DA. Spatial and temporal frequency selectivity of neurones in visual cortical areas V1 and V2 of the macaque monkey. J Physiol 1985;365:331–63. Halbleib A, Gratkowski M, Schwab K, Ligges C, Witte H, Haueisen J. Topographic analysis of engagement and disengagement of neural oscillators in photic driving: a combined electroencephalogram/magnetoencephalogram study. J Clin Neurophysiol 2012;29(1):33–41, http://dx.doi.org/10.1097/WNP.0b013e318246ad6e. Heil P. Aspects of temporal processing of FM stimuli in primary auditory cortex. Acta Oto-Laryngol Suppl 1997;532:99–102. Herrmann CS. Human EEG responses to 1–100 Hz flicker: resonance phenomena in visual cortex and their potential correlation to cognitive phenomena. Exp Brain Res 2001;137(3–4):346–53, http://dx.doi.org/10.1007/s002210100682. Herrmann CS, Rach S, Vosskuhl J, Strüber D. Time–frequency analysis of event-related potentials: a brief tutorial. Brain Topogr 2014;27(4):438–50, http://dx.doi.org/10.1007/s10548-013-0327-5. Hwang H-J, Lim J-H, Jung Y-J, Choi H, Lee SW, Im C-H. Development of an SSVEP-based BCI spelling system adopting a QWERTYLED keyboard. J Neurosci Methods 2012;208(1):59–65, style http://dx.doi.org/10.1016/j.jneumeth.2012.04.011. Kalloniatis M, Luu C. Temporal resolution. In: Kolb H, Fernandez E, Nelson R, editors. Webvision: the organization of the retina and visual system. University of Utah Health Sciences Center; 2007. Kimura Y, Tanaka T, Higashi H, Morikawa N. SSVEP-based brain–computer interfaces using FSK-modulated visual stimuli. IEEE Trans Bio-Med Eng 2013;60(10):2831–8, http://dx.doi.org/10.1109/TBME.2013.2265260. ˛ ´ Ku´s R, Duszyk A, Milanowski P, Labecki M, Bierzynska M, Radzikowska Z, et al. On the quantification of SSVEP frequency responses in human EEG in realistic BCI conditions. PLOS ONE 2013;8(10):e77536, http://dx.doi.org/10.1371/journal.pone.0077536. Lampl I, Yarom Y. Subthreshold oscillations and resonant behavior: two manifestations of the same mechanism. Neuroscience 1997;78(2):325–41. Levin LA, Nilsson SFE, ver Hoeve J, Wu S, Kaufman PL, Alm A. Adler’s physiology of the eye: expert consult – online and print, 11e. Saunders; 2011. Levy J. Autokinetic illusion: a systematic review of theories, measures, and independent variables. Psychol Bull 1972;78(6):457–74. Lin F-C, Zao JK, Tu K-C, Wang Y, Huang Y-P, Chuang C-W, et al., Jung T-P. SNR analysis of high-frequency steady-state visual evoked potentials from the foveal and extrafoveal regions of human retina. In: Conference Proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2012; 2012. p. 1810–4, http://dx.doi.org/10.1109/EMBC.2012.6346302. Luo H, Wang Y, Poeppel D, Simon JZ. Concurrent encoding of frequency and amplitude modulation in human auditory cortex: encoding transition. J Neurophysiol 2007;98(6):3473–85, http://dx.doi.org/10.1152/jn.00342.2007. Martens U, Hübner R. Functional hemispheric asymmetries of global/local processing mirrored by the steady-state visual evoked potential. Brain Cogn 2013;81(2):161–6, http://dx.doi.org/10.1016/j.bandc.2012.11.005. Meigen T, Bach M. On the statistical significance of electrophysiological steady-state responses. Doc Ophthalmol 1999;98(3):207–32. Müller MM, Picton TW, Valdes-Sosa P, Riera J, Teder-Sälejärvi WA, Hillyard SA. Effects of spatial selective attention on the steady-state visual evoked potential in the 20–28 Hz range. Brain Res Cogn Brain Res 1998;6(4):249–61. Ng KB, Bradley AP, Cunnington R. Stimulus specificity of a steady-state visual-evoked potential-based brain–computer interface. J Neural Eng 2012;9(3):036008, http://dx.doi.org/10.1088/1741-2560/9/3/036008. Othmer HG, Xie M. Subharmonic resonance and chaos in forced excitable systems. J Math Biol 1999;39(2):139–71. Picton TW, John MS, Dimitrijevic A, Purcell D. Human auditory steady-state responses. Int J Audiol 2003;42(4):177–219. Picton TW, Skinner CR, Champagne SC, Kellett AJ, Maiste AC. Potentials evoked by the sinusoidal modulation of the amplitude or frequency of a tone. J Acoust Soc Am 1987;82(1):165–78. Pikovsky A, Rosenblum M, Kurths J. Synchronization: a universal concept in nonlinear sciences. Cambridge University Press; 2003. R Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2013. Regan D. Human brain electrophysiology: evoked potentials and evoked magnetic fields in science and medicine. New York: Elsevier; 1989.

A.M. Dreyer, C.S. Herrmann / Journal of Neuroscience Methods 241 (2015) 1–9 Ross B, Borgmann C, Draganova R, Roberts LE, Pantev C. A high-precision magnetoencephalographic study of human auditory steady-state responses to amplitude-modulated tones. J Acoust Soc Am 2000;108(2):679, http://dx.doi.org/10.1121/1.429600. Ross B, Picton TW, Herdman AT, Pantev C. The effect of attention on the auditory steady-state response. Neurol Clin Neurophysiol 2004; 2004:22. Silberstein RB. Steady-state visually evoked potentials, brain resonances, and cognitive processes. In: Nunez PL, editor. Neocortical dynamics and human EEG rhythms. Oxford: Oxford University Press; 1995. p. 272–303.

9

Vialatte F-B, Maurice M, Dauwels J, Cichocki A. Steady-state visually evoked potentials: focus on essential paradigms and future perspectives. Prog Neurobiol 2010;90(4):418–38, http://dx.doi.org/10.1016/j.pneurobio.2009.11.005. Volosyak I, Valbuena D, Lüth T, Malechka T, Gräser A. BCI demographics II: how many (and what kinds of) people can use a high-frequency SSVEP BCI? IEEE Trans Neural Syst Rehabil Eng 2011;19(3):232–9, http://dx.doi.org/10.1109/TNSRE.2011.2121919. Zaehle T, Lenz D, Ohl FW, Herrmann CS. Resonance phenomena in the human auditory cortex: individual resonance frequencies of the cerebral cortex determine electrophysiological responses. Exp Brain Res 2010;203(3):629–35, http://dx.doi.org/10.1007/s00221-010-2265-8.