This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2015.2415520, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00218

1

EEG-Based Attention Tracking during Distracted Driving Yu-Kai Wang, Tzyy-Ping Jung, Fellow, IEEE, and Chin-Teng Lin, Fellow, IEEE  Abstract—Distracted driving might lead to many catastrophic consequences. Developing a countermeasure to track drivers’ focus of attention (FOA) and engagement of operators in dual (multi)–tasking conditions is thus imperative. Ten healthy volunteers participated in a dual-task experiment that comprised two tasks: a lane-keeping driving task and a mathematical problem-solving task (e.g., 24+15=37?) during which their electroencephalogram (EEG) and behaviors were concurrently recorded. Independent Component Analysis (ICA) was employed as a spatial filter to separate the contributions of independent sources from the recorded EEG data. The power spectra of six components (i.e., frontal, central, parietal, occipital, left motor, and right motor) extracted from single-task conditions were fed into support vector machine (SVM) based on the radial basis function (RBF) kernel to build an FOA assessment system. The system achieved 84.6±5.8% and 86.2±5.4% classification accuracies in detecting the participants’ FOAs on the math vs. driving tasks, respectively. This FOA assessment system was then applied to evaluate participants’ FOAs during dual-task conditions. The detected FOAs revealed that participants’ cognitive attention and strategies dynamically changed between tasks to optimize the overall performance, as attention was limited and competed. The empirical results of this study demonstrate the feasibility of a practical system to continuously estimating cognitive attention through EEG spectra. Index Terms—Distracted Driving, Focus of Attention, EEG

D

I. INTRODUCTION

riving is a skill that requires drivers to direct their full attention to control the cars [1]. For safe vehicle operation, drivers must concurrently process incoming information and stimuli from the dashboard, road, and environment, among others. The human brain also needs to coordinate information from various senses such as vision, hearing, body sensation, Manuscript received August 9, 2014; revised January 09, 2015; accepted March 3, 2015. Date of current version March 19, 2015.This work was supported in part by the UST-UCSD International Center of Excellence in Advanced Bioengineering sponsored by the Taiwan Ministry of Science and Technology I-RiCE Program under Grant MOST-103-2911-I-009-101, in part by the Aiming for the Top University Plan of National Chiao Tung University, the Ministry of Education, Taiwan, under Contract 104W963. Research was also sponsored in part by the Army Research Laboratory and was accomplished under Cooperative Agreement W911NF-10-2-0022. Asterisk indicates corresponding author. Y.-K. Wang and *C.-T. Lin are with the Department of Computer Science and Brain Research Center, National Chiao Tung University, Hsinchu, Taiwan (e-mail: [email protected]; [email protected]). *T.-P. Jung are with the Swartz Center for Computational Neuroscience, Institute of Engineering in Medicine, University of California-San Diego, La Jolla, CA, USA (e-mail: [email protected]).

and balance. Thus, lack of attention to relevant driving events is a leading cause of catastrophic motor vehicle crashes. Currently, the technological and informational capabilities of our driving environment have significantly increased [2]. The operation of automotive electronic devices and mobile devices has been greatly augmented during driving. Some of these technological devices increase safety and reduce the drivers’ attention load [3]–[5]. For example, adaptive cruise control systems automatically adjust speed to maintain a safe following distance from the vehicles in front of a driver [5]. Navigation systems are nearly indispensable for suggesting or providing directions. These in-vehicle systems are designed to improve safety and convenience. However, operating the in-vehicle systems sometimes impairs driver’s attention that is taken away from the primary driving tasks [3]. In particular, conversing on the phone while driving is easily distracting, even with hands-free systems [6]–[9]. The contents of the conversation have a greater effect on distracted driving than does the method of phone conversation [9]. Drivers divert a portion of their attention from the driving task to the conversation, which deteriorates driving performance. The negative effects of cell-phone conversations on driving safety have led to legislation in nearly every country now. In short, increasing the number of in-vehicle devices may reduce the driver’s attention and increase the possibility of distractions. The Yerkes-Dodson law details the empirical relationship between arousal and performance; in general, the performance improves as arousal increases [10], [11]. Although the individuals have high levels of arousal, the performance would be impaired under more complex or difficult situations, such as multitasking and working memory tasks [10], [11]. During driving, a large number of studies have reported that performing another cognitive task decreases driving performance [6]–[9], [12]–[16], but some aspects of driving performance are unaffected while performing the additional tasks concurrently [17]. In these present dual-task experiments, the participants are usually instructed to respond to the tasks as quickly and accurately as possible. Regardless of whether behavioral performance is significantly different in the distracted driving conditions, the mental workload or arousal should be influenced while drivers have to shift their attention away from the primary driving tasks to the additional events [6], [7], [12], [13], [15], [16]. Since distracted driving is a serious threat in daily life, it is imperative to develop a counter measure that is capable of detecting the driver’s attention. According to the behavioral

1534-4320 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2015.2415520, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00218 information such as gaze angle, head angle, lane position, and facial expression, Kutila proposed a support vector machine (SVM)-based model to detect drivers’ cognitive distraction with an approximately 65-80% accuracy rate [18]. In contrast to the behavioral information, physiological measurements related to attention and workload have been studied for many years, and the physiological information is more sensitive to the changes in cognition [19], [20]. The relationship between increased task demands and changes in heart rate, skin conductance, and respiration rate have been verified [16]. As the improvement of brain imaging techniques such as functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (fNIRS), electroencephalography (EEG), and magnetoencephalography (MEG), the activity of large neural ensembles is also one important source for exploring the changes of attention [21]–[23]. The aim of this study was to track the drivers’ focus of attention (FOA) during the dual-task conditions based on neurophysiological data. This study first assessed the differences in behavioral performance and EEG activity when participants performed a lane-keeping driving task and a mathematical problem-solving task (e.g., 24+15=37?). An FOA classifier that was built based on the data collected from the single-task conditions was then used to track the participants’ FOAs continuously during the dual-task conditions. The hypothesis is that the measured brain signals from the dual-task conditions would include the activities elicited by specific tasks that were performed in isolation [24]. Based on this previous finding, we demonstrated the feasibility of tracking FOA through brain activity collected from the scalp non-invasively in this study. II. MATERIALS AND METHODS A. Experiment and Subjects Continuous EEG data were recorded from ten healthy participants (20-28 years of age; mean age: 24.3 years; all males) at the National Chiao Tung University, Hsinchu, Taiwan. All participants had normal or corrected-to-normal vision and had owned legal driving licenses for more than one year. Caffeine, tobacco, alcohol, and drugs were prohibited before participating in the experiment. Before the experiment, the participants have two training sessions to acquaint themselves with the tasks (turning the steering wheel following lane deviation and button press for indicating reaching solutions to math equations) and the virtual-reality environment. Two designed tasks were always executed in the daily life; moreover, the associated responses for the designed tasks were same as the operations during driving. All participants reported that they were familiar with the experimental setups and tasks after the two training sessions. Experimental results also showed that the mean RT to lane deviation improved from 819.3 (in training sessions) to 665.7 ms (in regular sessions), whereas the mean ST of math problems improved from 1878.3 to 1666.6 ms. After two training sessions, there were four 15-min experimental (regular) sessions separated by 10-min rests.

2 Two types of visual tasks, including a lane-keeping driving task and a mathematical problem-solving task, were utilized. The mathematical problem solving involves both visual and cognitive workload [25], so it is employed to induce the attentional demands. Throughout the experiment, the car cruised at a fixed speed of 100 km/hr on a highway scene and drifted randomly either to the left or right [26]. When the participants detected the deviation, they were instructed to steer the car back to the cruising lane quickly by turning the steering wheel. The cruising lane was the center lane (the third lane as numbered from the left). The latency between the deviation onset and turning the steering wheel was defined as the reaction time (RT). In the math problem-solving task, arithmetic equations were presented as white letters in a green board, which mimicked the appearance of traffic information on the highway. The mathematical problem-solving task required the participants to verify the arithmetic equations (i.e., whether the equations were correct or incorrect). The ratio of correct to incorrect equations presented was 50:50, and the difficulty was at the same level. When the equation was correct (incorrect), the participants were instructed to press the right (left) button mounted on the steering wheel by the thumb. The latency from the presentation of equation to button press was defined as the solution time (ST). Fig. 1 lists the five different conditions utilized in this study. The lane-deviation event or the mathematic equation independently appeared on the screen in Case 1 (Fig. 1(a)) or Case 2 (Fig. 1(b)), respectively. They are both single-task conditions. Case 3~5 are the dual-task conditions. In Case 4 (Fig. 1(d)), two tasks appeared simultaneously on the screen. In Case 3 (Fig. 1(c)) and Case 5 (Fig. 1(e)), the events appeared sequentially with 400 ms latency i.e. the stimulus onset asynchrony (SOA, defined as the time interval between the presentations of the lane-deviation and the arithmetic question). The intervals between two consecutive trials varied randomly from six to eight seconds. The order of these five conditions was pseudo-random, and the same condition did not occur in twice in succession. On average, there were 25 occurrences of each condition during the 15-min experimental session. To study the brain activity that was associated with distracted driving in a realistic manner, a dynamic motion simulator was built with virtual-reality (VR) technologies using a WTK library. This driving simulator included a real car mounted on a dynamic 6-degrees-of-freedom motion platform and 360o scenes in which the highway and driving scenes were rendered by seven LCD projectors. The vehicle movements were controlled by the steering wheel, and the scenes were immediately updated according to the participants’ operations. The participants could receive both visual and kinesthetic stimuli to simulate an authentic driving situation on a highway. B. EEG Data Acquisition and Processing The EEG data were acquired with an electrode cap with a 32 Ag/AgCl electrodes that were placed according to a modified international 10-20 system and recorded with a 16-bit quantization level at a sampling rate of 500 Hz (NeuroScan, NeuroScan Inc., Herndon, VA, USA). All electrodes were

1534-4320 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2015.2415520, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00218

3 different brain and non-brain signals. Each brain and non-brain IC is identified with an activity time course (its ‘activation’) and its projection to the recording electrode (its ‘scalp map’). ICs accounting for eye blinks are usually highly spatially stereotyped. ICs accounting for muscle activities are also usually well modelled by ICA. Muscle component scalp maps are vary depending on the muscle they represent. The component scalp maps typically exhibit a sharp polarity reversal at the muscle’s point of insertion into the skull, and the spectra of muscle component typically are high power at frequencies above 20 Hz [29]. Furthermore, muscle tension is typically not maintained throughout an entire experiment. Once a component has been identified as artifactual, it may be removed from the data by reversing the ICA linear unmixing process [28]. Artifactual components accounting for eye blinks, eye movements, muscle activity, and bad channels are isolated and removed [28]. Non-artifact IC scalp maps often strongly resemble the projection of a single dipole, allowing the location and orientation of the best-fitting equivalent dipole (or other source model) to be easily determined [29]. The activities from the frontal, central, parietal, occipital, left motor, and right motor areas were selected for further analyses.

Fig. 1. Five conditions in this study. D: Lane-deviation event onset. M: math problem-solving task onset. (a) Case 1: only the lane-deviation task appeared on the screen. (b) Case 2: only the mathematic equation appeared on the screen, and the participants had to provide answers as quickly and accurately as possible. (c) Case 3 and (e) Case 5: the driving and the math tasks appeared sequentially with 400 ms latency. (d) Case 4: the two tasks appeared simultaneously.

referenced to linked mastoids (A1 and A2), and a single ground electrode was attached to the forehead. The contact impedance between the EEG electrodes and the scalp reduced to below 10 kΩ with conductive gel. EEGLAB, an open source toolbox (Swartz Center for Computational Neuroscience, La Jolla, CA; http://www.sccn.ucsd.edu/eeglab), was utilized to process EEG data. All EEG data were down-sampled to 250 Hz. Next, a low-pass filter with a cut-off frequency of 50 Hz and high-pass filter with a cut-off frequency of 0.5 Hz were applied to remove the line noise and the DC drift, respectively. Because the five conditions were randomly displayed throughout the experiment, the recorded EEG signals related to each condition were extracted from the continuous data. The EEG signals with incorrect reactions were removed from further analyses since the number of incorrect responses was too few to obtain consistent EEG features (3.7±1.1%). The filtered EEG signals were decomposed into independent brain sources by independent component analysis (ICA) [27], [28]. Since two reference electrodes (A1 and A2) were excluded, 30 independent components (ICs) were separated by ICA from the 30 channels of the EEG signals. ICA recovers the ICs from the recorded data that are considered to be a linear mixture of activities arising from

C. Feature Processing and Classifying The recorded EEG signals in the single-task conditions were training and validating data to build the FOA assessment system because the participants’ FOAs were clearly on the known task during the single-task conditions. For example, during the math task, the participants were instructed to put forth their best efforts to solve the mental calculation, thus their FOAs were obviously on solving the math problems. The EEG data in the dual-task conditions were the testing data. The FOA models trained by the training and validating data were then applied to these testing data from dual-task conditions. More specially, to collect training and validating data, the continuous EEG recordings were first segmented into epochs around the lane-deviation or math events. Each epoch was then segmented into overlapping 400-ms windows advancing in 50-ms steps as shown in Fig. 2, resulting in a total of 17 400-ms samples from a 1.2-s (0-1200ms) epoch following a lane-deviation or a math event. Each 400ms sample was transferred into the time-frequency domain through Fast Fourier Transform (FFT). The spectral magnitudes of delta (1-3 Hz), theta (4-7 Hz), alpha (8-13 Hz), and low beta (14-20 Hz) bands were calculated. Since six components were selected for obtaining the general information in the whole brain, the dimension of each 400-ms sample was 24 (4 frequency bands * 6 components). Stepwise Linear discriminant analysis (SWLDA) was then applied to the processed validating data to select the significant features [30], [31]. For the testing data from the dual-task conditions, a total of 83 400-ms samples from a 4.5-s (-500ms – 4000ms) epoch surrounding a driving or math event were segmented, and the dimension of each 400-ms sample was also 24 (4 frequency bands * 6 components). Under the three dual-task conditions, the participants’ FOAs were unclear as they might shift their attention in the early, middle, or later stage of the second

1534-4320 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TNSRE.2015.2415520, IEEE Transactions on Neural Systems and Rehabilitation Engineering

TNSRE-2014-00218

4 analysis in repeated-measure ANOVA. The Wilcoxon sign-rank tests were also performed on the brain activities between reacting the pure math and pure driving conditions. To evaluate the classification performance statistically, one-way ANOVA was utilized first, and then Turkey's post-hoc test were performed for pair comparison. Additionally, linear correlation between the number of 400-ms windows in which the participants was focusing on driving and the STs was calculated by Pearson’s correlation coefficient. III. RESULTS

Fig. 2. Extracted features from the recorded EEG data. Where n and m are the number of EEG epochs collected from pure drive or pure math, respectively; k is the number of sample extracted form one EEG epoch. The training and validating data were extracted from the brain activities during the single-task conditions (Case 1 and Case 2). The component activations were segmented into overlapping 400-ms samples that were advanced in 50-ms steps, which totally resulted in 17 samples from each 1200-ms epoch. For each sample, the delta (1-3 Hz), theta (4-7 Hz), alpha (8-13 Hz), and low beta (14-20 Hz) oscillations were extracted by FFT. The feature dimension of each 400-ms sample is 24 (4 frequency bands * six selected components).

stimulus presentation. The behavioral information was an important index to evaluate their performance that was assumed to be associated with their FOAs. All epochs were pooled and sorted by STs across participants. Subsequently, the performance-sorted FOAs were smoothed through an equal-weight moving average method, with a window of twenty-epochs at a step of one-epoch. In particular, various machine-learning classifiers were employed to the training and validating data for a comparison; these classifiers included linear discriminant analysis (LDA), nearest mean classifier (NMSC), k-nearest neighbors (k-NN with k=13), Parzen density estimation (PARZEN), perceptron classifier (PERLC), discriminative restricted Boltzmann machine (DRBMC), and support vector machine with radial basis function (SVMRBF) (PRTools, http://prtools.org/; LIBSVM toolbox, http://www.csie.ntu.edu.tw/~cjlin/libsvm/). A five-fold cross validation was considered to evaluate the classification performance. In particular, the data were selected based on the epochs to separate completely. The outputs of the FOA assessment system were either D or M, which indicated that the participants were paying attention to the driving or the math task, respectively. The receiver operating characteristic (ROC) curves and area under the curve (AUC) were employed to evaluate the performance of various classifiers. D. Statistical Analysis Two single factor repeated-measure ANOVA was performed to compare the mean STs for math question or RTs of lane deviation among one single- and three dual-task conditions. Post-hoc Wilcoxon sign-rank test was conducted for follow-up

A. Behavior Performance The behavioral performance is one critical factor to reveal the cognitive impairments during the dual-task experiment. Table 1 lists the mean STs for the mental calculation and the RTs to the lane-deviation in each designed condition. In the single-task conditions, the RTs were significantly shorter (p

EEG-Based Attention Tracking During Distracted Driving.

Distracted driving might lead to many catastrophic consequences. Developing a countermeasure to track drivers' focus of attention (FOA) and engagement...
3MB Sizes 0 Downloads 4 Views