This paper proposes two multimodal fusion methods between brain and peripheral signals for emotion recognition. The input signals are electroencephalogram and facial expression. The stimuli are based on a subset of movie clips that correspond to four specific areas of valance-arousal emotional space (happiness, neutral, sadness, and fear). For facial expression detection, four basic emotion states (happiness, neutral, sadness, and fear) are detected by a neural network classifier. For EEG detection, four basic emotion states and three emotion intensity levels (strong, ordinary, and weak) are detected by two support vector machines (SVM) classifiers, respectively. Emotion recognition is based on two decision-level fusion methods of both EEG and facial expression detections by using a sum rule or a production rule. Twenty healthy subjects attended two experiments. The results show that the accuracies of two multimodal fusion detections are 81.25% and 82.75%, respectively, which are both higher than that of facial expression (74.38%) or EEG detection (66.88%). The combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources.
Emotion recognition plays an essential role in human–computer interaction. Previous studies have investigated the use of facial expression and electroencephalogram (EEG) signals from single modal for emotion recognition separately, but few have paid attention to a fusion between them. In this paper, we adopted a multimodal emotion recognition framework by combining facial expression and EEG, based on a valence-arousal emotional model. For facial expression detection, we followed a transfer learning approach for multi-task convolutional neural network (CNN) architectures to detect the state of valence and arousal. For EEG detection, two learning targets (valence and arousal) were detected by different support vector machine (SVM) classifiers, separately. Finally, two decision-level fusion methods based on the enumerate weight rule or an adaptive boosting technique were used to combine facial expression and EEG. In the experiment, the subjects were instructed to watch clips designed to elicit an emotional response and then reported their emotional state. We used two emotion datasets—a Database for Emotion Analysis using Physiological Signals (DEAP) and MAHNOB-human computer interface (MAHNOB-HCI)—to evaluate our method. In addition, we also performed an online experiment to make our method more robust. We experimentally demonstrated that our method produces state-of-the-art results in terms of binary valence/arousal classification, based on DEAP and MAHNOB-HCI data sets. Besides this, for the online experiment, we achieved 69.75% accuracy for the valence space and 70.00% accuracy for the arousal space after fusion, each of which has surpassed the highest performing single modality (69.28% for the valence space and 64.00% for the arousal space). The results suggest that the combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources. The novelty of this work is as follows. To begin with, we combined facial expression and EEG to improve the performance of emotion recognition. Furthermore, we used transfer learning techniques to tackle the problem of lacking data and achieve higher accuracy for facial expression. Finally, in addition to implementing the widely used fusion method based on enumerating different weights between two models, we also explored a novel fusion method, applying boosting technique.
Merging behavior is inevitable for drivers at on-ramp bottlenecks and a significant factor in triggering a traffic breakdown. Empirical data were collected by extracting trajectories from merging vehicles and adjacent vehicles at two on-ramp bottlenecks in Shanghai, China. These data included 58 normal (free-flow) lane changes (NLCs), 36 cooperative lane changes (CLCs), 135 forced lane changes (FLCs), and 188 unsuccessful lane changes (USLCs). The objective was to develop and compare five discrete choice models (two multinomial logit and three nested logit) to understand merging behavior at on-ramp bottlenecks better. Estimation results showed that the two-level nested logit model considering three merging types (NLC, CLC, and FLC) provided the best fit. The traffic flow condition (bottleneck), the time gap and the space gap of the lag vehicle, and the speed of the merging vehicle were key factors when choosing merging types. The resulting quantitative models can be used to perform a microscopic analysis of the breakdown mechanism and develop a traffic simulation model.
To increase the capacity of urban expressways in Shanghai, China, additional lanes were created during the past decade through reconstruction. Field measurements indicate that maximum lane width is 3.97 m and minimum width is only 2.73 m. To investigate the relationship between lane width and capacity for urban expressways, 3 months of traffic flow data were extracted and filtered from the system to manage inductive detectors on Shanghai's urban expressways. Analysis of all 440 segments of expressways in Shanghai showed that only 60 sites could reach their capacity; the characteristics of these 60 sites were further analyzed statistically. An analysis of a variance, a regression analysis, and a t-test were used to explore the relationship between lane width and capacity. The research showed that lane width had no statistically significant effect on capacity. Two causes that might account for this finding are discussed further: (a) free-flow speeds are similar for different lane widths according to the findings of a t-test that found no statistically significant differences and (b) the critical speed when capacity is reached for a Shanghai expressway is very low, only 45 km/h. Thus, drivers can deal with a narrow lane width at such a low critical speed. This finding suggests that geometric design policies for capacity purposes should provide substantial flexibility for use of narrower lane widths on urban expressways with low speed limits, although those lane widths must be subject to safety considerations.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.