Multimodal Approach for Emotion Recognition Based on Simulated Flight Experiments

Abstract

The present work tries to fill part of the gap regarding the pilots' emotions and their bio-reactions during some flight procedures such as, takeoff, climbing, cruising, descent, initial approach, final approach and landing. A sensing architecture and a set of experiments were developed, associating it to several simulated flights (N f lights = 13) using the Microsoft Flight Simulator Steam Edition (FSX-SE). The approach was carried out with eight beginner users on the flight simulator (N pilots = 8). It is shown that it is possible to recognize emotions from different pilots in flight, combining their present and previous emotions. The cardiac system based on Heart Rate (HR), Galvanic Skin Response (GSR) and Electroencephalography (EEG), were used to extract emotions, as well as the intensities of emotions detected from the pilot face. We also considered five main emotions: happy, sad, angry, surprise and scared. The emotion recognition is based on Artificial Neural Networks and Deep Learning techniques. The Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) were the main methods used to measure the quality of the regression output models. The tests of the produced output models showed that the lowest recognition errors were reached when all data were considered or when the GSR datasets were omitted from the model training. It also showed that the emotion surprised was the easiest to recognize, having a mean RMSE of 0.13 and mean MAE of 0.01; while the emotion sad was the hardest to recognize, having a mean RMSE of 0.82 and mean MAE of 0.08. When we considered only the higher emotion intensities by time, the most matches accuracies were between 55% and 100%.

Emotion and Stress Recognition Related Sensors and Machine Learning Technologies

“…The authors in [ 25 ] proposed a multimodal approach to emotion recognition in the aviation domain with the goal of filling some of the gap between pilots’ emotions and their bioreactions during flight procedures such as take-off, climbing, cruising, descent, initial approach, final approach and landing. Building around a sensing architecture and a set of simulated flight experiments, the study showed that it was indeed possible to recognize emotions from different pilots in flight, combining their present and previous emotions.…”

β-Band Analysis from Simulated Flight Experiments

“…The setup is described in Figure 2, of which the EEG context (the biosignal analyzed in this present work) represents part of the proposed multisensing architecture described in [7]. In addition, this final setup was the improvement of two different proof of concepts (PoCs) [5], resulting in several changes: a large screen to improve the immersive experience during the simulation, maintaining an average distance of 1.70 to 1.90 m of the volunteer; a computer to run the flight simulator and to record facial expression; the use of only one hand to control the aircraft by the joystick, due to the GSR sensors being placed on another hand; a headcap with dry electrodes that was used to acquire EEG data; the GSR electrodes placed on the free hand, i.e., without movements to avoid motion artifacts; a microcontroller (e.g., Arduino board) used to acquire the HR data of the HR device (Medlab P100); the supervisor using two computers, one to receive HR and GSR data of Bluetooth communication, and another to receive the Bluetooth data of EEG device; and an auxiliary or additional camera, used to record the volunteers' body gestures.…”

“…These 13 datasets include the emotion questionnaires, face recordings, HR, GSR and EEG data. The dataset names are a sequence of two letters and one number to indicate the volunteer name and the flight sequence of such volunteer, respectively, [5][6][7].…”

“…It is important to emphasize that in this present work, only the EEG datasets were considered. Other datasets such as HR and GSR were only presented in this section, to show the reason of the EEG dataset resampling, since it is a continuation of a bigger experiment [5].…”

“…These volunteers are not real pilots, and before the experiment, they signed a consent form. Furthermore, this work is a continuation of the work presented by [5][6][7], with the possibility to adapt the present work methodology (changing the experiments and results) to be applied on other contexts, e.g., smart city [8], biophilia, automobilism [9], music [10] or even administrative works (secretaries or customer services, for instance). Regarding the aviation context, this research brings a contribution, showing that it is indeed necessary to improve the training of the pilot, decreasing the cognition effect over their actions and thus attenuating the level of brain activity (also called as brain energy) during critical phases of a real flight.…”