SeDMiD for Confusion Detection: Uncovering Mind State from Time Series Brain Wave Data

Abstract

Understanding how brain functions has been an intriguing topic for years. With the recent progress on collecting massive data and developing advanced technology, people have become interested in addressing the challenge of decoding brain wave data into meaningful mind states, with many machine learning models and algorithms being revisited and developed, especially the ones that handle time series data because of the nature of brain waves. However, many of these time series models, like HMM with hidden state in discrete space or State Space Model with hidden state in continuous space, only work with one source of data and cannot handle different sources of information simultaneously. In this paper, we propose an extension of State Space Model to work with different sources of information together with its learning and inference algorithms. We apply this model to decode the mind state of students during lectures based on their brain waves and reach a significant better results compared to traditional methods.

Addressing Data Scarcity in Multimodal User State Recognition by Combining Semi-Supervised and Supervised Learning

“…Similarly to Wang et al, Bousmalis et al developed an approach based on a hidden conditional random field algorithm that, through the combination of verbal and nonverbal cues, achieved a combined accuracy of 64.2% for dis-/agreement detection [5]. The detection of confusion is primarily focused on EEG-based data [32,45,49] or based on natural language processing with text input [17,47]. Using automatically recognized Action Units from the Facial Action Coding System, Borges et al trained an LSTM neural network on their own collected data, with which they reported an F1-score of 80.89% for their confusion class.…”