Multimodal approach for cognitive task performance prediction from body postures, facial expressions and EEG signal

Babu, Ashwin Ramesh; Rajavenkatanarayanan, Akilesh; Brady, James; Makedon, Fillia

doi:10.1145/3279810.3279849

Cited by 20 publications

(7 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From the analysis, each subject mental conditions are predicted with upto 99% accuracy. In addition to this, the accuracy of another classifier called SVM efficiency is examined and result [6] is demonstrated in Table 3.…”

Section: Results and Interpretationmentioning

confidence: 99%

Brain–computer interface for assessment of mental efforts in e‐learning using the nonmarkovian queueing model

Balamurugan

Mullai

Soundararajan³

et al. 2020

Comp Applic In Engineering

View full text Add to dashboard Cite

The rapid advancement in information and communication technology has made e‐learning an alternative learning method for many learners. In the last few years, a huge number of learners around the world have registered in massive open online courses (MOOCs) provided by various online learning platforms. However, MOOC platforms have a vital task for the online course provider to provide enhanced students' learning experiences and satisfaction. In this work, we developed a brain–computer interface for gathering data and detecting a learner's mental situation by observing MOOC videos and electroencephalogram (EEG) devices based on John Sweller's Cognitive Load Theory. The acquired EEG signals are preprocessed with two different normalization methods to scale signals. To validate the introduced framework, the system adopted three machine learning algorithms (random forest using non‐Markovian model, support vector machine, and k‐nearest neighbors) to develop a model with preprocessed training data and test the classifiers to validate their ensemble classifiers' performance. Finally, experimental analysis showed that the random forest classifier with the non‐Markovian approach achieved more than the other two techniques in the form of overall accuracy (99.15%) and F‐measures (99.21%).

show abstract

Section: Results and Interpretationmentioning

confidence: 99%

Brain–computer interface for assessment of mental efforts in e‐learning using the nonmarkovian queueing model

Balamurugan

Mullai

Soundararajan³

et al. 2020

Comp Applic In Engineering

View full text Add to dashboard Cite

show abstract

“…They used the HOG model for face detection; then, facial features were extracted and classified with a deep CNN. Furthermore, Babu et al [ 29 ] presented a multi-modal algorithm for NAO robots that consists of obtaining face and body expressions to determine the user’s emotional state when interacting with a robot: (a) the Adam Optimizer (AO) algorithm detects and extracts the facial features (eyes, eyebrows, and mouth); (b) the body’s position is extracted utilizing an RGB-D camera; (c) combining the face and body information into a CNN, expressions can be classified as positive, neutral, or negative emotional states, allowing the robot to behave appropriately.…”

Section: Algorithms Used For Face Recognition and Trackingmentioning

confidence: 99%

Artificial Vision Algorithms for Socially Assistive Robot Applications: A Review of the Literature

Montaño-Serrano

Jacinto‐Villegas

Vilchis-González

et al. 2021

Sensors

View full text Add to dashboard Cite

Today, computer vision algorithms are very important for different fields and applications, such as closed-circuit television security, health status monitoring, and recognizing a specific person or object and robotics. Regarding this topic, the present paper deals with a recent review of the literature on computer vision algorithms (recognition and tracking of faces, bodies, and objects) oriented towards socially assistive robot applications. The performance, frames per second (FPS) processing speed, and hardware implemented to run the algorithms are highlighted by comparing the available solutions. Moreover, this paper provides general information for researchers interested in knowing which vision algorithms are available, enabling them to select the one that is most suitable to include in their robotic system applications.

show abstract

“…In contrast to most of the referenced works, we suggest a machine learning-based analysis of EEG signals towards identifying CF. Our method comes as an extension of our previous findings, originally presented in [23,24], where similar modeling solutions were deployed to predict cognitive performance on a short-term memory cognitive task. Long term scope of this exploratory research is to develop advanced computational methods for fatigue prediction and modeling able to enhance the efficiency of current approaches in assistive technologies related to medical conditions such as MS [25] or workplace training [26].…”

Section: Background-computational Modeling Of Cognitive Fatiguementioning

confidence: 99%

CogBeacon: A Multi-Modal Dataset and Data-Collection Platform for Modeling Cognitive Fatigue

2019

Self Cite

View full text Add to dashboard Cite

In this work, we present CogBeacon, a multi-modal dataset designed to target the effects of cognitive fatigue in human performance. The dataset consists of 76 sessions collected from 19 male and female users performing different versions of a cognitive task inspired by the principles of the Wisconsin Card Sorting Test (WCST), a popular cognitive test in experimental and clinical psychology designed to assess cognitive flexibility, reasoning, and specific aspects of cognitive functioning. During each session, we record and fully annotate user EEG functionality, facial keypoints, real-time self-reports on cognitive fatigue, as well as detailed information of the performance metrics achieved during the cognitive task (success rate, response time, number of errors, etc.). Along with the dataset we provide free access to the CogBeacon data-collection software to provide a standardized mechanism to the community for collecting and annotating physiological and behavioral data for cognitive fatigue analysis. Our goal is to provide other researchers with the tools to expand or modify the functionalities of the CogBeacon data-collection framework in a hardware-independent way. As a proof of concept we show some preliminary machine learning-based experiments on cognitive fatigue detection using the EEG information and the subjective user reports as ground truth. Our experiments highlight the meaningfulness of the current dataset, and encourage our efforts towards expanding the CogBeacon platform. To our knowledge, this is the first multi-modal dataset specifically designed to assess cognitive fatigue and the only free software available to allow experiment reproducibility for multi-modal cognitive fatigue analysis.

show abstract

Multimodal approach for cognitive task performance prediction from body postures, facial expressions and EEG signal

Cited by 20 publications

References 28 publications

Brain–computer interface for assessment of mental efforts in e‐learning using the nonmarkovian queueing model

Brain–computer interface for assessment of mental efforts in e‐learning using the nonmarkovian queueing model

Artificial Vision Algorithms for Socially Assistive Robot Applications: A Review of the Literature

CogBeacon: A Multi-Modal Dataset and Data-Collection Platform for Modeling Cognitive Fatigue

Contact Info

Product

Resources

About