Measuring and Integrating Facial Expressions and Head Pose as Indicators of Engagement and Affect in Tutoring Systems

Yu, Haiming; Gupta, Ankit; Lee, William; Arroyo, Ivon; Betke, Margrit; Allesio, Danielle; Murray, Tom; Magee, John J.; Woolf, Beverly Park

doi:10.1007/978-3-030-77873-6_16

Cited by 10 publications

(3 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Empathetic characters that provide interventions generate superior results both to improve student interactions with the system, address negative student emotions, and in the overall learning experience [47]. Predicting outcomes of problems for students is a valuable source of information for planning and executing ITS interventions for improving learning [48], [49]. For example the ITS could provide hints when the system predicts that the student will not be able to successfully complete the problem.…”

Section: Related Workmentioning

confidence: 99%

ATL-BP: A Student Engagement Dataset and Model for Affect Transfer Learning for Behavior Prediction

Ruiz

Allessio

et al. 2023

IEEE Trans. Biom. Behav. Identity Sci.

Self Cite

View full text Add to dashboard Cite

We propose a video-based transfer learning approach for predicting problem outcomes of students working with an intelligent tutoring system (ITS) by analyzing their faces and gestures. The ability to predict such outcomes enables tutoring systems to adjust interventions and ultimately yield improved student learning. We collected and released a labeled dataset of 2,749 problem-solving interaction samples of 54 students working with an intelligent online math tutor. Our transfer-learning challenge was then to design a representation in the source domain of images obtained from the Internet for facial expression analysis, and transfer this learned representation for human behavior prediction in the domain of webcam videos of students in a classroom environment. We developed a novel facial affect representation and a user-personalized training scheme that unlocks the potential of this representation. We designed several variants of a recurrent neural network that models the temporal structure of video sequences. Our final model, named ATL-BP for Affect Transfer Learning for Behavior Prediction, achieves a relative increase in the mean F-score of 50% over the state-of-the-art method on this new dataset. We also propose an additional set of annotations to predict students' engagement while solving a specific problem, and present models that can predict such engagement.

show abstract

Section: Related Workmentioning

confidence: 99%

ATL-BP: A Student Engagement Dataset and Model for Affect Transfer Learning for Behavior Prediction

Ruiz

Allessio

et al. 2023

IEEE Trans. Biom. Behav. Identity Sci.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In addition, an online report and dashboard were also suggested in [46] that offer instructors an assessment of students' involvement and emotional states while using an online tutoring system. Likewise, a head pose app that predicts students' realtime head direction and reacts to potential disengagement was also proposed in [128]. The app also includes a facial expression dashboard that detects students' emotional states and delivers this information to instructors to help instructors assess student development and identify students who need more help.…”

Section: Facial Expression Detection and Recognition Appsmentioning

confidence: 99%

Front-end deep learning web apps development and deployment: a review

Goh

Abas

2022

Appl Intell

View full text Add to dashboard Cite

Machine learning and deep learning models are commonly developed using programming languages such as Python, C++, or R and deployed as web apps delivered from a back-end server or as mobile apps installed from an app store. However, recently front-end technologies and JavaScript libraries, such as TensorFlow.js, have been introduced to make machine learning more accessible to researchers and end-users. Using JavaScript, TensorFlow.js can define, train, and run new or existing, pre-trained machine learning models entirely in the browser from the client-side, which improves the user experience through interaction while preserving privacy. Deep learning models deployed on front-end browsers must be small, have fast inference, and ideally be interactive in real-time. Therefore, the emphasis on development and deployment is different. This paper aims to review the development and deployment of these deep-learning web apps to raise awareness of the recent advancements and encourage more researchers to take advantage of this technology for their own work. First, the rationale behind the deployment stack (front-end, JavaScript, and TensorFlow.js) is discussed. Then, the development approach for obtaining deep learning models that are optimized and suitable for front-end deployment is then described. The article also provides current web applications divided into seven categories to show deep learning potential on the front end. These include web apps for deep learning playground, pose detection and gesture tracking, music and art creation, expression detection and facial recognition, video segmentation, image and signal analysis, healthcare diagnosis, recognition, and identification.

show abstract

“…At present, the commonly used emotion recognition methods in online learning include: emotion recognition based on facial expression, speech, and physiological signals such as EEG, EDA and skin temperature [5][6][7][8]. However, facial expression and voice to a certain extent, easily influenced and controlled by people's subjective consciousness, and can be artificially camouflage and concealing the internal psychological state, the emotional state identified by analyzing these psychological signals may lead to inconsistencies with the true emotional state, so it is affecting the reliability of this method in practical application.…”

Section: Introductionmentioning

confidence: 99%

Emotion recognition research of eye-movement feature extraction classification network in online video learning environment

Liu¹,

Li²,

Tao³

2023

Third International Seminar on Artificial Intelligence, Networking, and Information Technology (AINIT 2022)

View full text Add to dashboard Cite

With the rapid development of artificial intelligence technology, emotion recognition has been applied in all aspects of life, using eye movement tracking technology for emotion recognition has become an important branch of emotion computing. In order to explore the relationship between eye movement signals and learners' emotional states in the online video learning environment, we used machine learning and convolutional neural network methods to recognize eye movement signals, and classify learners' emotional states into two categories, positive and negative. The study of eye movement data under different time windows mainly includes four stages: data collection, data preprocessing, classifier modeling, training and testing. In this paper, a Eye-movement Feature Extraction Classification Network(EFECN) based on convolutional neural network is proposed for small sample data and the classification of emotion state based on eye movement. The eye movement data were transformed into images through cross-modal conversion as input of multiple different deep convolutional neural networks, and the emotional states were classified in positive and negative directions. The accuracy was used as the evaluation index to evaluate and compare the different models. The accuracy of the eye movement emotion recognition model reached 72% in the SVM model and 91.62% in the EFECN model. Experimental results show that the convolutional neural network based on deep learning has a significant improvement in recognition accuracy compared with traditional machine learning methods.

show abstract

Measuring and Integrating Facial Expressions and Head Pose as Indicators of Engagement and Affect in Tutoring Systems

Cited by 10 publications

References 35 publications

ATL-BP: A Student Engagement Dataset and Model for Affect Transfer Learning for Behavior Prediction

ATL-BP: A Student Engagement Dataset and Model for Affect Transfer Learning for Behavior Prediction

Front-end deep learning web apps development and deployment: a review

Emotion recognition research of eye-movement feature extraction classification network in online video learning environment

Contact Info

Product

Resources

About