A Facial and Vocal Expression Based Comprehensive Framework for Real-Time Student Stress Monitoring in an IoT-Fog-Cloud Environment

Abstract

In this era of digital and modern education, the existence of psychological stress on students cannot be denied. The surplus aggregation of the stress may lead to different problems like a decline in student grade (performance), an increase of violence in behavior, and even more extreme cases. The advent of Information Communication and Technology (ICT) and its tools opened the doors to innovations that facilitate interactions among things and humans. In this utilization, the paper proposes a novel, IoT-aware student-centric stress monitoring and real-time alert generating framework to predict student stress index in a particular context. In elaboration, we respectively used extended VGG16, Bidirectional Long Short Term Memory network (Bi -LSTM), and Multinomial Naïve Bayes techniques to generate the scores of emotions from student facial expressions, speech pitch, and content of student speech at the cloud layer. Specifically, the model aims to classify the stress events as normal or abnormal on basis of the overall emotion of the students' physiological data readings. The activation of the abnormal event in case of higher values for negative emotions like stress, fear, sadness, disgust, etc.; a stern alert is sent to the student, coordinators, and caretakers. This proposed framework will ultimately be a great tool that will support the education institutions, students, their parents, and guardians to get a real-time alert on students' overall emotions. The prior knowledge of stress accumulated on the mind of the student will help in overcoming major problems of student dropout, decrease student academic performance, and tackle the stress situation that may lead to the student attempting suicide.

Research on cloud-native monitoring system based on Prometheus

“…Alert management plays a crucial role in a monitoring system [11], as it identifies and notifies about anomalies in the system in a timely and effective manner, ensuring smooth business operations. In daily production, it is important to avoid a large number of false alarms caused by system-level failures while ensuring that genuine root cause alerts are not missed.…”

At the Confluence of Artificial Intelligence and Edge Computing in IoT-Based Applications: A Review and New Perspectives

“…The authors in [ 65 ] proposed a framework for monitoring student stress and generating real-time alerts to predict student stress. The authors used Visual Geometry Group (VGG16) for facial expression, bi-LSTM for speech texture analysis, and multinomial NB techniques to generate emotion scores and classify stress events as normal or abnormal.…”

Intelligent speech technologies for transcription, disease diagnosis, and medical equipment interactive control in smart hospitals: A review

“…13 (b). In addition, they proposed adding an IoT layer as a data source [ 86 ] so that guardians could obtain a real-time alert on students’ overall emotions in response to their stressful situations.

Fig.

…”

“…(Others) Disease Data sources Voice type Voice feature Classifier Effect Ref. Juvenile Idiopathic Arthritis 5 HP, 3 Ps Knee Acoustical Spectral, MFCC, or band power feature Gradient Boosted Trees, neural network Accuracy = 92.3% using GBT, Accuracy = 72.9% using neural network [ 150 ] Stress 6 categories of emotions, namely: Surprise, Fear, Neutral, Anger, Sad, and Happy SS (facial expressions, content of speech) Mel scaled spectrogram Multinomial Naïve Bayes, Bi-LSTM, CNN Assess students' stress by facial expressions and speech is effective [ 86 ] Depression and Other Psychiatric Conditions Gruop1: depression (DP) 27 S; Gruop2: other psychiatric conditions (OP) 12 S; Gruop3: normal controls (NC) 27 S SS Features extracted by openSMILE and Weka program [ 151 ] Five multiclass classifier schemes of scikit-learn Accuracy = 83.33%, sensitivity = 83.33%, and specificity = 91.67% [ 152 ] Depression AVEC 2014 dataset: 84 S; TIMIT dataset SS TEO-CB-Auto-Env, Cepstral, Prosodic, Spectral, and Glottal, MFCC Cosine similarity Accuracy = 90% [ 154 ] SV=Sustained vowel, SS=Spontaneous speech, Ps = Patients, HP=Healthy People, S=Subjects. …”