Automatic Detection of Reflective Thinking in Mathematical Problem Solving Based on Unconstrained Bodily Exploration

Abstract

For technology (like serious games) that aims to deliver interactive learning, it is important to address relevant mental experiences such as reflective thinking during problem solving. To facilitate research in this direction, we present the weDraw-1 Movement Dataset of body movement sensor data and reflective thinking labels for 26 children solving mathematical problems in unconstrained settings where the body (full or parts) was required to explore these problems. Further, we provide qualitative analysis of behaviours that observers used in identifying reflective thinking moments in these sessions. The body movement cues from our compilation informed features that lead to average F1 score of 0.73 for automatic detection of reflective thinking based on Long Short-Term Memory neural networks. We further obtained 0.79 average F1 score for end-toend detection of reflective thinking periods, i.e. based on raw sensor data. Finally, the algorithms resulted in 0.64 average F1 score for period subsegments as short as 4 seconds. Overall, our results show the possibility of detecting reflective thinking moments from body movement behaviours of a child exploring mathematical concepts bodily, such as within serious game play.

A Movement in Multiple Time Neural Network for Automatic Detection of Pain Behaviour

“…The window-level label was set as negative if all the frames in the window were negative for guarding, positive if all frames in the window were positive, and mixed otherwise. To manage class imbalance in the training and validation sets, we used data augmentation methods similar to [9] to randomly oversample minority classes. This resulted in 17,185 and 1,394 instances respectively.…”

Modeling Multiple Temporal Scales of Full-Body Movements for Emotion Classification

“…Given a data chunk D represented by the X, Y and Z coordinates of the 30 markers, we obtain a new chunk by: L2RR2L assumes that the person can display the same expressive quality by moving left and right part of his/her body. Strategies similar to 3D-RotX, 3D-RotY, and 3D-RotZ have been recently used in [41] for automatic detection of reflective thinking. Herein, we have adapted them for emotion classification, and they are based on the assumption that emotion recognition should be invariant to 3D-rotations.…”

“…We used radial basis function (RBF) kernel when the penalty parameter C of the error term is ranging from 0.001 to 10000, and γ kernel coefficient is ranging from 0.001 to 1000. Bi-LSTMs and SVM-RBF have been frequently applied to process MoCap data of nonverbal behaviors in various contexts including emotion classification [26], [14], [41], [25], therefore, we included them to the comparisons.…”

Leveraging Activity Recognition to Enable Protective Behavior Detection in Continuous Data

“…Automatic detection of continuous affective behavior across different daily activities is still rare. For example, [46] explored the detection of bodily expressions of reflective thinking in the context of diverse full-body mathematical games. While this study developed activity-independent models over continuous data sequences, their proposed LSTM-based architecture needs to be trained on pre-segmented affective events (e.g.…”