Self-supervised Learning for Reading Activity Classification

Abstract

Reading analysis can relay information about user's confidence and habits and can be used to construct useful feedback. A lack of labeled data inhibits the effective application of fully-supervised Deep Learning (DL) for automatic reading analysis. We propose a Self-supervised Learning (SSL) method for reading analysis. Previously, SSL has been effective in physical human activity recognition (HAR) tasks, but it has not been applied to cognitive HAR tasks like reading. We first evaluate the proposed method on a four-class classification task on reading detection using electrooculography datasets, followed by an evaluation of a two-class classification task of confidence estimation on multiple-choice questions using eye-tracking datasets. Fully-supervised DL and support vector machines (SVMs) are used as comparisons for the proposed SSL method. The results show that the proposed SSL method is superior to the fully-supervised DL and SVM for both tasks, especially when training data is scarce. This result indicates the proposed method is the superior choice for reading analysis tasks. These results are important for informing the design of automatic reading analysis platforms.

Exploring Sensor Modalities to Capture User Behaviors for Reading Detection

“…We adapt and reproduce the SSL method proposed in [11] that consists of pre-training and target task training, as shown in Fig. 2.…”

“…2, we use the unlabeled data from EOG, ACC, and GYRO sensors and formulate the pretext task using data augmentation by applying signal transformations. We use eight signal transformations; noise addition, scale, vertical flip, horizontal flip, permutation, time-warp, channelshuffle, and rotation [11] for ACC and GYRO data and the first seven for EOG data. We format the pretext task as transformation prediction.…”

“…A baseline study [4] used data from three sensors for reading detection. The combination of three sensors has also been employed by Islam et al [11] for fine-grained reading detection with advanced algorithms. Ishimaru et al [2], [5] also employed the EOG and ACC sensors data for reading/not reading detection and also reading detection with other activities.…”

“…We used two datasets, labeled and unlabeled, collected outside of the laboratory (in-the-wild) that were first reported by Ishimaru et al [4] and also used by Islam et al [11]. These are time-series datasets recorded using the J!NS MEME glasses and contain data for EOG, ACC, and GYRO sensors.…”

Stepping Into the Next Decade of Ubiquitous and Pervasive Computing: UbiComp and ISWC 2021

“…Md. Rabiul Islam et al 35 adapted self-supervised learning methods for cognitive HAR tasks like reading, illustrating how self-supervised learning can accelerate progress toward real-world implementation. Lago et al 36 proposed a method for improving single-sensorbased activity recognition by leveraging data from multiple sensors during training time.…”