First-person reading activity recognition by deep learning with synthetically generated images

Abstract

We propose a vision-based method for recognizing first-person reading activity with deep learning. For the success of deep learning, it is well known that a large amount of training data plays a vital role. Unlike image classification, there are less publicly available datasets for reading activity recognition, and the collection of book images might cause copyright trouble. In this paper, we develop a synthetic approach for generating positive training images. Our approach synthesizes computer-generated images and real backround images. In experiments, we show that this synthesis is effective in combination with pre-trained deep convolutional neural networks and also our trained neural network outperforms other baselines.

Improving Robot Perception Skills Using a Fast Image-Labelling Method with Minimal Human Intervention

“…Moreover, several authors used synthesized images to train deep learning model successfully [24][25][26][27][28]. This fact demonstrates that image synthetization is a technique that reduces the effort of manual annotations.…”

“…Castro et al [27] generated synthetic structural magnetic resonance images for learning schizophrenia. Segawa et al [28] recognized first-person reading activity by synthetizing computer-generated images and real background images.…”

Exploring built environment correlates of older adults’ walking travel from lifelogging images

“…Recent advancement in automatic reviewing using machine vision such as deep learning and convolutional neural networks (CNN) have allowed the processing of large data sets to identify various features in images (Nweke, Teh, Al-Garadi, & Alo, 2018). However, processing of large dataset of egocentric/first-person images are still at their early stage of development (Segawa, Kawamoto, & Okamoto, 2018). Castro et al (2015) showed the prediction of daily activities from egocentric images using CNN.…”