In-Bed Pose Estimation: Deep Learning With Shallow Dataset

Abstract

This paper presents a robust human posture and body parts detection method under a specific application scenario known as in-bed pose estimation. Although the human pose estimation for various computer vision (CV) applications has been studied extensively in the last few decades, the in-bed pose estimation using camera-based vision methods has been ignored by the CV community because it is assumed to be identical to the general purpose pose estimation problems. However, the in-bed pose estimation has its own specialized aspects and comes with specific challenges, including the notable differences in lighting conditions throughout the day and having pose distribution different from the common human surveillance viewpoint. In this paper, we demonstrate that these challenges significantly reduce the effectiveness of the existing general purpose pose estimation models. In order to address the lighting variation challenge, the infrared selective (IRS) image acquisition technique is proposed to provide uniform quality data under various lighting conditions. In addition, to deal with the unconventional pose perspective, a 2- end histogram of oriented gradient (HOG) rectification method is presented. The deep learning framework proves to be the most effective model in human pose estimation; however, the lack of large public dataset for in-bed poses prevents us from using a large network from scratch. In this paper, we explored the idea of employing a pre-trained convolutional neural network (CNN) model trained on large public datasets of general human poses and fine-tuning the model using our own shallow (limited in size and different in perspective and color) in-bed IRS dataset. We developed an IRS imaging system and collected IRS image data from several realistic life-size mannequins in a simulated hospital room environment. A pre-trained CNN called convolutional pose machine (CPM) was fine-tuned for in-bed pose estimation by re-training its specific intermediate layers. Using the HOG rectification method, the pose estimation performance of CPM improved significantly by 26.4% in the probability of correct key-point (PCK) criteria at PCK0.1 compared to the model without such rectification. Even testing with only well aligned in-bed pose images, our fine-tuned model still surpassed the traditionally tuned CNN by another 16.6% increase in pose estimation accuracy.

A Semi-supervised Data Augmentation Approach Using 3D Graphical Engines

“…When dealing with deep learning in small data domains, fine-tuning already trained DNNs proves to be effective [25,7,8,10,40]. Fine-tuning is a form of transfer learning, when fine-tuned DNNs applied on the new (but small in size) dataset hugely benefit from knowledge learned from large amount of real world image samples (even being from different domain).…”

“…However, each image in the training set needs to be labeled for the supervised learning process and it get quite expensive for large datasets [38]. On the other hand, for certain applications where data is scarce such as personalized medicine, robot reinforcement learning, environmental/weather behavior prediction, and military applications, forming a large scale dataset itself could be infeasible [25]. The million dollar question here is if one can benefit from the flexibility and accuracy of DNNs in small data domains or domains with expensive labeling process by virtually synthesizing large scale labeled datasets.…”

A Data Augmentation Method for Deep Learning Based on Multi-Degree of Freedom (DOF) Automatic Image Acquisition

“…One method is to install camera equipment in the detection environment, and multi-attitude datasets can be obtained with the movement of the structure to be tested. Liu et al [8] collect body posture by installing an indoor camera to acquire the different postures of people in the process of daily activities. Berriel et al [9,10] obtained a road dataset by vehicle cameras.…”

Automated Non-Contact Detection of Head and Body Positions During Sleep

“…However, regular cameras cannot be applied in low light sleep environment. To address this issue, a more recent solution by Liu et al used an infrared camera to capture the videos [17]. A pre-trained convolutional neural network (CNN) -namely the convolutional pose machine (CPM) [18] -was then repurposed for sleeping position detection via fine-tuning.…”