Egocentric video: a new tool for capturing hand use of individuals with spinal cord injury at home

Abstract

Background Current upper extremity outcome measures for persons with cervical spinal cord injury (cSCI) lack the ability to directly collect quantitative information in home and community environments. A wearable first-person (egocentric) camera system is presented that aims to monitor functional hand use outside of clinical settings. Methods The system is based on computer vision algorithms that detect the hand, segment the hand outline, distinguish the user’s left or right hand, and detect functional interactions of the hand with objects during activities of daily living. The algorithm was evaluated using egocentric video recordings from 9 participants with cSCI, obtained in a home simulation laboratory. The system produces a binary hand-object interaction decision for each video frame, based on features reflecting motion cues of the hand, hand shape and colour characteristics of the scene. Results The output from the algorithm was compared with a manual labelling of the video, yielding F1-scores of 0.74 ± 0.15 for the left hand and 0.73 ± 0.15 for the right hand. From the resulting frame-by-frame binary data, functional hand use measures were extracted: the amount of total interaction as a percentage of testing time, the average duration of interactions in seconds, and the number of interactions per hour. Moderate and significant correlations were found when comparing these output measures to the results of the manual labelling, with ρ = 0.40, 0.54 and 0.55 respectively. Conclusions These results demonstrate the potential of a wearable egocentric camera for capturing quantitative measures of hand use at home.

Analysis of the Hands in Egocentric Vision: A Survey

“…Lastly, pixel-level segmentation was achieved by backprojecting using an adaptively selected region in the colour space. In [30], the coarse segmentation obtained with a mixture of Gaussian skin model [20], [29] was refined by using a structured forest edge detection [31], specifically trained on available datasets [19], [32].…”

“…CaffeNet [75] was used for classifying the proposals. Faster R-CNN was used in [30], [77], [78]. In particular, Likitlersuang et al [30] fine-tuned the network on videos from individuals with cSCI performing ADLs.…”

“…Faster R-CNN was used in [30], [77], [78]. In particular, Likitlersuang et al [30] fine-tuned the network on videos from individuals with cSCI performing ADLs. False positives were removed based on the arm angle information computed by applying a Haar-like feature rotated 360 degrees around the bounding box centroid.…”

“…Hand identification is the process of disambiguating the left and right hands of the camera wearer, as well as the hands of other persons in the scene. The egocentric POV has intrinsic advantages that allow discriminating the hands by using simple spatial and geometrical constraints [30], [86], [95]. Usually, the user's hands appear in the lower part of the image, with the right hand to the right of the user's left hand, and vice versa.…”

“…By contrast, other people's hands tend to appear in the upper part of the frame [86]. The orientation of the arm regions was used in [30], [95] to distinguish the left from the right user's hand. To estimate the angle, the authors rotated a Haar-like feature around the segmented hand region, making this approach robust to the presence of sleeves and different skin colors, since it did not require any calibrations [95].…”