Egocentric recognition of handled objects: Benchmark and analysis

Abstract

Recognizing objects being manipulated in hands can provide essential information about a person's activities and have far-reaching impacts on the application of visionin everyday life. The egocentric viewpoint from a wearable camera has unique advantages in recognizing handled objects, such as having a close view and seeing objects in their natural positions. We collect a comprehensive dataset and analyze the feasibilities and challenges of the egocentric recognition of handled objects.We use a lapel-worn camera and record uncompressed video streams as human subjects manipulate objects in daily activities. We use 42 day-to-day objects that vary in size, shape, color and textureness. 10 video sequences are shot for each object under different illuminations and backgrounds. We use this dataset and a SIFT-based recognition system to analyze and quantitatively characterize the main challenges in egocentric object recognition, such as motion blur and hand occlusion, along with its unique constraints, such as hand color, location prior and temporal consistency. SIFT-based recognition has an average recognition rate of 12%, and reaches 20% through enforcing temporal consistency. We use simulations to estimate the upper bound for SIFT-based recognition at 64%, the loss of accuracy due to background clutter at 20%, and that of hand occlusion at 13%. Our quantitative evaluations show that the egocentric recognition of handled objects is a challenging but feasible problem with many unique characteristics and many opportunities for future research.

Fusion of Multiple Visual Cues for Visual Saliency Extraction from Wearable Camera Settings with Strong Motion

“…Nowadays, the attention of computer vision community is more and more turned to the new forms of video content: such as wearable video cameras, or "egocentric" view of the world [8,9]. Some attempts to identify visual saliency mainly on the basis of the frequency of repetition of visual objects and regions in the wearable video content have recently been made in [10]. We are specifically interested in building visual saliency maps by fusion of all cues in the pixel domain for the case of "egocentric" video content recorded with wearable cameras.…”

A review on vision techniques applied to Human Behaviour Analysis for Ambient-Assisted Living

“…Papers reviewed using such techniques [93,111,112] emphasize how wearable cameras avoid the body to occlude what is being managed with the hands (see Fig. 7).…”

An Introduction to the 3rd Workshop on Egocentric (First-Person) Vision

“…The first workshop organized by Philipose, Hebert and Ren, featured topics such as object analysis [63,72], activity analysis [74,68] and scene understanding [25,16,20]. The first workshop helped to bring together computer vision researchers to develop more advanced component-level technologies and to understand the challenges of working with egocentric vision.…”

“…To date researchers have explored the use of egocentric vision for activity recognition [60,11,10], object recognition [63,14], summarization [31,35], temporal segmentation [68,26,61], scene understanding [65], interaction analysis [13,66], hand detection [33,32], gaze estimation [76], gaze analysis [54,80,34,10], visual saliency [78,79], social saliency [57] and motion capture [67].…”