Understanding Egocentric Hand-Object Interactions from Hand Pose Estimation

Abstract

In this paper, we address the problem of estimating the hand pose from the egocentric view when the hand is interacting with objects. Specifically, we propose a method to label a dataset Ego-Siam which contains the egocentric images pair-wisely. We also use the collected pairwise data to train our encoder-decoder style network which has been proven efficient in [31]. This could bring extra training efficiency and testing accuracy. Our network is light weight and can be performed with over 30 FPS with an outdated GPU. We demonstrate that our method outperforms Mueller et al. [18] which is the state of the art work dealing with egocentric hand-object interaction problem on the GANerated [19] dataset. To show the ability to preserve the semantic information of our method, we also report the performance of grasp type classification on dataset and outperforms the benchmark with only using predicted 3-d hand pose.

An Outlook into the Future of Egocentric Vision

“…It consists of 100K frames acquired over 131 days in which humans were involved in 11 categories of interactions labeled with bounding boxes around the hands and the active object, hand side and hand contact state (indicating if there is a contact between the hand and an object or not). Lu and Mayol-Cuevas (2021) introduced a dataset to study hand poses during manipulation with objects. The dataset has been captured in a multi-cam setting using two HD cameras and an iPhone 12.…”

E$^2$(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition

“…Egocentric vision has introduced a variety of new challenges to the computer vision community, such as humanobject interaction [18,65], action anticipation [1,30,39,64], action recognition [52], and video summarization [23,57,58]. With the advent of novel large-scale datasets [14,15], new tasks are being proposed, such as wearer's pose estimation [105] and egocentric videos anonymization [95].…”