Hand-Object Interaction Detection with Fully Convolutional Networks

Schröder, M.; Ritter, Helge

doi:10.1109/cvprw.2017.163

Cited by 15 publications

(5 citation statements)

References 22 publications

(39 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Likitlersuang et al [9] put attention on higher level features, they build a hand-object interaction classifier with the optical flow and hand shape as input. The work from Schroder et al [10] use hand and object mask to predict the hand-object interaction status. The idea of using masks inspires our work; Differently, we explore the relationship between the hand-object pair and interaction with hand pose and masks.…”

Section: Related Work a Hoi Detectionmentioning

confidence: 99%

Egocentric Hand-object Interaction Detection

Lu¹,

Liu²

2022

Preprint

View full text Add to dashboard Cite

In this paper, we propose a method to jointly determine the status of hand-object interaction. This is crucial for egocentric human activity understanding and interaction. From a computer vision perspective, we believe that determining whether a hand is interacting with an object depends on whether there is an interactive hand pose and whether the hand is touching the object. Thus, we extract the hand pose, handobject masks to jointly determine the interaction status. In order to solve the problem of hand pose estimation due to inhand object occlusion, we use a multi-cam system to capture hand pose data from multiple perspectives. We evaluate and compare our method with the most recent work from Shan et al.[1] on selected images from EPIC-KITCHENS [2] dataset and achieve 89% accuracy on HOI (hand-object interaction) detection which is comparative to Shan's (92%). However, for real-time performance, our method can run over 30 FPS which is much more efficient than Shan's (1 ∼ 2 FPS). A demo can be found from https://www.youtube.com/watch?v=XVj3zBuynmQ.

show abstract

Section: Related Work a Hoi Detectionmentioning

confidence: 99%

Egocentric Hand-object Interaction Detection

Lu¹,

Liu²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…There are some topics in the computer-vision community that can be considered as related to the unique challenges of open surgery, such as Hand-Object Interaction [39], Human-Object Interaction [26] and Object-Object Interaction [21]. In [21] the authors propose inter-object graph representation for recognition of activities in self-driving scenarios.…”

Section: Related Workmentioning

confidence: 99%

Video-based fully automatic assessment of open surgery suturing skills

Goldbraikh¹,

D’Angelo²,

Pugh³

et al. 2021

Preprint

View full text Add to dashboard Cite

Purpose: The goal of this study was to develop new reliable open surgery suturing simulation system for training medical students in situation where resources are limited or in the domestic setup. Namely, we developed an algorithm for tools and hands localization as well as identifying the interactions between them based on simple webcam video data, calculating motion metrics for assessment of surgical skill Methods: Twenty-five participants performed multiple suturing tasks using our simulator.The YOLO network has been modified to a multi-task network, for the purpose of tool localization and tool-hand interaction detection. This was accomplished by splitting the YOLO detection heads so that they supported both tasks with minimal addition to computer run-time. Furthermore, based on the outcome of the system, motion metrics were calculated. These metrics included traditional metrics such as time and path length as well as new metrics assessing the technique participants use for holding the tools. Results: The dual-task network performance was similar to that of two networks, while computational load was only slightly bigger than one network. In addition, the motion metrics showed significant differences between experts and novices. Conclusion: While video capture is an essential part of minimal invasive surgery, it is not an integral component of open surgery. Thus, new algorithms, focusing on the unique challenges open surgery videos present, are required. In this study a dual-task network was developed to solve both a localization task and a hand-tool interaction task. The dual network may be easily expanded to a multi-task network, which may be useful for images with multiple layers and for evaluating the interaction between these different layers.

show abstract

“…Likitlersuang et al [12] build a hand-object interaction classifier with the optical flow and hand shape as input. The work from Schroder et al [23] use hand and object mask to predict the hand-object interaction status. The idea of using masks inspires our work; simultaneously, we explore the relationship between the hand-object pair and interaction with hand pose and masks.…”

Section: Hoi Detectionmentioning

confidence: 99%

Egocentric Hand-object Interaction Detection and Application

Lu¹,

Mayol-Cuevas²

2021

Preprint

View full text Add to dashboard Cite

In this paper, we present a method to detect the handobject interaction from egocentric perspective. In contrast to massive data driven discriminator based method like [24], we propose a novel workflow that utilise the cues of hand and object. Specifically, we train networks predicting hand pose, hand mask and in-hand object mask to jointly predict the hand-object interaction status. We compare our method with the most recent work from Shan et al. [24] on selected images from EPIC-KITCHENS [4] dataset and achieve 89% accuracy on HOI (hand-object interaction) detection which is comparative to Shan's (92%). However, for real-time performance, with the same machine, our method can run over 30 FPS which is much efficient than Shan's (1 ∼ 2 FPS). Furthermore, with our approach, we are able to segment script-less activities from where we extract the frames with the HOI status detection. We achieve 68.2% and 82.8% F1 score on GTEA [7] and the UTGrasp [1] dataset respectively which are all comparative to the SOTA methods.

show abstract

Hand-Object Interaction Detection with Fully Convolutional Networks

Cited by 15 publications

References 22 publications

Egocentric Hand-object Interaction Detection

Egocentric Hand-object Interaction Detection

Video-based fully automatic assessment of open surgery suturing skills

Egocentric Hand-object Interaction Detection and Application

Contact Info

Product

Resources

About