Categorizing object-action relations from semantic scene graphs

Abstract

Abstract-In this work we introduce a novel approach for detecting spatiotemporal object-action relations, leading to both, action recognition and object categorization. Semantic scene graphs are extracted from image sequences and used to find the characteristic main graphs of the action sequence via an exact graph-matching technique, thus providing an event table of the action scene, which allows extracting objectaction relations. The method is applied to several artificial and real action scenes containing limited context. The central novelty of this approach is that it is model free and needs a priori representation neither for objects nor actions. Essentially actions are recognized without requiring prior object knowledge and objects are categorized solely based on their exhibited role within an action sequence. Thus, this approach is grounded in the affordance principle, which has recently attracted much attention in robotics and provides a way forward for trial and error learning of object-action relations through repeated experimentation. It may therefore be useful for recognition and categorization tasks for example in imitation learning in developmental and cognitive robotics.

Hallucinated Humans as the Hidden Context for Labeling 3D Scenes

“…Moreover, from the point of view of modeling and learning, this explanation is parsimonious and efficient as compared to modeling the object-object relationships [19] such as chair-keyboard, table-monitor, monitor-keyboard, etc. 1 …”

“…In most previous works, object detection and activity recognition have been addressed as separate tasks. Only recently, some works [9,32,1,25,20] have shown that modeling the interaction between human poses and objects in 2D images and videos result in a better performance on the tasks of object detection and activity recognition. In contemporary work, Fouhey et al [6] and Delaitre et al [4] observe humans in videos for estimating 3D geometry and estimating affordances respectively.…”

Active learning of manipulation sequences

“…Moreover, it is robust to a considerable amount of clutter nodes and edges that are unrelated to the action of interest. All these aspects are allowed to vary, and still the same SEC is observed and captures the "essence of the action", as demonstrated with diverse sets of real actions in our earlier work [2], [3].…”

“…The robotic actor and its environment are simulated in Virtual Reality (VR) 1 . The evolution of the manipulation sequence is extracted and represented in the form of so-called Semantic Event Chains [2], [3]. From these, the ManipulationRecognition module extracts plausible actions in progress and recognizes completed actions.…”

Recognizing object affordances in terms of spatio-temporal object-object relationships

“…The scenario we focus on in this paper is a human demonstrator teaching a robot about the affordances of objects by showing how they are used. To that end we will, in line with previous work [4], assume that the human is responsible for all the movement in the scene. Furthermore, we assume that the relationships of each pair of objects involved in an activity are dependent, and use a graphical model to model correlation between all objectobject interactions, in order to improve the recognition of functional classes of all objects in the scene and mitigate misleading information.…”