Learning the semantics of object–action relations by observation

Abstract

Recognizing manipulations performed by a human and the transfer and execution of this by a robot is a difficult problem. We address this in the current study by introducing a novel representation of the relations between objects at decisive time points during a manipulation. Thereby, we encode the essential changes in a visual scenery in a condensed way such that a robot can recognize and learn a manipulation without prior object knowledge. To achieve this we continuously track image segments in the video and construct a dynamic graph sequence. Topological transitions of those graphs occur whenever a spatial relation between some segments has changed in a discontinuous way and these moments are stored in a transition matrix called the semantic event chain (SEC). We demonstrate that these time points are highly descriptive for distinguishing different manipulations. Employing simple sub-string search algorithms, semantic event chains can be compared and type-similar manipulations can be recognized with high confidence. As the approach is generic, statistical learning can be used to find the archetypal SEC of a given manipulation class. The performance of the algorithm is demonstrated on a set of real videos showing hands manipulating various objects and performing different actions. In experiments with a robotic arm, we show that the SEC can be learned by observing human manipulations, transferred to a new scenario, and then reproduced by the machine.

Anticipatory Planning for Human-Robot Teams

“…There is some recent work in interpreting human actions and interaction with objects [25,1,17] in context of learning to perform actions from demonstrations. Lopes et al [25] use context from objects in terms of possible grasp a↵ordances to focus the attention of their recognition system.…”

“…Lopes et al [25] use context from objects in terms of possible grasp a↵ordances to focus the attention of their recognition system. Aksoy et al [1] construct a dynamic graph sequence of tracked image segments from human demonstrations and this representation is used by the robot for manipulating objects. A↵ordances have also been used in planning (e.g., [26,43]).…”

“…1 Once we have the set of states, set of actions and the task model, we can solve the MDP using dynamic programming techniques [35]. However, with large joint stateaction space, computing the optimal policy is computationally very expensive and therefore, we take the model-free approach of Q-learning and learn the Qfunctions o✏ine with the help of the learnt task models.…”

Physically Grounded Spatio-temporal Object Affordances

“…Aldoma et al [2] proposed a method to find affordances which depends solely on the objects of interest and their position and orientation in the scene. There is some recent work in interpreting human actions and interaction with objects [26,1,20] in context of learning to perform actions from demonstrations. Lopes et al [26] use context from objects in terms of possible grasp affordances to focus the attention of their recognition system.…”

“…For evaluating the quality of the predicted temporal affordance, we compute the modified Hausdorff distance (MHD) as a physical measure of the distance between the predicted object motion trajectories and the true object trajectory from the test data. 1 Baseline Algorithms. We compare our method against the following baselines: 1) Chance: It selects a random training instance for the given human intention and uses its affordances as the predictions.…”