Learning-driven Coarse-to-Fine Articulated Robot Tracking

Abstract

In this work we present an articulated tracking approach for robotic manipulators, which relies only on visual cues from colour and depth images to estimate the robot's state when interacting with or being occluded by its environment. We hypothesise that articulated model fitting approaches can only achieve accurate tracking if subpixel-level accurate correspondences between observed and estimated state can be established. Previous work in this area has exclusively relied on either discriminative depth information or colour edge correspondences as tracking objective and required initialisation from joint encoders. In this paper we propose a coarse-to-fine articulated state estimator, which relies only on visual cues from colour edges and learned depth keypoints, and which is initialised from a robot state distribution predicted from a depth image. We evaluate our approach on four RGB-D sequences showing a KUKA LWR arm with a Schunk SDH2 hand interacting with its environment and demonstrate that this combined keypoint and edge tracking objective can estimate the palm position with an average error of 2.5cm without using any joint encoder sensing.

Sparse-Dense Motion Modelling and Tracking for Manipulation Without Prior Object Models

“…Such semantic tracking methods initially use a labelled training set of the target objects and eventually represents them in a latent space [3], [4]. Hybrid methods can use the geometric and visual model to generate such training data [5].…”

Intermittent Visual Servoing: Efficiently Learning Policies Robust to Instrument Changes for High-precision Surgical Manipulation

“…This enables the system to rapidly learn a robust policy for the region of interest, while relying on a coarse, openloop policy otherwise. Coarse-to-fine control architectures combining geometric planners with adaptive error correction strategies have a long history in robotics [24], [43], [38], [50], [23], with works such as Lozano-Pérez et al [24] studying the combination of geometric task descriptions with sensing and error correction for compliant motions. We extend the work of Lee et al [21], who use a model-based planner for moving a robot arm in free space, and reinforcement learning for learning an insertion policy when the gripper is near the region of interest.…”

Do We Need to Create Big Datasets to Learn a Task?

“…We mimic the relevant material selection process in humans to propose algorithm for selecting the optimal dataset necessary to learn a task, as illustrated in Figure 1. We use robotics terminology (Rauch et al, 2019) to explain the stages of learning (i) coarse action (ii) fine action. Algorithm 1 details our approach.…”