Monocular Visual Shape Tracking and Servoing for Isometrically Deforming Objects

Abstract

We address the monocular visual shape servoing problem. This pushes the challenging visual servoing problem one step further from rigid object manipulation towards deformable object manipulation. Explicitly, it implies deforming the object towards a desired shape in 3D space by robots using monocular 2D vision. We specifically concentrate on a scheme capable of controlling large isometric deformations. Two important open subproblems arise for implementing such a scheme. (P1) Since it is concerned with large deformations, perception requires tracking the deformable object's 3D shape from monocular 2D images which is a severely underconstrained problem. (P2) Since rigid robots have fewer degrees of freedom than a deformable object, the shape control becomes underactuated. We propose a template-based shape servoing scheme in which we solve these two problems. The template allows us to both infer the object's shape using an improved Shape-from-Template algorithm and steer the object's deformation by means of the robots' movements. We validate the scheme via simulations and real experiments.

Modal-graph 3D shape servoing of deformable objects with raw point clouds

“…Other researchers explored the use of object geometries to approximate deformation behaviors under robot manipulation. They proposed control laws using the diminishing-rigidity approximation (McConachie et al (2020)), the As-Rigid-As-Possible (ARAP) model (Shetab-Bushehri et al (2022)), and the shape-template-based method (Aranda et al (2020)). Recently, increasing attention has been put to the deformation learning approaches that embed geometric graph structures into neural architectures.…”

An efficient approach to closed-loop shape control of deformable objects using finite element models

“…If the task requires to position multiple nodes, then userdefined values u * t must define a physically feasible solution. Through pseudo-inverse computation, the control law (12) can only minimize a global error on all target nodes, but not the local error on each of the target nodes.…”

“…Analysis: As shown by expressions ( 17) and ( 18), stiffness matrices have a linear dependency on E. From control law (12), the terms in E in K d , K t cancel out and the error signal still follows the first-order convergent behavior defined by (10). Practically, misidentification of the Young's modulus and neglect of other effects such as bending can be compensated by tuning the proportional gain matrix G p accordingly.…”

“…Model-based approaches: Even though popular older works are known, as evidenced by the review [11], modelbased shape servoing gets less attention than its modelfree counterpart nowadays. Non-mechanical models such as templates [12] are computationally inexpensive and well suited to visual servoing, but their results guarantee the visual convergence of some points over the surface of the object; consequently, the mechanical constraints related with its material properties are not taken into account. In some cases, low-complexity mechanical models can be used, such as catenaries for isometric linear objects [13].…”

3D visual-based tension control in strip-like deformable objects using a catenary model

“…The deformation of the surface object is estimated in [11] using non-uniform rational basis splines approximation with a RGB-D camera. A shape control of isometrically deforming objects with 2D camera is proposed in [12]. These methods show a good balance between computational cost and accuracy.…”