Model Predictive Manipulation of Compliant Objects with Multi-Objective Optimizer and Adversarial Network for Occlusion Compensation

Abstract

The robotic manipulation of compliant objects is currently one of the most active problems in robotics due to its potential to automate many important applications. Despite the progress achieved by the robotics community in recent years, the 3D shaping of these types of materials remains an open research problem. In this paper, we propose a new vision-based controller to automatically regulate the shape of compliant objects with robotic arms. Our method uses an efficient online surface/curve fitting algorithm that quantifies the object's geometry with a compact vector of features; This feedback-like vector enables to establish an explicit shape servo-loop. To coordinate the motion of the robot with the computed shape features, we propose a receding-time estimator that approximates the system's sensorimotor model while satisfying various performance criteria. A deep adversarial network is developed to robustly compensate for visual occlusions in the camera's field of view, which enables to guide the shaping task even with partial observations of the object. Model predictive control is utilized to compute the robot's shaping motions subject to workspace and saturation constraints. A detailed experimental study is presented to validate the effectiveness of the proposed control framework.

Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds

“…Solutions to 3D deformable object shape control [1] can be categorized into learning-based and learning-free approaches. Among the learning-free methods, a series of papers [17][18][19][20][21][22] define a set of geometric feature points on the object as the state representation. The authors use this representation to perform visual servoing with an adaptive linear controller that estimates the Jacobian matrix of the deformable object.…”

“…Further, this formulation controls the displacements of individual points which may not fully reflect the 3D shape of the object. Other learning-free works [20,[23][24][25] represent the object shape using 2D image contours; limiting the space of controllable 3D deformations. Most recently, Shetab-Bushehri et al [26] model the deformable obeject as a 3D lattice and successfully achieve full 3D control.…”