Accurate Vision-based Manipulation through Contact Reasoning

Kloss, Alina; Bauzá, Maria; Tenenbaum, Joshua B.; Rodríguez, Alberto; Bohg, Jeannette

doi:10.1109/icra40945.2020.9197409

Cited by 16 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Past work in pushing manipulation incorporates simulation and analytical models in a motion planning loop to get the next feasible state [13] [14]. Additional contributions in push modeling involve combining object state estimation with affordance prediction from image data to determine contact points for achieving the optimal push [15] and creating a deep recurrent neural network model to model push outcomes for a variety of objects [16]. However, both approaches use a greedy planner operating in obstaclefree environments.…”

Section: Background and Related Workmentioning

confidence: 99%

PokeRRT: Poking as a Skill and Failure Recovery Tactic for Planar Non-Prehensile Manipulation

Pasricha¹,

Tung²,

Hayes³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this work, we introduce PokeRRT, a novel motion planning algorithm that demonstrates poking as an effective non-prehensile manipulation skill to enable fast manipulation of objects and increase the size of a robot's reachable workspace. We showcase poking as a failure recovery tactic used synergistically with pick-and-place for resiliency in cases where pick-and-place initially fails or is unachievable. Our experiments demonstrate the efficiency of the proposed framework in planning object trajectories using poking manipulation in uncluttered and cluttered environments. In addition to quantitatively and qualitatively demonstrating the adaptability of PokeRRT to different scenarios in both simulation and real-world settings, our results show the advantages of poking over pushing and grasping in terms of success rate and task time.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

PokeRRT: Poking as a Skill and Failure Recovery Tactic for Planar Non-Prehensile Manipulation

Pasricha¹,

Tung²,

Hayes³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In Bejjani et al (2019), rendered images from a simulator were used as state representation to exploit the generalization ability of CNNs. The work of Kloss et al (2020) exploits imagebased representations for pushing scenarios, similar to our encoding. In our work, the network learns a representation in image space that is able to reason over complex action sequences from an initial observation only and is able to generalize over changing numbers of objects.…”

Section: Learning Heuristics For Tamp and Mip In Roboticsmentioning

confidence: 99%

Learning to solve sequential physical reasoning problems from a scene image

Driess

Toussaint

2021

The International Journal of Robotics Research

View full text Add to dashboard Cite

In this article, we propose deep visual reasoning, which is a convolutional recurrent neural network that predicts discrete action sequences from an initial scene image for sequential manipulation problems that arise, for example, in task and motion planning (TAMP). Typical TAMP problems are formalized by combining reasoning on a symbolic, discrete level (e.g., first-order logic) with continuous motion planning such as nonlinear trajectory optimization. The action sequences represent the discrete decisions on a symbolic level, which, in turn, parameterize a nonlinear trajectory optimization problem. Owing to the great combinatorial complexity of possible discrete action sequences, a large number of optimization/motion planning problems have to be solved to find a solution, which limits the scalability of these approaches. To circumvent this combinatorial complexity, we introduce deep visual reasoning: based on a segmented initial image of the scene, a neural network directly predicts promising discrete action sequences such that ideally only one motion planning problem has to be solved to find a solution to the overall TAMP problem. Our method generalizes to scenes with many and varying numbers of objects, although being trained on only two objects at a time. This is possible by encoding the objects of the scene and the goal in (segmented) images as input to the neural network, instead of a fixed feature vector. We show that the framework can not only handle kinematic problems such as pick-and-place (as typical in TAMP), but also tool-use scenarios for planar pushing under quasi-static dynamic models. Here, the image-based representation enables generalization to other shapes than during training. Results show runtime improvements of several orders of magnitudes by, in many cases, removing the need to search over the discrete action sequences.

show abstract

“…Prior work in visual manipulation has involved learning predictive models from videos of a robot interacting with its environment. These models are used to find actions with simulation roll-outs, in a model-predictive fashion [16,17,1,18,19,20,21]. These approaches have been successful at solving prehensile manipulation problems where the dynamics are hard to model.…”

Section: Related Workmentioning

confidence: 99%

A Differentiable Recipe for Learning Visual Non-Prehensile Planar Manipulation

Aceituno¹,

Rodríguez²,

Tulsiani³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Specifying tasks with videos is a powerful technique towards acquiring novel and general robot skills. However, reasoning over mechanics and dexterous interactions can make it challenging to scale learning contact-rich manipulation. In this work, we focus on the problem of visual non-prehensile planar manipulation: given a video of an object in planar motion, find contact-aware robot actions that reproduce the same object motion. We propose a novel architecture, Differentiable Learning for Manipulation (DLM), that combines video decoding neural models with priors from contact mechanics by leveraging differentiable optimization and finite difference based simulation. Through extensive simulated experiments, we investigate the interplay between traditional model-based techniques and modern deep learning approaches. We find that our modular and fully differentiable architecture performs better than learning-only methods on unseen objects and motions. https://github.com/baceituno/dlm.

show abstract

Accurate Vision-based Manipulation through Contact Reasoning

Cited by 16 publications

References 21 publications

PokeRRT: Poking as a Skill and Failure Recovery Tactic for Planar Non-Prehensile Manipulation

PokeRRT: Poking as a Skill and Failure Recovery Tactic for Planar Non-Prehensile Manipulation

Learning to solve sequential physical reasoning problems from a scene image

A Differentiable Recipe for Learning Visual Non-Prehensile Planar Manipulation

Contact Info

Product

Resources

About