FlowControl: Optical Flow Based Visual Servoing

Argus, Max; Hermann, Lukás; Long, Jon; Brox, Thomas

doi:10.1109/iros45743.2020.9340942

Cited by 12 publications

(6 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, unlike prior work [35] which iteratively minimizes flow with pick and place actions, or other work [36] which uses optical flow on tactile sensors, we use flow to derive continuous tool motions in 3D space from visual input. A recent work [37] estimates optical flow using RGBD images from the current frame to the demonstration and extracts a transformation to align them. In contrast, we do not use flow for aligning frames to demonstrations but for deriving the transformations the tool should follow.…”

Section: Related Workmentioning

confidence: 99%

ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds

Seita¹,

Wang²,

Shetty³

et al. 2022

Preprint

View full text Add to dashboard Cite

Point clouds are a widely available and canonical data modality which convey the 3D geometry of a scene. Despite significant progress in classification and segmentation from point clouds, policy learning from such a modality remains challenging, and most prior works in imitation learning focus on learning policies from images or state information. In this paper, we propose a novel framework for learning policies from point clouds for robotic manipulation with tools. We use a novel neural network, ToolFlowNet, which predicts dense perpoint flow on the tool that the robot controls, and then uses the flow to derive the transformation that the robot should execute. We apply this framework to imitation learning of challenging deformable object manipulation tasks with continuous movement of tools, including scooping and pouring, and demonstrate significantly improved performance over baselines which do not use flow. We perform 50 physical scooping experiments with ToolFlowNet and attain 82% scooping success. See https://tinyurl.com/toolflownet for supplementary material.

show abstract

Section: Related Workmentioning

confidence: 99%

ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds

Seita¹,

Wang²,

Shetty³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Another set of methods include approaches that design task-specific controllers (such as for pick and place [2] or cloth folding [11]), and can imitate variations within this task. Similar to our work, FlowControl [1], utilises optical flow for aligning to demonstration frames to complete a task. Unlike our method however, it relies on manually supplied object segmentations, and its core flow computation does not work well when the deployment frames differ significantly from the demonstration frames, or when there are background flows, which would occur with the addition of distractor objects.…”

Section: Related Workmentioning

confidence: 99%

Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learning

Valassakis¹,

Papagiannis²,

Palo³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present DOME, a novel method for one-shot imitation learning, where a task can be learned from just a single demonstration and then be deployed immediately, without any further data collection or training. DOME does not require prior task or object knowledge, and can perform the task in novel object configurations and with distractors. At its core, DOME uses an image-conditioned object segmentation network followed by a learned visual servoing network, to move the robot's end-effector to the same relative pose to the object as during the demonstration, after which the task can be completed by replaying the demonstration's end-effector velocities. We show that DOME achieves near 100% success rate on 7 real-world everyday tasks, and we perform several studies to thoroughly understand each individual component of DOME.

show abstract

“…[32], [33] attempt to align the robot's current image observation with a goal image, but require task-specific controllers to be manually defined for object interaction, whereas our method can learn from just a demonstration. [34] use visual servoing to directly track a demonstration, but require close initial alignment and only evaluate on tasks with very limited object interaction, whereas our method generalises across a wide task space and can facilitate complex interaction trajectories.…”

Section: Related Workmentioning

confidence: 99%

Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration

Johns¹

2021

Preprint

View full text Add to dashboard Cite

We introduce a simple new method for visual imitation learning, which allows a novel robot manipulation task to be learned from a single human demonstration, without requiring any prior knowledge of the object being interacted with. Our method models imitation learning as a state estimation problem, with the state defined as the end-effector's pose at the point where object interaction begins, as observed from the demonstration. By modelling a manipulation task as a coarse, approach trajectory followed by a fine, interaction trajectory, this state estimator can be trained in a self-supervised manner, by automatically moving the end-effector's camera around the object. At test time, the end-effector is moved to the estimated state through a linear path, at which point the demonstration's end-effector velocities are simply repeated, enabling convenient acquisition of a complex interaction trajectory without actually needing to explicitly learn a policy. Real-world experiments on 8 everyday tasks show that our method can learn a diverse range of skills from just a single human demonstration, whilst also yielding a stable and interpretable controller.

show abstract

FlowControl: Optical Flow Based Visual Servoing

Cited by 12 publications

References 30 publications

ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds

ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds

Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learning

Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration

Contact Info

Product

Resources

About