Intermittent Visual Servoing: Efficiently Learning Policies Robust to Instrument Changes for High-precision Surgical Manipulation

Paradis, Samuel; Hwang, Mina; Thananjeyan, Brijen; Ichnowski, Jeffrey; Seita, Daniel; Fer, Danyal; Low, Thomas; Goldberg, Ken

doi:10.48550/arxiv.2011.06163

Cited by 3 publications

(8 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their results suggest that the autonomous robot attains almost the same block transfer success rate while operating at twice the speed of the human operator. We extend Rosen and Ma's work, and our prior work, [18][19][20] by considering 3 variants of the peg transfer task, all of which require 6 blocks and involve transfer in both directions. We focus on the FLS peg-transfer task, using the setup in Hwang et al 19 that uses red 3D-printed blocks and a 3D-printed pegboard (see Fig.…”

Section: Peg Transfer Taskmentioning

confidence: 95%

“…Second, while Hwang et al 18,19 and Paradis et al 20 are able to meet human-level success rates on the peg transfer task, these approaches are 1.5x to 2x slower than a skilled human operator, respectively. In this new paper, we speed up motions by optimizing the robot arm trajectories.…”

Section: Introductionmentioning

confidence: 98%

“…This paper combines and extends our prior work on automating peg transfer using calibrated depth-sensing, 18 fiducial tracking to learn models of cabling effects, 19 and intermittent visual servoing, 20 by additionally proposing several novel techniques for improving surgical robot precision, speed, and ease of use:…”

Section: Introductionmentioning

confidence: 99%

“…The paper evaluates a new robot controller on the peg-transfer task and improves the physical properties of the peg board and blocks using 3D printing software, making the setup more reproducible and less likely to cause hardware damage. • Paradis et al 20 propose intermittent visual servoing (IVS), a paradigm for using a coarse open-loop controller in free space, while using learned visual servoing 21 when the robot is close to an area of interest, such as when picking or placing a block. Paradis et al learn visual servoing from demonstrations, and in peg-transfer experiments, show that the learned model can be robust to tool changes, which would not be the case with the calibration methods used in Hwang et al 18,19 However, IVS requires additional time during the correction phase.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Automating Surgical Peg Transfer: Calibration with Deep Learning Can Exceed Speed, Accuracy, and Consistency of Humans

Hwang,

Ichnowski,

Thananjeyan

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

We consider the automation of the well-known peg-transfer task from the Fundamentals of Laparoscopic Surgery (FLS). While human surgeons teleoperate robots to perform this task with great dexterity, it remains challenging to automate. We present an approach that leverages emerging innovations in depth sensing, deep learning, and Peiper's method for computing inverse kinematics with time-minimized joint motion. We use the da Vinci Research Kit (dVRK) surgical robot with a Zivid depth sensor, and automate three variants of the peg-transfer task: unilateral, bilateral without handovers, and bilateral with handovers. We use 3D-printed fiducial markers with depth sensing and a deep recurrent neural network to improve the precision of the dVRK to less than 1 mm. We report experimental results for 1800 block transfer trials. Results suggest that the fully automated system can outperform an experienced human surgical resident, who performs far better than untrained humans, in terms of both speed and success rate. For the most difficult variant of peg transfer (with handovers) we compare the performance of the surgical resident with performance of the automated system over 120 trials for each. The experienced surgical resident achieves success rate 93.2 % with mean transfer time of 8.6 seconds. The automated system achieves success rate 94.1 % with mean transfer time of 8.1 seconds. To our knowledge this is the first fully automated system to achieve "superhuman" performance in both speed and success on peg transfer. Supplementary material is available at https://sites.google.com/view/surgicalpegtransfer.

show abstract

Section: Peg Transfer Taskmentioning

confidence: 95%

Section: Introductionmentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Automating Surgical Peg Transfer: Calibration with Deep Learning Can Exceed Speed, Accuracy, and Consistency of Humans

Hwang,

Ichnowski,

Thananjeyan

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…LaGrassa et al [15] propose a similar idea, but they focus on detecting model failures to execute a model free policy that is learned from demonstrations. Paradis et al [18] showcase a coarse-to-fine approach for surgical robotics, where they intermittently switch between the coarse and fine controllers for a pick and place task. Raj et al [21] study different schemes for switching neural networks between two modes of operation, namely large and small scale displacements within a given task.…”

Section: B Coarse-to-fine Controllers For Manipulationmentioning

confidence: 99%

Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spaces

Valassakis¹,

Palo²,

Johns³

2021

Preprint

View full text Add to dashboard Cite

When training control policies for robot manipulation via deep learning, sim-to-real transfer can help satisfy the large data requirements. In this paper, we study the problem of zero-shot sim-to-real when the task requires both highly precise control, with sub-millimetre error tolerance, and full workspace generalisation. Our framework involves a coarse-tofine controller, where trajectories initially begin with classical motion planning based on pose estimation, and transition to an end-to-end controller which maps images to actions and is trained in simulation with domain randomisation. In this way, we achieve precise control whilst also generalising the controller across the workspace and keeping the generality and robustness of vision-based, end-to-end control. Real-world experiments on a range of different tasks show that, by exploiting the best of both worlds, our framework significantly outperforms purely motion planning methods, and purely learning-based methods.Furthermore, we answer a range of questions on best practices for precise sim-to-real transfer, such as how different image sensor modalities and image feature representations perform.

show abstract

Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration

Johns¹

2021

Preprint

View full text Add to dashboard Cite

We introduce a simple new method for visual imitation learning, which allows a novel robot manipulation task to be learned from a single human demonstration, without requiring any prior knowledge of the object being interacted with. Our method models imitation learning as a state estimation problem, with the state defined as the end-effector's pose at the point where object interaction begins, as observed from the demonstration. By modelling a manipulation task as a coarse, approach trajectory followed by a fine, interaction trajectory, this state estimator can be trained in a self-supervised manner, by automatically moving the end-effector's camera around the object. At test time, the end-effector is moved to the estimated state through a linear path, at which point the demonstration's end-effector velocities are simply repeated, enabling convenient acquisition of a complex interaction trajectory without actually needing to explicitly learn a policy. Real-world experiments on 8 everyday tasks show that our method can learn a diverse range of skills from just a single human demonstration, whilst also yielding a stable and interpretable controller.

show abstract

Intermittent Visual Servoing: Efficiently Learning Policies Robust to Instrument Changes for High-precision Surgical Manipulation

Cited by 3 publications

References 43 publications

Automating Surgical Peg Transfer: Calibration with Deep Learning Can Exceed Speed, Accuracy, and Consistency of Humans

Automating Surgical Peg Transfer: Calibration with Deep Learning Can Exceed Speed, Accuracy, and Consistency of Humans

Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spaces

Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration

Contact Info

Product

Resources

About