A Robotic Recording and Playback Platform for Training Surgeons and Learning Autonomous Behaviors Using the da Vinci Surgical System

Abstract

This paper describes a recording and playback system developed using a da Vinci Standard Surgical System and research kit. The system records stereo laparoscopic videos, robot arm joint angles, and surgeon–console interactions in a synchronized manner. A user can then, on-demand and at adjustable speeds, watch stereo videos and feel recorded movements on the hand controllers of entire procedures or sub procedures. Currently, there is no reported comprehensive ability to capture expert surgeon movements and insights and reproduce them on hardware directly. This system has important applications in several areas: (1) training of surgeons, (2) collection of learning data for the development of advanced control algorithms and intelligent autonomous behaviors, and (3) use as a “black box” for retrospective error analysis. We show a prototype of such an immersive system on a clinically-relevant platform along with its recording and playback fidelity. Lastly, we convey possible research avenues to create better systems for training and assisting robotic surgeons.

Development and evaluation of an autonomous camera control algorithm on the da Vinci Surgical System

“…This recording system can provide objective data for the evaluation of different camera control methodologies (see Figure ). As a participant performs a task that needs evaluation, all the data are collected for later analysis …”

“…As a participant performs a task that needs evaluation, all the data are collected for later analysis. 2 The data that were collected during this human factors study included the joint angles of all the instrument arms, camera arm, and hand controllers. In addition, the images from the stereo cameras were also collected.…”

“…We describe a hardware test platform that was used to perform these evaluations on a da Vinci Surgical System (Intuitive Surgical, Inc, Sunnyvale, CA). Features of this test platform include baseline camera control mechanisms (clutched, joystick, and automated), software modularity that allows alternative camera control mechanisms/algorithms, and a recording capability that enables synchronized recordings of kinematic and video data for participant evaluation . These features have allowed rigorous human operator testing to compare current manual methods of camera control with newer methods (eg, autonomous computerized control).…”

“…Features of this test platform include baseline camera control mechanisms (clutched, joystick, and automated), software modularity that allows alternative camera control mechanisms/algorithms, and a recording capability that enables synchronized recordings of kinematic and video data for participant evaluation. 2 These features have allowed rigorous human operator testing to compare current manual methods of camera control with newer methods (eg, autonomous computerized control). In this light, the system has provided an important and clinically relevant test platform for the evaluation and comparison of new control techniques for camera navigation.…”

Review of Advanced Medical Telerobots

“…Another interest of surgical telerobots is their ability to help train surgeons. For instance, in [77], the daVinci robot is able to record and simultaneously playback laparoscopic videos, robot arm motions, and surgeon-console interactions. A user can then replay ondemand recorded procedure sections with the opportunity to watch stereo videos and simultaneously feel the recorded movements on the hand controllers, at adjustable speeds.…”