Learning by observation for surgical subtasks: Multilateral cutting of 3D viscoelastic and 2D Orthotropic Tissue Phantoms

Abstract

Abstract-Automating repetitive surgical subtasks such as suturing, cutting and debridement can reduce surgeon fatigue and procedure times and facilitate supervised tele-surgery. Programming is difficult because human tissue is deformable and highly specular. Using the da Vinci Research Kit (DVRK) robotic surgical assistant, we explore a "Learning By Observation" (LBO) approach where we identify, segment, and parameterize sub-trajectories ("surgemes") and sensor conditions to build a finite state machine (FSM) for each subtask. The robot then executes the FSM repeatedly to tune parameters and if necessary update the FSM structure. We evaluate the approach on two surgical subtasks: debridement of 3D Viscoelastic Tissue Phantoms (3d-DVTP), in which small target fragments are removed from a 3D viscoelastic tissue phantom, and Pattern Cutting of 2D Orthotropic Tissue Phantoms (2d-PCOTP), a step in the standard Fundamentals of Laparoscopic Surgery training suite, in which a specified circular area must be cut from a sheet of orthotropic tissue phantom. We describe the approach and physical experiments, which yielded a success rate of 96% for 50 trials of the 3d-DVTP subtask and 70% for 20 trials of the 2d-PCOTP subtask.

Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization

“…Many of the FSRS procedures, including MTS, are decomposable into long sequences of simpler sub-tasks. This decomposition allows the parametrization and building of Finite State Machines (FSM) for complex procedures using a learning by observation approach, for tasks such as tissue debridement [13], pattern cutting [21], and tumor localization & resection [19]. Our work on segmentation of multi-step task demonstrations [15] suggests that unsupervised learning of semantic transitions is feasible and can be analyzed to construct FSMs for these multi-step tasks.…”

“…This paper builds on prior work in optimization-based planning [4,27], sub-task level segmentation of demonstrations [15,16], gripper mounted interchangeable tools [19], and building robust finite state machines [21]. We are not aware of any system that can perform autonomous multithrow suturing.…”

“…We use the Intuitive Surgical da Vinci Research Kit (dVRK) surgical robot assistant as in [21], along with opensource electronics and software developed by WPI and Johns Hopkins University [12]. We use a pair of 8mm Needle Drivers with each gripper having one Suture Needle Angular Positioner (SNAP).…”

Haptics in Robot-Assisted Surgery: Challenges and Benefits

“…This is mainly due to substantial technical complications involved in executing automatic actions in medical interventions with the demanded reliability due to the criticality of the field. Current stage of autonomous execution of surgical subtasks [15] is encouraging, but far from approvable for clinical implementation and replacing clinicians. Robots of non-autonomous type are not designed to replace the surgeon.…”

Surgical data science, an emerging field of medicine

“…Recently several surgical phase detection algorithms have been presented using Random Forests (RF), Hidden Markov Models (HMM) and deep neural networks [30], [31], [32]. Using the da Vinci Research kit [33], [34], it has been shown in a clinical environment that subtask automation is feasible [35], and SPMs were used to integrate automated robotic intraoperative imaging [36]. Another aspect of workflow monitoring is OR workflow scheduling [37].…”