Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources

Abstract

Many tasks in robot-assisted surgeries (RAS) can be represented by finite-state machines (FSMs), where each state represents either an action (such as picking up a needle) or an observation (such as bleeding). A crucial step towards the automation of such surgical tasks is the temporal perception of the current surgical scene, which requires a real-time estimation of the states in the FSMs. The objective of this work is to estimate the current state of the surgical task based on the actions performed or events occurred as the task progresses. We propose Fusion-KVE, a unified surgical state estimation model that incorporates multiple data sources including the Kinematics, Vision, and system Events. Additionally, we examine the strengths and weaknesses of different state estimation models in segmenting states with different representative features or levels of granularity. We evaluate our model on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), as well as a more complex dataset involving robotic intra-operative ultrasound (RIOUS) imaging, created using the da Vinci R Xi surgical system. Our model achieves a superior frame-wise state estimation accuracy up to 89.4%, which improves the state-of-the-art surgical state estimation models in both JIGSAWS suturing dataset and our RIOUS dataset.

daVinciNet: Joint Prediction of Motion and Surgical State in Robot-Assisted Surgery

“…Implementation details: Each endoscope video frame was resized to a 224 × 224 × 3 RGB image before being input to the VGG-16 model. The VGG-16 model was pre-trained following our previous work [17]. m = 1024 CNN features were extracted.…”

“…It is worth noting that in order to obtain the historic sequence of surgical state s from t − T obs + 1 to t, we implemented Fusion-KVE -a unified surgical state estimation model we proposed recently [17]-instead of using the ground truth (GT) state sequence. In real-time RAS settings, the surgical state prediction model does not have access to the manually-labeled historic surgical state sequence; therefore, a state estimation model is needed to provide the historic state sequence.…”

“…Surgical task such as suturing can be practically modeled as a Finite State Machine (FSM), with a list of discrete states (actions and non-actions) and possible transitions between states [17]. Classically, the task is formulated as a MM and the transition probability matrix is learned from data [8,12,[18][19][20].…”

“…This is especially useful in the prediction of surgical states, since different sources of input data have their respective strengths and weaknesses in representing states with various kinematics and visual features. Previously, we have proposed a unified model for surgical state estimation -Fusion-KVE -that incorporated multiple types of input data and exceeded the state-of-the-art state estimation performance [17]. Building on this, we explored the task of concurrent instrument path and surgical state predictions with multiple data streams and the incorporation of historic state transition sequences.…”

“…Our model's performance was evaluated using the JIGSAWS suturing dataset [32] and the extended Robotic Intra-Operative Ultrasound (RIOUS+) imaging dataset [17]. The RIOUS+ dataset contains ultrasound imaging trials in various experimental settings (phantom, in-vivo, and cadaver) and endoscopic motion.…”