Automatic Operating Room Surgical Activity Recognition for Robot-Assisted Surgery

Abstract

Automatic recognition of surgical activities in the operating room (OR) is a key technology for creating next generation intelligent surgical devices and workflow monitoring/support systems. Such systems can potentially enhance efficiency in the OR, resulting in lower costs and improved care delivery to the patients. In this paper, we investigate automatic surgical activity recognition in robot-assisted operations. We collect the first large-scale dataset including 400 full-length multiperspective videos from a variety of robotic surgery cases captured using Time-of-Flight cameras. We densely annotate the videos with 10 most recognized and clinically relevant classes of activities. Furthermore, we investigate state-of-the-art computer vision action recognition techniques and adapt them for the OR environment and the dataset. First, we finetune the Inflated 3D ConvNet (I3D) for clip-level activity recognition on our dataset and use it to extract features from the videos. These features are then fed to a stack of 3 Temporal Gaussian Mixture layers which extracts context from neighboring clips, and eventually go through a Long Short Term Memory network to learn the order of activities in full-length videos. We extensively assess the model and reach a peak performance of ∼88% mean Average Precision.

Know your sensORs -- A Modality Study For Surgical Action Classification

“…RGB and depth images are fused by concatenation to generate a four-channel image according to previous early fusion approaches [34]. Furthermore, we attempt to reproduce the depth-ir fusion from [30,28], by color-coding the depth data and alpha-blending them with the raw IR images. All images are augmented using a small random affine transformation followed by cropping the 224x224 pixel center.…”

“…In these settings, action recognition is typically performed on short image sequences, referred to as "clips", which average only 10 seconds in length for Kinetics [17]. Surgical workflow analysis from room cameras is currently limited to one large dataset, which is not publicly available [30,28]. Despite publishing both RGB and depth sequences, the MVOR [31] dataset consists of 732 frames at relatively low FPS, making video action recognition infeasible.…”

“…Cliplevel architectures, however, remain relevant even for longer videos, as they can be similarly combined with LSTMs to aggregate both short and long-term image features [28]. Extracting clip features can provide performance benefits as they reduce dimensionality compared to image level features [30]. Furthermore, they can extract information from short video segments where training LSTMs may be infeasible.…”

“…Classifying procedures obtained from OR room cameras has become a topic of interest in the past several years [28,30]. Temporally aggregating features extracted from video frames enables an understanding of complex workflow tasks that single image analysis alone could not achieve.…”

“…The deployment of such systems holds promise for many applications, such as optimizing real-time OR scheduling, surgical workflow, or enabling autonomous anomaly detection [21,22]. However, existing datasets have thus far been limited in size [31], or modalities captured [30], leaving many questions on how to build automated systems for the OR unanswered. Modern commercial sensors are typically able to acquire RGB, depth, and infrared video streams, although regulatory and technological constraints continue to make acquisition, storage and access challenging in clinical settings [22].…”