PEg TRAnsfer Workflow recognition challenge report: Does multi-modal data improve recognition?

Abstract

Background and Objective: Context-aware computer-assisted surgical systems require real time automatic and accurate surgical workflow recognition. Since several years, video has been the most common modality used to develop such methods. With the democratization of robotic-assisted surgery and segmentation method, new modalities are now accessible, such as kinematics. Some previous works used these new modalities as input of their models, but the added value of these modalities is rarely studied. This paper presents the design and results of the "PEg TRAnsfert Workflow recognition" (PETRAW) challenge whose objective was to develop surgical workflow recognition methods based on one or several modalities in order to study their added value. Methods: The PETRAW challenge provided a data set of 150 peg transfer sequences performed on a virtual simulator. This data set was composed of videos, kinematics, semantic segmentation, and workflow annotations which described the sequences at three different granularity levels: phase, step, and activity. Five tasks were proposed to the participants: three of them were related to the recognition of all granularities with one of the available modalities, while the others addressed the recognition with a combination of modalities. Average application-dependent balanced accuracy (AD-Accuracy) was used as evaluation metric to take unbalanced classes into account and because it is more clinically relevant than a frame-by-frame score. Results: Seven teams participated in at least one task and four of them in all tasks. Best results are obtained with the use of the video and the kinematics data with an AD-Accuracy between 93% and 90% for the four teams who participated in all tasks. Conclusion:The improvement between video/kinematic-based methods and the uni-modality ones was significant for all of the teams. However, the difference in testing execution time between the video/kinematic-based and the kinematic-based methods has to be taken into consideration. Is it relevant to spend 20 to 200 times more computing time for less than 3% of improvement? The PETRAW data set is publicly available at www.synapse.org/PETRAW to encourage further research in surgical workflow recognition.

Evaluating the Task Generalization of Temporal Convolutional Networks for Surgical Gesture and Motion Recognition Using Kinematic Data

“…Note that there are inconsistencies in label and granularity definitions across datasets. For example, the tasks of Suturing, Knot Tying, and Peg Transfer in JIGSAWS and DESK are considered phases in MISAW [31] and PETRAW [36]. [13] trained a GRU for gesture and maneuver recognition on the JIGSAWS and MISTIC-SL datasets, respectively.…”

“…Interestingly, [31] found that multi-granularity recognition models performed better because such models may be learning that certain activities only occur during specific phases and steps. Also, recent works on action triplet recognition in laparoscopic procedures focus on concurrent phase, step, and action recognition [36]. The poor performance of activity recognition models is a barrier to clinical applications, but understanding the relationship between granularity levels can address this challenge and guide model development.…”

Know your sensORs -- A Modality Study For Surgical Action Classification

“…It has been suggested that surgical gesture recognition can be learned from optical flow data alone, highlighting the importance of motion cues for action recognition [27]. Combining modalities from different sensors continues to be an active topic in the medical domain [13], as data stemming from various sources become ubiquitous in the OR. Fusion strategies for RGB-D data are being explored at length for many applications such as depth estimation [16], 6DoF pose estimation [26,3], and object classification [29].…”

“…The analysis of surgical videos is no longer limited to medical devices such as endoscopic cameras -in the past years, several works have explored the use of ceiling-mounted cameras in an effort to understand OR workflows from an outside perspective. As the amount of data stemming from OR sensors increases, new questions arise, such as how to best integrate various modalities into automated surgical systems [13] or where to optimally place cameras for specific tasks [11,18]. This study seeks to understand which camera modalities are best suited for surgical action recognition, exploring their relative performance in a unique set of multi-view surgical recordings.…”

AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy

“…Recently, several research teams have worked on developing dataset at large scales [1,3,31], but most are only designed and annotated for one certain task. In terms of clinical applicability, data from different modalities are needed to better understand the whole scenario, make proper decisions, as well as enrich perception with multi-task learning strategy [9,16]. Besides, there are few datasets designed for automation tasks in surgical application, among which the automatic laparoscopic field-of-view (FoV) control is a popular topic as it can liberate the assistant from such tedious manipulations with the help from surgical robots [5].…”