CholecTriplet2021: A benchmark challenge for surgical action triplet recognition

Abstract

Context-aware decision support in the operating room can foster surgical safety and efficiency by leveraging real-time feedback from surgical workflow analysis. Most existing works recognize surgical activities at a coarse-grained level, such as phases, steps or events, leaving out fine-grained interaction details about the surgical activity; yet those are needed for more helpful AI assistance in the operating room. Recognizing surgical actions as triplets of instrument, verb, target combination delivers comprehensive details about the activities taking place in surgical videos. This paper presents CholecTriplet2021: an endoscopic vision challenge organized at MICCAI 2021 for the recognition of surgical action triplets in laparoscopic videos. The challenge granted private access to the large-scale CholecT50 dataset, which is annotated with action triplet information. In this paper, we present the challenge setup and assessment of the stateof-the-art deep learning methods proposed by the participants during the challenge. A total of 4 baseline methods from the challenge organizers and 19 new deep learning algorithms by competing teams are presented to recognize surgical action triplets directly from surgical videos, achieving mean average precision (mAP) ranging from 4.2% to 38.1%. This study also analyzes the significance of the results obtained by the presented approaches, performs a thorough methodological comparison between them, in-depth result analysis, and proposes a novel ensemble method for enhanced recognition. Our analysis shows that surgical workflow analysis is not yet solved, and also highlights interesting directions for future research on fine-grained surgical activity recognition which is of utmost importance for the development of AI in surgery.

CholecTriplet2022: Show me a tool and tell me the triplet - an endoscopic vision challenge for surgical action triplet detection

“…Of these teams, a total of 11 teams progressed beyond the validation phase; as detailed in Fig 2, these teams were drawn from 9 different countries and 3 continents. This final participation was just over half of the CholecTriplet 2021 challenge (Nwoye et al, 2022a); we also observed a drop in the relative ratio between the final participants and total registrants (45.5% in 2021; 34.4% in 2022). We identified 2 likely reasons for the drop in each of these statistics:…”

“…In 2021, we introduce the first endoscopic vision challenge focusing on the recognition of surgical activities in the form of triplets (Nwoye et al, 2022a). A rerun of this challenge in the current edition is with two notable advances: (1) the increment of the task difficulty from presence to spatial detection, which also increases the task usefulness, and (2) the addition of spatial bounding box labels to challenge evaluation data thereby enriching the dataset.…”

“…At the evaluation, submitted Docker containers are executed on the challenge test set producing a JSON file per video containing the outputs for the three sub-tasks. Triplet recognition evaluation is based on the Y IVT probability scores for valid 94 out of 100 triplet classes ignoring the last 6 labels with at least one NULL component in the triplet composition, as done in the previous challenge (Nwoye et al, 2022a). Model's direct outputs for any component of the triplet are not considered, instead, the predictions for instruments, verbs, and targets are filtered from Y IVT using Nwoye et al (2022b)'s disentanglement function for uniformity across all architectural designs.…”

“…We equally analyze the quality of the model predictions by computing the topK accuracy where K ∈ [5,10,15,20] and the average over topK@[5:20] at intervals of 5 steps as done in Nwoye et al (2022a).…”

“…The quest for a better and more comprehensive activity modeling led to the formalization of surgical activities as triplets (Nwoye, 2021;Katic et al, 2014Katic et al, , 2015 taking into account the details of the operating instruments, manipulated tissues/targets, and verbs of action describing the interaction. Earlier work on surgical action triplet recognition (Nwoye et al, 2020(Nwoye et al, , 2022bXi et al, 2022;Li et al, 2022;Cheng et al, 2022) including our previous CholecTriplet2021 challenge (Nwoye et al, 2022a) concentrate primarily on the presence detection of these triplets from laparoscopic videos. Localizing also the spatial regions of action triplets across video frames would advance research on tool-tissue interaction understanding and help develop assistance systems delivering valuable AI feedback and automated warnings.…”