Multicentric exploration of tool annotation in robotic surgery: lessons learned when starting a surgical artificial intelligence project

Abstract

Background Artificial Intelligence (AI) holds tremendous potential to reduce surgical risks and improve surgical assessment. Machine Learning, a subfield of AI, relies on video and image data, where annotations provide veracity about the desired target features. Yet, methodological annotation explorations are limited to date. Here, we provide an exploratory analysis of the requirements and methods of instrument annotation in a multi-institutional team from two specialized AI centers and compile a structured manual for future AI projects focusing on instrument detection. Methods We developed a bottom-up approach for team annotation of robotic instruments in robotassisted partial nephrectomy (RAPN), after which it was validated in robot-assisted minimally invasive esophagectomy (RAMIE). Furthermore, instrument annotation methods were evaluated for their use in Machine Learning algorithms. Overall, we evaluated the efficiency and transferability of the proposed team approach and quantified performance metrics (e.g. time per frame required for each annotation modality) between RAPN and RAMIE. ResultsThe proposed annotation methodology was transferrable between both RAPN and RAMIE. The bottom-up approach of annotation management and training resulted in accurate annotations and demonstrated efficiency in annotating large datasets and diverse annotator groups. The average annotation time for RAPN for pixel annotation ranged from 4.49 to 12.6 minutes per image; for vector annotation this was decreased to 2.92 minutes. Similar ranges of pixel annotation times were denoted for RAMIE. Lastly, we elaborate on common pitfalls encountered throughout the annotation process. Conclusions We propose a successful bottom-up approach for annotator team composition, applicable to any annotation project. Our results set the foundation to start AI projects for instrument detection, segmentation and pose estimation. Due to the immense annotation burden resulting from spatial instrumental annotation, further analysis into sampling frequency and annotation detail needs to be conducted.

Surgical Phase Duration in Robot-Assisted Partial Nephrectomy: A Surgical Data Science Exploration for Clinical Relevance

“…This, in turn, enables larger patient cohort investigations, without the need for time-intensive manual annotations [29]. Furthermore, automated phase detection acts as an enabler toward fully automated surgical scene understanding [30], which, in turn, unlocks a myriad of other possible clinical applications [31]. Informed Consent Statement: Patient consent was waived due to full data anonymization (General Data Protection Regulation art.…”

First‐in‐human real‐time AI‐assisted instrument deocclusion during augmented reality robotic surgery

“…The binary segmentation data set contains 31,812 images on which all non‐organic items were manually delineated in the annotation platform SuperAnnotate (Sunnyvale, CA, USA) [ 11 ]. The 37 different non‐organic items include robotic and laparoscopic instruments, needles, wires, clips, vessel loops, bulldogs, gauzes etc.…”

The future implementation of artificial intelligence technology in esophageal surgery

“…Otherwise, the potential benefits of AI algorithms for surgical education and assessment could be undermined [24] . Furthermore, AI algorithms can act as decision-support tools during an operation, providing real-time analysis of data and offering recommendations to surgeons [25] . For novice surgeons, identifying crucial anatomical structures in the context of RAMIE remains a significant challenge.…”