Multi‐instance multi‐label learning for surgical image annotation

Abstract

Background Various techniques have been proposed in the literature for phase and tool recognition from laparoscopic videos. In comparison, research in multilabel annotation of still frames is limited. Methods We describe a framework for multilabel annotation of images extracted from laparoscopic cholecystectomy (LC) videos based on multi‐instance multiple‐label learning. The image is considered as a bag of features extracted from local regions after coarse segmentation. A method based on variational Bayesian gaussian mixture models (VBGMM) is proposed for bag representation. Three techniques based on different feature extraction and bag representation models are employed for comparison. Results Four anatomical structures (abdominal wall, gallbladder, fat, and liver bed) and a tool‐like object (specimen bag) were annotated in 482 images. Our method achieved the best performance on single label accuracy: 0.87 (highest) and 0.69 (lowest). Moreover, the performance was >20% higher in terms of four multilabel classification error metrics (one‐error, ranking‐loss, hamming‐loss, and coverage). Conclusions Our approach provides an accurate and efficient image representation for multilabel classification of still images captured in LC.

Computer-aided anatomy recognition in intrathoracic and -abdominal surgery: a systematic review

“…The majority of studies ( n = 20; 87%) used a retrospective study design, a prospective or mixed design was used in three studies [ 20 , 23 , 28 ]. The most frequently reported procedure was cholecystectomy ( n = 8; 35%) [ 10 , 11 , 16 , 19 , 23 , 26 , 27 , 29 ] followed by hysterectomy and other gynecological procedures ( n = 5; 22%)[ 20 , 24 , 30 – 32 ], nephrectomy ( n = 3; 13%) [ 25 , 28 , 33 ], abdominal laparoscopy (exact procedure not specified; n = 3; 13%) [ 17 , 34 , 35 ], hepatectomy ( n = 1; 4%) [ 36 ], lung cancer resection ( n = 1; 4%) [ 21 ], robot-assisted gastrectomy ( n = 1; 4%) [ 18 ], and transanal total mesorectal excision (TATME) ( n = 1; 4%)[ 22 ]. Five studies (22%) analyzed robot-assisted procedures [ 18 , 25 , 28 , 33 , 36 ], 18 studies (78%) used laparoscopic or thoracoscopic procedures [ 10 , 11 , 16 , 17 , 19 – 24 , 26 , 27 , 29 – 32 , 34 , 35 ].…”

“…Year of publication ranged from 2008 until 2021. Fourteen of the 23 studies (61%) were published in 2020 and 2021, and all used AI-based algorithms [10,11,[16][17][18][19][20][21][22][23][24][25][26][27]. The majority of studies (n = 20; 87%) used a retrospective study design, a prospective or mixed design was used in three studies [20,23,28].…”

Automated recognition of objects and types of forceps in surgical images using deep learning

“…In the field of surgery, AI-based decision support systems have provided a broad range of technological approaches to augment the information available to surgeons that have accelerated intraoperative pathology and surgical step recommendations 19 . Accurate and efficient object representation and segmentation are necessary for multilabel object classification in surgery based on the annotation of objects and frameworks 21 . Further, skill and motion assessments in surgical videos using CNN have been reported in recent years 22 – 24 .…”

Automated Recognition of Objects and Types of Forceps in Surgical Images Using Deep Learning

“…In the eld of surgery, AI-based decision support systems have provided a broad range of technological approaches to augment the information available to surgeons that have accelerated intraoperative pathology and surgical step recommendations 19 . Accurate and e cient object representation and segmentation are necessary for multilabel object classi cation in surgery based on the annotation of objects and frameworks [21]. Further, skill and motion assessments in surgical videos using CNN have been reported in recent years [22][23][24].…”