Surgical tool segmentation using a hybrid deep CNN-RNN auto encoder-decoder

Attia, Mohamed; Hossny, Mohammed; Nahavandi, Saeid; Asadi, Hamed

doi:10.1109/smc.2017.8123151

Cited by 68 publications

(33 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These tasks can be categorized according the precision of the desired outputs. The finest task is tool segmentation (Bodenstedt et al, 2016 Feb-Mar;García-Peraza-Herrera et al, 2016Attia et al, 2017;Lee et al, 2017b;Zhou et al, 2017;Ross et al, 2018;Su et al, 2018). This includes multi-label tool segmentation for articulated tools (Laina et al, 2017): each tool part is associated with one label.…”

Section: Computer Vision Tasksmentioning

confidence: 99%

“…The use of region proposal networks was also investigated (Sarikaya et al, 2017;Jin et al, 2018). Several CNN architectures were experimented for tool segmentation: fully convolutional networks (García-Peraza-Herrera et al, 2016;Zhou et al, 2017), U-net (Ross et al, 2018) or custom encoder/decoder CNN architectures (García-Peraza-Herrera et al, 2017;Attia et al, 2017;Laina et al, 2017). The use of generative adversarial networks was proposed to train or pre-train segmentation CNNs: a tool segmentation CNN (Ross et al, 2018) and a specular highlight segmentation and removal CNN (Funke et al, 2018).…”

Section: Computer Vision Algorithmsmentioning

confidence: 99%

“…Note that multi-task CNNs have been designed: Laina et al (2017) jointly segments the tools and detects the joints between tool parts, Kurmann et al (2017) jointly recognizes the tools and detects the joints between tool parts, Du et al (2018) jointly detects the joints between tool parts and estimates 2-D poses, Jin et al (2018) and Hu et al (2017) jointly determines tool presence and tool localization. To take time information into account, proposed solutions sometimes took advantage of the optical flow (Czajkowska et al, 2018) or relied on temporal filtering techniques, such as a Kalman filter (Ryu et al, 2018) or a recurrent neural network (RNN) (Attia et al, 2017;Marban et al, 2017). This is typically useful for tool tracking (Chen et al, 2017;Marban et al, 2017;Ryu et al, 2018;Czajkowska et al, 2018), but it was also used to speed up tool segmentation (García-Peraza-Herrera et al, 2016) or to improve tool presence detection (Al Hajj et al, 2017a).…”

Section: Computer Vision Algorithmsmentioning

confidence: 99%

See 2 more Smart Citations

CATARACTS: Challenge on automatic tool annotation for cataRACT surgery

Hajj¹,

Lamard²,

Conze³

et al. 2019

Medical Image Analysis

View full text Add to dashboard Cite

Surgical tool detection is attracting increasing attention from the medical image analysis community. The goal generally is not to precisely locate tools in images, but rather to indicate which tools are being used by the surgeon at each instant. The main motivation for annotating tool usage is to design efficient solutions for surgical workflow analysis, with potential applications in report generation, surgical training and even real-time decision support. Most existing tool annotation algorithms focus on laparoscopic surgeries. However, with 19 million interventions per year, the most common surgical procedure in the world is cataract surgery. The CATARACTS challenge was organized in 2017 to evaluate tool annotation algorithms in the specific context of cataract surgery. It relies on more than nine hours of videos, from 50 cataract surgeries, in which the presence of 21 surgical tools was manually annotated by two experts. With 14 participating teams, this challenge can be considered a success. As might be expected, the submitted solutions are based on deep learning. This paper thoroughly evaluates these solutions: in particular, the quality of their annotations are compared to that of human interpretations. Next, lessons learnt from the differential analysis of these solutions are discussed. We expect that they will guide the design of efficient surgery monitoring tools in the near future.

show abstract

Section: Computer Vision Tasksmentioning

confidence: 99%

Section: Computer Vision Algorithmsmentioning

confidence: 99%

Section: Computer Vision Algorithmsmentioning

confidence: 99%

See 1 more Smart Citation

CATARACTS: Challenge on automatic tool annotation for cataRACT surgery

Hajj¹,

Lamard²,

Conze³

et al. 2019

Medical Image Analysis

View full text Add to dashboard Cite

show abstract

“…Understand not only the presence but also the position gives great improvement in phase recognition. This can evolve in tracking, 35,36 and segmentation, 37‐41 a technique to define the contours of all the different tools in the current frame, shown in Figure 4C. All these discussed studies: Used supervised learning; conversely, in References 42 and 43, the authors applied unsupervised learning to assign phases to specific frame and avoid the time‐consuming labelling phase. Focused on endoscopic videos; same techniques have been applied to external cataracts video, in References 44 and 45.…”

Section: Literature Reviewmentioning

confidence: 99%

Intraoperative surgery room management: A deep learning perspective

Tanzi

Piazzolla

Vezzetti

2020

Robotics Computer Surgery

View full text Add to dashboard Cite

Purpose: The current study aimed to systematically review the literature addressing the use of deep learning (DL) methods in intraoperative surgery applications, focusing on the data collection, the objectives of these tools and, more technically, the DLbased paradigms utilized. Methods: A literature search with classic databases was performed: we identified, with the use of specific keywords, a total of 996 papers. Among them, we selected 52 for effective analysis, focusing on articles published after January 2015. Results: The preliminary results of the implementation of DL in clinical setting are encouraging. Almost all the surgery sub-fields have seen the advent of artificial intelligence (AI) applications and the results outperformed the previous techniques in the majority of the cases. From these results, a conceptualization of an intelligent operating room (IOR) is also presented. Conclusion: This evaluation outlined how AI and, in particular, DL are revolutionizing the surgery field, with numerous applications, such as context detection and room management. This process is evolving years by years into the realization of an IOR, equipped with technologies perfectly suited to drastically improve the surgical workflow.

show abstract

“…An early implementation proposed by Haase et al [7] augments photometric information by range data, exploiting two modalities to create a 3D segmentation of the instruments. Other publications driving the research include Attia et al [8], Pakhomov et al [9], Garcia-Peraza-Herrera et al [10], and Shvets et al [11]. All implementations make use of CNN models.…”

Section: Related Workmentioning

confidence: 99%

DL-based segmentation of endoscopic scenes for mitral valve repair

Ivantsits

Tautz

Sündermann

et al. 2020

Current Directions in Biomedical Engineering

View full text Add to dashboard Cite

Minimally invasive surgery is increasingly utilized for mitral valve repair and replacement. The intervention is performed with an endoscopic field of view on the arrested heart. Extracting the necessary information from the live endoscopic video stream is challenging due to the moving camera position, the high variability of defects, and occlusion of structures by instruments. During such minimally invasive interventions there is no time to segment regions of interest manually. We propose a real-time-capable deep-learning-based approach to detect and segment the relevant anatomical structures and instruments. For the universal deployment of the proposed solution, we evaluate them on pixel accuracy as well as distance measurements of the detected contours. The U-Net, Google’s DeepLab v3, and the Obelisk-Net models are cross-validated, with DeepLab showing superior results in pixel accuracy and distance measurements.

show abstract

Surgical tool segmentation using a hybrid deep CNN-RNN auto encoder-decoder

Cited by 68 publications

References 14 publications

CATARACTS: Challenge on automatic tool annotation for cataRACT surgery

CATARACTS: Challenge on automatic tool annotation for cataRACT surgery

Intraoperative surgery room management: A deep learning perspective

DL-based segmentation of endoscopic scenes for mitral valve repair

Contact Info

Product

Resources

About