Deep learning for semantic segmentation of organs and tissues in laparoscopic surgery

Scheikl, Paul Maria; Laschewski, Stefan; Kisilenko, Anna; Davitashvili, Tornike; Müller, Benjamin; Capek, Manuela; Müller‐Stich, Beat P.; Wagner, Martin; Mathis-Ullrich, Franziska

doi:10.1515/cdbme-2020-0016

Cited by 26 publications

(30 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The hand movements of the user control the DT in virtual reality and are additionally projected to a robot arm in real-time. Modern machine learning frameworks may eventually be able to segment semantic information of endoscopic images as stated by Scheikl et al (2020) ; however, Ahmed and Devoto (2020) consider the modeling of tissue with deformations and movement as the main limitation of DT technologies in surgical robotics. Nevertheless DT-based assistance functions may very well be able to assist in training surgeons where the environment is well-defined by abstracted training pads.…”

Section: State Of the Artmentioning

confidence: 99%

A Digital Twin Approach for Contextual Assistance for Surgeons During Surgical Robotics Training

et al. 2021

View full text Add to dashboard Cite

Minimally invasive robotic surgery copes with some disadvantages for the surgeon of minimally invasive surgery while preserving the advantages for the patient. Most commercially available robotic systems are telemanipulated with haptic input devices. The exploitation of the haptics channel, e.g., by means of Virtual Fixtures, would allow for an individualized enhancement of surgical performance with contextual assistance. However, it remains an open field of research as it is non-trivial to estimate the task context itself during a surgery. In contrast, surgical training allows to abstract away from a real operation and thus makes it possible to model the task accurately. The presented approach exploits this fact to parameterize Virtual Fixtures during surgical training, proposing a Shared Control Parametrization Engine that retrieves procedural context information from a Digital Twin. This approach accelerates a proficient use of the robotic system for novice surgeons by augmenting the surgeon’s performance through haptic assistance. With this our aim is to reduce the required skill level and cognitive load of a surgeon performing minimally invasive robotic surgery. A pilot study is performed on the DLR MiroSurge system to evaluate the presented approach. The participants are tasked with two benchmark scenarios of surgical training. The execution of the benchmark scenarios requires basic skills as pick, place and path following. The evaluation of the pilot study shows the promising trend that novel users profit from the haptic augmentation during training of certain tasks.

show abstract

Section: State Of the Artmentioning

confidence: 99%

A Digital Twin Approach for Contextual Assistance for Surgeons During Surgical Robotics Training

et al. 2021

View full text Add to dashboard Cite

show abstract

“…[19,21]) or, more often, in the context of full scene segmentation (e.g. [5,31,39,55]). The data sets used differ highly in terms of annotation sparsity (e.g.…”

Section: Deep Learning-based Organ Segmentation On Rgb Datamentioning

confidence: 99%

“…Input to the models are video frames of varying size (e.g. 960 × 540 in [55], 512 × 512 in [38]). Relatively few works have tackled organ segmentation in open surgery, where, compared to minimally invasive surgery, image acquisition is often more difficult to realize and challenges arise from the even larger complexity and variability of the surgical scene [22].…”

Section: Deep Learning-based Organ Segmentation On Rgb Datamentioning

confidence: 99%

Robust deep learning-based semantic organ segmentation in hyperspectral images

Seidlitz,

Sellner,

Odenthal

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Semantic image segmentation is an important prerequisite for context-awareness and autonomous robotics in surgery. The state of the art has focused on conventional RGB video data acquired during minimally invasive surgery, but full-scene semantic segmentation based on spectral imaging data and obtained during open surgery has received almost no attention to date. To address this gap in the literature, we are investigating the following research questions based on hyperspectral imaging (HSI) data of pigs acquired in an open surgery setting: (1) What is an adequate representation of HSI data for neural network-based fully automated organ segmentation, especially with respect to the spatial granularity of the data (pixels vs. superpixels vs. patches vs. full images)? ( 2) Is there a benefit of using HSI data compared to other modalities, namely RGB data and processed HSI data (e.g. tissue parameters like oxygenation), when performing semantic organ segmentation? According to a comprehensive validation study based on 506 HSI images from 20 pigs, annotated with a total of 19 classes, deep learning-based segmentation performance increases -consistently across modalities -with the spatial context of the input data. Unprocessed HSI data offers an advantage over RGB data or processed data from the camera provider, with the advantage increasing with decreasing size of the input to the neural network. Maximum performance (HSI applied to whole images) yielded a mean dice similarity coefficient (DSC) of 0.89 (standard deviation (SD) 0.04), which is in the range of the inter-rater variability (DSC of 0.89 (SD 0.07)). We conclude that HSI could become a powerful image modality for fully-automatic surgical scene understanding with many advantages over traditional imaging, including the ability to recover additional functional tissue information.

show abstract

“…Nur eine ausgereifte Wahrnehmung der Umgebung (z. B. semantische Segmentierung von chirurgischen Werkzeugen und Organen in Endoskopbildern [ 19 ]) bildet die Basis für sichere autonome Chirurgie- und Endoskopieroboter (Abb. 5 ).…”

Section: Perspektive: Kognitive Assistenzrobotikunclassified

Robotik im Operationssaal – (Ko‑)Operieren mit Kollege Roboter

Mathis-Ullrich

Scheikl

2020

Gastroenterologe

View full text Add to dashboard Cite

Hintergrund Medizinrobotik birgt das Potenzial, chirurgische oder endoluminale Eingriffe zu verbessern, indem diese Technologie hochpräzise Bewegungen und übermenschliche Perzeption ermöglicht. Ziel der Arbeit Darstellung historischer, existierender und zukünftiger robotischer Assistenten sowie Herausstellung deren Auswirkungen auf die robotische Chirurgie und Endoskopie. Methoden Insbesondere werden historische Medizinroboter sowie konventionelle Telemanipulatoren vorgestellt und diese mit minimal-invasiven Kontinuumsrobotern und neuartigen robotischen Konzepten aus Praxis und Forschung verglichen. Zusätzlich wird eine Perspektive für die zukünftige Generation von Chirurgie- und Endoskopierobotern geboten. Schlussfolgerung Roboterassistierte Medizin bietet einen großen Mehrwert für die Qualität der Intervention sowie Sicherheit für Chirurgen und Patientinnen. Zukünftig werden vermehrt Prozessschritte (teil‑)autonom in Kooperation mit Experten durchgeführt.

show abstract

Deep learning for semantic segmentation of organs and tissues in laparoscopic surgery

Cited by 26 publications

References 23 publications

A Digital Twin Approach for Contextual Assistance for Surgeons During Surgical Robotics Training

A Digital Twin Approach for Contextual Assistance for Surgeons During Surgical Robotics Training

Robust deep learning-based semantic organ segmentation in hyperspectral images

Robotik im Operationssaal – (Ko‑)Operieren mit Kollege Roboter

Contact Info

Product

Resources

About