SERV-CT: A disparity dataset from cone-beam CT for validation of endoscopic 3D reconstruction

Edwards, Philip; Psychogyios, Dimitris; Speidel, Stefanie; Maier-Hein, Lena; Stoyanov, Danail

doi:10.1016/j.media.2021.102302

Cited by 39 publications

(26 citation statements)

References 43 publications

(41 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In CAI, until recently data scarcity limited stereo-based 3D reconstruction to handcrafted approaches [ 19 ], [ 20 ], [ 21 ] with datasets [ 22 ], [ 23 ], [ 24 ] designed for validation. Despite promising unsupervised disparity estimation methods [ 25 ], [ 26 ], [ 27 ] and methods attempting domain transfer from nonsurgical training data [ 28 ] practical performance deteriorates in surgical scenes.…”

Section: Related Workmentioning

confidence: 99%

MSDESIS: Multitask Stereo Disparity Estimation and Surgical Instrument Segmentation

Psychogyios

Mazomenos

Vasconcelos

et al. 2022

IEEE Trans. Med. Imaging

Self Cite

View full text Add to dashboard Cite

Reconstructing the 3D geometry of the surgical site and detecting instruments within it are important tasks for surgical navigation systems and robotic surgery automation. Traditional approaches treat each problem in isolation and do not account for the intrinsic relationship between segmentation and stereo matching. In this paper, we present a learning-based framework that jointly estimates disparity and binary tool segmentation masks. The core component of our architecture is a shared feature encoder which allows strong interaction between the aforementioned tasks. Experimentally, we train two variants of our network with different capacities and explore different training schemes including both multi-task and single-task learning. Our results show that supervising the segmentation task improves our network's disparity estimation accuracy. We demonstrate a domain adaptation scheme where we supervise the segmentation task with monocular data and achieve domain adaptation of the adjacent disparity task, reducing disparity End-Point-Error and depth mean absolute error by 77.73% and 61.73% respectively compared to the pre-trained baseline model. Our best overall multi-task model, trained with both disparity and segmentation data in subsequent phases, achieves 89.15% mean Intersection-over-Union in RIS and 3.18 millimetre depth mean absolute error in SCARED test sets. Our proposed multi-task architecture is real-time, able to process (1280x1024) stereo input and simultaneously estimate disparity maps and segmentation masks at 22 frames per second. The model code and pre-trained models are made available: https://github.com/dimitrisPs/msdesis

show abstract

Section: Related Workmentioning

confidence: 99%

MSDESIS: Multitask Stereo Disparity Estimation and Surgical Instrument Segmentation

Psychogyios

Mazomenos

Vasconcelos

et al. 2022

IEEE Trans. Med. Imaging

Self Cite

View full text Add to dashboard Cite

show abstract

“…All the in-vivo stereo images (34 overall) were rectified, undistorted, calibrated, and vertically aligned with the provided intrinsic and extrinsic parameters. The ex-vivo data set SERV-CT [33] was also obtained from the classic da Vinci Surgical System. SERV-CT contains 16 stereo endoscopic image pairs with reference anatomical segmentation derived from CT. Two different ex-vivo porcine samples were imaged using the straight and 30 • endoscopes.…”

Section: A Baseline Algorithms Data Sets and Metricsmentioning

confidence: 99%

“…In the robotic community, "true value" or "reference value" obtained from hardware/software with higher precision is often termed as "ground truth". Following[33], we name it as "reference" considering its precision inaccuracies. The rest content, figures, and tables use "reference" instead of "ground truth".…”

mentioning

confidence: 99%

BDIS: Bayesian Dense Inverse Searching Method for Real-Time Stereo Surgical Image Matching

Song¹,

Zhu²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

In stereoscope-based Minimally Invasive Surgeries (MIS), dense stereo matching plays an indispensable role in 3D shape recovery, AR, VR, and navigation tasks. Although numerous Deep Neural Network (DNN) approaches are proposed, the conventional prior-free approaches are still popular in the industry because of the lack of open-source annotated data set and the limitation of the task-specific pre-trained DNNs. Among the prior-free stereo matching algorithms, there is no successful real-time algorithm in none GPU environment for MIS. This paper proposes the first CPU-level real-time priorfree stereo matching algorithm for general MIS tasks. We achieve an average 17 Hz on 640 × 480 images with a singlecore CPU (i5-9400) for surgical images. Meanwhile, it achieves slightly better accuracy than the popular ELAS. The patchbased fast disparity searching algorithm is adopted for the rectified stereo images. A coarse-to-fine Bayesian probability and a spatial Gaussian mixed model were proposed to evaluate the patch probability at different scales. An optional probability density function estimation algorithm was adopted to quantify the prediction variance. Extensive experiments demonstrated the proposed method's capability to handle ambiguities introduced by the textureless surfaces and the photometric inconsistency from the non-Lambertian reflectance and dark illumination. The estimated probability managed to balance the confidences of the patches for stereo images at different scales. It has similar or higher accuracy and fewer outliers than the baseline ELAS in MIS, while it is 4-5 times faster. The code and the synthetic data sets are available at https://github.com/JingweiSong/BDIS-v2.

show abstract

“…In this paper we analyse the effect of specularity inpainting both qualitatively and quantitatively on stereo disparity, optical flow, feature matching, and camera motion estimation. The disparity evaluation uses the ex-vivo porcine data with depth ground truth (SERV-CT dataset (Edwards et al, 2020)). Effects on optical flow and feature matching are evaluated by using them for relative camera motion estimation, which can be compared against robot kinematics ground truth on the SCARED dataset (Allan et al, 2021).…”

Section: Specular Highlight Removal Effect On Computer Vision Tasks I...mentioning

confidence: 99%

“…This is performed to assess the model's effect on datasets with available 3D reconstruction ground truth. The tested datasets include in-vivo GI endoscopic data from the publicly available Hyper-Kvasir dataset (Borgli et al, 2020), colonoscopic data from a private dataset, and laparoscopic exvivo porcine data (SERV-CT (Edwards et al, 2020), SCARED (Allan et al, 2021)). Some results on the tested datasets are shown in 1.…”

Section: Introductionmentioning

confidence: 99%

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

Daher¹,

Vasconcelos²,

Stoyanov³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Video streams are utilised to guide minimally-invasive surgery and diagnostic procedures in a wide range of procedures, and many computer assisted techniques have been developed to automatically analyse them. These approaches can provide additional information to the surgeon such as lesion detection, instrument navigation, or anatomy 3D shape modeling. However, the necessary image features to recognise these patterns are not always reliably detected due to the presence of irregular light patterns such as specular highlight reflections. In this paper, we aim at removing specular highlights from endoscopic videos using machine learning. We propose using a temporal generative adversarial network (GAN) to inpaint the hidden anatomy under specularities, inferring its appearance spatially and from neighbouring frames where they are not present in the same location. This is achieved using in-vivo data of gastric endoscopy (Hyper-Kvasir) in a fully unsupervised manner that relies on automatic detection of specular highlights. System evaluations show significant improvements to traditional methods through direct comparison as well as other machine learning techniques through an ablation study that depicts the importance of the network's temporal and transfer learning components. The generalizability of our system to different surgical setups and procedures was also evaluated qualitatively on in-vivo data of gastric endoscopy and ex-vivo porcine data (SERV-CT, SCARED). We also assess the effect of our method in computer vision tasks that underpin 3D reconstruction and camera motion estimation, namely stereo disparity, optical flow, and sparse point feature matching. These are evaluated quantitatively and qualitatively and results show a positive effect of specular highlight inpainting on these tasks in a novel comprehensive analysis.

show abstract

SERV-CT: A disparity dataset from cone-beam CT for validation of endoscopic 3D reconstruction

Cited by 39 publications

References 43 publications

MSDESIS: Multitask Stereo Disparity Estimation and Surgical Instrument Segmentation

MSDESIS: Multitask Stereo Disparity Estimation and Surgical Instrument Segmentation

BDIS: Bayesian Dense Inverse Searching Method for Real-Time Stereo Surgical Image Matching

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

Contact Info

Product

Resources

About