Self-Supervised Learning for Monocular Depth Estimation on Minimally Invasive Surgery Scenes

Shao, Shuwei; Pei, Zhongcai; Chen, Weihai; Zhang, Baochang; Wu, Xiaofeng; Sun, Dianmin; Doermann, David

doi:10.1109/icra48506.2021.9561508

Cited by 8 publications

(6 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ozyoruk et al [1] developed an affine brightness transformer to align the inter-frame brightness condition, and a spatial attention module to encourage the network to emphasize on highly textured regions. Shao et al [17] introduced appearance flow to take into variations in brightness patterns. In addition, they adopted a feature scaling module to mitigate the inadequate representation learning issue induced by low and homogeneous textures.…”

Section: Related Work a Monocular Depth Estimationmentioning

confidence: 99%

“…In addition, they adopted a feature scaling module to mitigate the inadequate representation learning issue induced by low and homogeneous textures. Nevertheless, the attention-based manners as have been done for [1] and [17] do not fundamentally resolve the problem of poor discriminative ability of photometric error in low-texture and homogeneoustexture regions, but rather refines the feature representation on the basis of feature statistics, hence the performance gain is limited. Later in [7], Shao et al generalized the brightness constancy assumption to a dynamic image constraint, allowing for a comprehensive information representation from frame to frame, and designed an automatic registration step to enhance the derivation of appearance flow.…”

Section: Related Work a Monocular Depth Estimationmentioning

confidence: 99%

“…SAM: SPATIAL ATTENTION MODULE[1]; FSM: FEATURE SCALING MODULE[17]; PM[9]: PATCHMATCH MODULE; LPM: LEARNABLE PATCHMATCH MODULE.…”

mentioning

confidence: 99%

See 2 more Smart Citations

SMUDLP: Self-Teaching Multi-Frame Unsupervised Endoscopic Depth Estimation with Learnable Patchmatch

Shao¹,

Pei²,

Chen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Unsupervised monocular trained depth estimation models make use of adjacent frames as a supervisory signal during the training phase. However, temporally correlated frames are also available at inference time for many clinical applications, e.g., surgical navigation. The vast majority of monocular systems do not exploit this valuable signal that could be deployed to enhance the depth estimates. Those that do, achieve only limited gains due to the unique challenges in endoscopic scenes, such as low and homogeneous textures and inter-frame brightness fluctuations. In this work, we present SMUDLP, a novel and unsupervised paradigm for multi-frame monocular endoscopic depth estimation. The SMUDLP integrates a learnable patchmatch module to adaptively increase the discriminative ability in low-texture and homogeneous-texture regions, and enforces cross-teaching and self-teaching consistencies to provide efficacious regularizations towards brightness fluctuations. Our detailed experiments on both SCARED and Hamlyn datasets indicate that the SMUDLP exceeds state-of-the-art competitors by a large margin, including those that use single or multiple frames at inference time. The source code and trained models will be publicly available upon the acceptance.

show abstract

Section: Related Work a Monocular Depth Estimationmentioning

confidence: 99%

Section: Related Work a Monocular Depth Estimationmentioning

confidence: 99%

See 1 more Smart Citation

SMUDLP: Self-Teaching Multi-Frame Unsupervised Endoscopic Depth Estimation with Learnable Patchmatch

Shao¹,

Pei²,

Chen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…gation [16,45,37,13], robotics [31,11,40] and augmented reality [10,29]. Current methods for training CNNbased Monocular Depth Estimators (MDE) include two major methodologies: full-supervision and self-supervision.…”

Section: Introductionmentioning

confidence: 99%

One scalar is all you need -- absolute depth estimation using monocular self-supervision

Dana¹,

Carmel²,

Shomer³

et al. 2023

Preprint

View full text Add to dashboard Cite

Self-supervised monocular depth estimators can be trained or fine-tuned on new scenes using only images and no ground-truth depth data, achieving good accuracy. However, these estimators suffer from the inherent ambiguity of the depth scale, significantly limiting their applicability. In this work, we present a method for transferring the depth-scale from existing source datasets collected with ground-truth depths to depth estimators that are trained using self-supervision on a newly collected target dataset consisting of images only, solving a significant limiting factor.We show that self-supervision based on projective geometry results in predicted depths that are linearly correlated with their ground-truth depths. Moreover, the linearity of this relationship also holds when jointly training on images from two different (real or synthetic) source and target domains. We utilize this observed property and model the relationship between the ground-truth and the predicted up-to-scale depths of images from the source domain using a single global scalar. Then, we scale the predicted up-to-scale depths of images from the target domain using the estimated global scaling factor, performing depth-scale transfer between the two domains. This suggested method was evaluated on the target KITTI and DDAD datasets, while using other real or synthetic source datasets, that have a larger field-of-view, other image style or structural content. Our approach achieves competitive accuracy on KITTI, even without using the specially tailored vKITTI or vKITTI2 datasets, and higher accuracy on DDAD, when using both real or synthetic source datasets.

show abstract

“…Estimating depth map from one single RGB image has been a longstanding research topic and is critical for a variety of applications e.g., scene understanding [19], autonomous driving [34], augmented reality [33], and minimally invasive surgery [49]. Saxena et al [48] introduced one of the first learning-based studies in this area.…”

Section: Introductionmentioning

confidence: 99%

NENet: Monocular Depth Estimation via Neural Ensembles

Shao¹,

Li²,

Pei³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Depth estimation is getting a widespread popularity in the computer vision community, and it is still quite difficult to recover an accurate depth map using only one single RGB image. In this work, we observe a phenomenon that existing methods tend to exhibit asymmetric errors, which might open up a new direction for accurate and robust depth estimation. We carefully investigate into the phenomenon, and construct a two-level ensemble scheme, NENet, to integrate multiple predictions from diverse base predictors. The NENet forms a more reliable depth estimator, which substantially boosts the performance over base predictors. Notably, this is the first attempt to introduce ensemble learning and evaluate its utility for monocular depth estimation to the best of our knowledge. Extensive experiments demonstrate that the proposed NENet achieves better results than previous state-of-the-art approaches on the NYU-Depth-v2 and KITTI datasets. In particular, our method improves previous state-of-the-art methods from 0.365 to 0.349 on the metric RMSE on the NYU dataset. To validate the generalizability across cameras, we directly apply the models trained on the NYU dataset to the SUN RGB-D dataset without any fine-tuning, and achieve the superior results, which indicate its strong generalizability. The source code and trained models will be publicly available upon the acceptance.

show abstract

Self-Supervised Learning for Monocular Depth Estimation on Minimally Invasive Surgery Scenes

Cited by 8 publications

References 48 publications

SMUDLP: Self-Teaching Multi-Frame Unsupervised Endoscopic Depth Estimation with Learnable Patchmatch

SMUDLP: Self-Teaching Multi-Frame Unsupervised Endoscopic Depth Estimation with Learnable Patchmatch

One scalar is all you need -- absolute depth estimation using monocular self-supervision

NENet: Monocular Depth Estimation via Neural Ensembles

Contact Info

Product

Resources

About