A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

Abstract

Computer vision has been utilized to analyze mini- mally invasive surgery videos and aid with polyp detec- tion, tool localization, and organ 3D modelling tasks. However, irregular light patterns such as saturation, specular highlights, or extreme contrasts occlude texture and hinder these tasks. In this work, specular highlights were removed and the occluded data was reconstructed. To do that, an unsupervised temporal generative ad- versarial network (GAN) was used to inpaint specular highlights spatially and temporally. Due to the absence of a dataset with ground truth occluded textures, the network was trained on the in-vivo gastric endoscopy dataset (Hyper-Kvasir [1]) with specular highlight masks that were automatically created and processed to act as pseudo ground truths. Ablation studies and direct comparison with other methods were used to show the improved results of our system. In addition, the results on various datasets show the generalizability of our net- work on different environments and procedures as shown in Fig. 1. Finally, experiments also show the positive effect of inpainting on other computer vision tasks under the umbrella of 3D reconstruction and localization in endoscopy including feature matching as well as optical flow and disparity estimation.

The applications of Generative Adversarial Network in surgical videos

“…Compared with the STTN model, the training method based on pseudo dataset can generate images with more real details, which has a positive effect on image filling [13]. The PSNR value of this model is 29.542, which is higher than that of STTN model (28.683), but the advantage is not obvious.…”

“…The latest method used at present is to eliminate the reflection of the endoscope in the video by removing the highlights in the video stream and then automatically filling the exact image according to the training results of the model [13]. The system generates "a pseudo ground truth dataset" by separating part of Hyper-Kvasir's video data, which is used for feature learning of the STTN system to make the image easily filled more real.…”

SpecReFlow: an algorithm for specular reflection restoration using flow-guided video completion

“…The result of these lighting aberrations is seen as temporal inconsistency and color flickering 23 , 24 . Recent efforts, such as those by Daher et al., 25 have tried to harness temporal data using traditional nonlearning methods to detect SR regions. However, their limited accuracy in SR detection causes temporal-aware generative adversarial models to unintentionally include undetected SR from other regions, resulting in noisy reconstructions.…”

Development of endoscopic SLAM and the challenges

“…A weakly supervised approach employed a two-stage network to detect and suppress specular highlights on laparoscopy endometriosis images using U-Net architecture [13]. In [14], they fine-tuned a Spatial-Temporal Transformer Network with pseudo-groundtruth of specularity masks, achieving an unsupervised approach to remove specular highlight on gastric videos.…”