Zero-Shot Super-Resolution With a Physically-Motivated Downsampling Kernel for Endomicroscopy

Abstract

Super-resolution (SR) methods have seen significant advances thanks to the development of convolutional neural networks (CNNs). CNNs have been successfully employed to improve the quality of endomicroscopy imaging. Yet, the inherent limitation of research on SR in endomicroscopy remains the lack of ground truth highresolution (HR) images, commonly used for both supervised training and reference-based image quality assessment (IQA). Therefore, alternative methods, such as unsupervised SR are being explored. To address the need for non-reference image quality improvement, we designed a novel zero-shot super-resolution (ZSSR) approach that relies only on the endomicroscopy data to be processed in a self-supervised manner without the need for groundtruth HR images. We tailored the proposed pipeline to the idiosyncrasies of endomicroscopy by introducing both: a physically-motivated Voronoi downscaling kernel accounting for the endomicroscope's irregular fibre-based sampling pattern, and realistic noise patterns. We also took advantage of video sequences to exploit a sequence of images for self-supervised zero-shot image quality improvement. We run ablation studies to assess our contribution in regards to the downscaling kernel and noise simulation. We validate our methodology on both synthetic and original data. Synthetic experiments were assessed with reference-based IQA, while our results for original images were evaluated in a user study conducted with both expert and non-expert observers. The results demonstrated superior performance in image quality of ZSSR reconstructions in comparison to the baseline method. The ZSSR is also competitive when compared to supervised single-image SR, especially being the preferred reconstruction technique by experts.

A Transcutaneous Fetal Visual Stimulator

“…This resolution does not match with the stimulation file, therefore, the data has to be resized to fit the TCFVS display. Downscaling techniques can be used to resize an image to a suitable size, thus providing significant capabilities in several applications, for instance, microscopy, remote sensing, soil moisture mapping, and so on [19]- [21]. However, these algorithms result in information loss and surely, in our context, can alter the Pattern coordinates [22].…”

Deep-learning-based image super-resolution of an end-expandable optical fiber probe for application in esophageal cancer diagnostics

“…To address the need for non-reference image quality (IQ) improvement, Szczotka et al. designed a zero-shot super-resolution approach with a physically-motivated downsampling Voronoi kernel tailored to the acquisition physics by incorporating fiber bundle geometry and noise simulation 11 . More recently, a multi-frame super-resolution algorithm exploiting bundle rotation to extract features and reconstruct underlying tissue is developed by Eadie et al 12 .…”

“…designed a zero-shot super-resolution approach with a physically-motivated downsampling Voronoi kernel tailored to the acquisition physics by incorporating fiber bundle geometry and noise simulation. 11 More recently, a multi-frame super-resolution algorithm exploiting bundle rotation to extract features and reconstruct underlying tissue is developed by Eadie et al. 12 Motivated by the advances of image super-resolution using deep learning approaches, 8 , 10 – 16 here we proposed a deep-learning-based image super-resolution (DL-SR) method that estimates a HRME image from its acquired low-resolution (LR) counterpart.…”

“…10 To address the need for non-reference image quality (IQ) improvement, Szczotka et al designed a zero-shot super-resolution approach with a physically-motivated downsampling Voronoi kernel tailored to the acquisition physics by incorporating fiber bundle geometry and noise simulation. 11 More recently, a multi-frame super-resolution algorithm exploiting bundle rotation to extract features and reconstruct underlying tissue is developed by Eadie et al 12 Motivated by the advances of image super-resolution using deep learning approaches, 8,[10][11][12][13][14][15][16] here we proposed a deep-learning-based image super-resolution (DL-SR) method that estimates a HRME image from its acquired low-resolution (LR) counterpart. In this way, sparse images can be acquired by the end-expandable optical fiber probe to increase the FOV at cost of spatial resolution, which could be subsequently restored by use of a DL-SR method to enhance IQ for diagnostic purposes.…”

Context-aware dynamic filtering network for confocal laser endomicroscopy image denoising

“…Unfortunately, acquiring high-resolution images is challenging due to the unavailability of the high-magnification CLE fiber bundle probe. Some unsupervised SISR methods (Szczotka et al 2021) are available for ground-truth-free SR reconstruction for CLE images. On the other hand, Butola et al (2020) proposed Discrete cosine transform (DCT) and total variation (TV) denoising based method to remove inherent honeycomb structure of CLE and improve the quality of images.…”