Endoscopy image enhancement method by generalized imaging defect models based adversarial training

Abstract

Objective: Smoke, uneven lighting, and color deviation are common issues in endoscopic surgery, which have increased the risk of surgery and even lead to failure. Approach: In this study, we present a new physics model driven semi-supervised learning framework for high-quality pixel-wise endoscopic image enhancement, which is generalizable for smoke removal, light adjustment, and color correction. To improve the authenticity of the generated images, and thereby improve the network performance, we integrated specific physical imaging defect models with the CycleGAN framework. No ground-truth data in pairs are required. In addition, we propose a transfer learning framework to address the data scarcity in several endoscope enhancement tasks and improve the network performance. Main results: Qualitative and quantitative studies reveal that the proposed network outperforms the state-of-the-art image enhancement methods. In particular, the proposed method performs much better than the original CycleGAN, for example, the structural similarity (SSIM) improved from 0.7925 to 0.8648, feature similarity for color images (FSIMc) from 0.8917 to 0.9283, and quaternion structural similarity (QSSIM) from 0.8097 to 0.8800 in the smoke removal task. Experimental results of the proposed transfer learning method also reveal its superior performance when trained with small datasets of target tasks. Significance: Experimental results on endoscopic images prove the effectiveness of the proposed network in smoke removal, light adjustment, and color correction, showing excellent clinical usefulness.

Research progress of frontier image processing in medical endoscopes

“…The article demonstrates that deep unsupervised endoscopic image enhancement based on multi-image fusion can effectively improve the visual quality and enhance the details of endoscopic images without the need for manual annotations. Li et al [40] Endoscopy image enhancement using adversarial training with generalized imaging defect models.…”