AIM 2020 Challenge on Image Extreme Inpainting

Ntavelis, Evangelos; Romero, Andrés; Bigdeli, Siavash Arjomand; Timofte, Radu; Hui, Zheng; Xiu-mei, Wang; Gao, Xinbo; Shin, Chajin; Kim, Taeoh; Son, Hanbin; Lee, Sangyoun; Li, Chao; Fu, Li; He, Dongliang; Wen, Shifeng; Ding, Errui; Bai, Mengmeng; Li, Shu-Chen; Zeng, Yu; Lin, Zhe; Yang, Jimei; Zhang, Jianming; Shechtman, Eli; Lu, Huchuan; Zeng, Weijian; Ni, Haopeng; Cai, Yiyang; Li, Chenghua; Xu, Dejia; Wu, Haoning; Han, Yu; Nadim, Uddin S.M.; Jang, Hae Woong; Ahmed, Soikat Hasan; Yoon, Jungmin; Jung, Yong Ju; Li, Chu Tak; Liu, Zhi Song; Wang, Li Wen; Siu, Wan Chi; Lun, Daniel Pak-Kong; Suin, Maitreya; Purohit, Kuldeep; Rajagopalan, A. N.; Narang, Pratik; Mandal, Murari; Chauhan, Pranjal Singh

doi:10.1007/978-3-030-67070-2_43

Cited by 27 publications

(11 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…22 Our proposed CropGAN uses GANs to synthesize missing data, a technique that has been well tested in network-based inpainting tasks. 17,31 Specifically, we based our method on the winner of the AIM 2020 Challenge on Extreme Inpainting, 19,32 which used deep features in the generator, discriminator, and a VGG net as terms in the loss function. That study showed impressive results filling in holes of 2D color photos, yet has not yet been pursued for image extension nor for 3D medical images.…”

Section: Discussionmentioning

confidence: 99%

Using neural networks to extend cropped medical images for deformable registration among images with differing scan extents

et al. 2021

View full text Add to dashboard Cite

Purpose Missing or discrepant imaging volume is a common challenge in deformable image registration (DIR). To minimize the adverse impact, we train a neural network to synthesize cropped portions of head and neck CT's and then test its use in DIR. Methods Using a training dataset of 409 head and neck CT's, we trained a generative adversarial network to take in a cropped 3D image and output an image with synthesized anatomy in the cropped region. The network used a 3D U‐Net generator along with Visual Geometry Group (VGG) deep feature losses. To test our technique, for each of the 53 test volumes, we used Elastix to deformably register combinations of a randomly cropped, full, and synthetically full volume to a single cropped, full, and synthetically full target volume. We additionally tested our method's robustness to crop extent by progressively increasing the amount of cropping, synthesizing the missing anatomy using our network, and then performing the same registration combinations. Registration performance was measured using 95% Hausdorff distance across 16 contours. Results We successfully trained a network to synthesize missing anatomy in superiorly and inferiorly cropped images. The network can estimate large regions in an incomplete image, far from the cropping boundary. Registration using our estimated full images was not significantly different from registration using the original full images. The average contour matching error for full image registration was 9.9 mm, whereas our method was 11.6, 12.1, and 13.6 mm for synthesized‐to‐full, full‐to‐synthesized, and synthesized‐to‐synthesized registrations, respectively. In comparison, registration using the cropped images had errors of 31.7 mm and higher. Plotting the registered image contour error as a function of initial preregistered error shows that our method is robust to registration difficulty. Synthesized‐to‐full registration was statistically independent of cropping extent up to 18.7 cm superiorly cropped. Synthesized‐to‐synthesized registration was nearly independent, with a −0.04 mm of change in average contour error for every additional millimeter of cropping. Conclusions Different or inadequate in scan extent is a major cause of DIR inaccuracies. We address this challenge by training a neural network to complete cropped 3D images. We show that with image completion, the source of DIR inaccuracy is eliminated, and the method is robust to varying crop extent.

show abstract

Section: Discussionmentioning

confidence: 99%

Using neural networks to extend cropped medical images for deformable registration among images with differing scan extents

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Deterministic Image Inpainting Since the introduction of GANs [6], most of the existing methods follow a standard configuration, first proposed by Pathak et al [26], that is, using an encoder-decoder architecture as the main inpainting generator, adversarial training, and tailored losses that aim at photo-realism. Follow-up works have produced impressive results in recent years [12,17,24,28,43].…”

Section: Related Workmentioning

confidence: 99%

RePaint: Inpainting using Denoising Diffusion Probabilistic Models

Lugmayr¹,

Danelljan²,

Romero³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Figure 1. We use Denoising Diffusion Probabilistic Models (DDPM) for inpainting. The process is conditioned on the masked input (left). It starts from a random Gaussian noise sample that is iteratively denoised until it produces a high-quality output. Since this process is stochastic, we can sample multiple diverse outputs. The DDPM prior forces a harmonized image, is able to reproduce texture from other regions, and inpaint semantically meaningful content.

show abstract

“…Deep learning model and training procedure: Segmentation While it is common to use pre-processed images in conventional deep neural networks, [31,32] it is now possible to utilize more features from original raw images than processed images with advanced DL. [33] Thus, we used original infrared MG images with investigators' annotated segmentations to train DL.…”

Section: Meiboscore Grading By 2 Mgd Expertsmentioning

confidence: 99%

AI-based automated Meibomian gland segmentation, classification and reflection correction in infrared Meibography

Saha¹,

Chowdhury²,

Na³

et al. 2022

Preprint

View full text Add to dashboard Cite

Purpose: Develop a deep learning-based automated method to segment meibomian glands (MG) and eyelids, quantitatively analyze the MG area and MG ratio, estimate the meiboscore, and remove specular reflections from infrared images. Methods: A total of 1600 meibography images were captured in a clinical setting. 1000 images were precisely annotated with multiple revisions by investigators and graded 6 times by meibomian gland dysfunction (MGD) experts. Two deep learning (DL) models were trained separately to segment areas of the MG and eyelid. Those segmentation were used to estimate MG ratio and meiboscores using a classification-based DL model. A generative adversarial network was implemented to remove specular reflections from original images. Results: The mean ratio of MG calculated by investigator annotation and DL segmentation was consistent 26.23% vs 25.12% in the upper eyelids and 32.34% vs. 32.29% in the lower eyelids, respectively. Our DL model achieved 73.01% accuracy for meiboscore classification on validation set and 59.17% accuracy when tested on images from independent center, compared to 53.44% validation accuracy by MGD experts. The DL-based approach successfully removes reflection from the original MG images without affecting meiboscore grading. Conclusions: DL with infrared meibography provides a fully automated, fast quantitative evaluation of MG morphology (MG Segmentation, MG area, MG ratio, and meiboscore) which are sufficiently accurate for diagnosing dry eye disease. Also, the DL removes specular reflection from images to be used by ophthalmologists for distraction-free assessment.

show abstract

AIM 2020 Challenge on Image Extreme Inpainting

Cited by 27 publications

References 26 publications

Using neural networks to extend cropped medical images for deformable registration among images with differing scan extents

Using neural networks to extend cropped medical images for deformable registration among images with differing scan extents

RePaint: Inpainting using Denoising Diffusion Probabilistic Models

AI-based automated Meibomian gland segmentation, classification and reflection correction in infrared Meibography

Contact Info

Product

Resources

About