Deep Multi-class Adversarial Specularity Removal

Lin, John Y.; Seddik, Mohamed El Amine; Tamaazousti, Mohamed; Tamaazousti, Youssef; Bartoli, Adrien

doi:10.1007/978-3-030-20205-7_1

Cited by 29 publications

(25 citation statements)

References 30 publications

(45 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is because the evaluated image datasets are often collected in-house with limited image diversity that do not fully reflect the tissue appearance and different image acquisition practices worldwide. Most works have been performed on short video clips, selected artefact type, single imaging modality and single organ datasets [8][9][10][11] .…”

mentioning

confidence: 99%

An objective comparison of detection and segmentation algorithms for artefacts in clinical endoscopy

Ali

Zhou

Braden

et al. 2020

Sci Rep

View full text Add to dashboard Cite

We present a comprehensive analysis of the submissions to the first edition of the Endoscopy Artefact Detection challenge (EAD). Using crowd-sourcing, this initiative is a step towards understanding the limitations of existing state-of-the-art computer vision methods applied to endoscopy and promoting the development of new approaches suitable for clinical translation. Endoscopy is a routine imaging technique for the detection, diagnosis and treatment of diseases in hollow-organs; the esophagus, stomach, colon, uterus and the bladder. However the nature of these organs prevent imaged tissues to be free of imaging artefacts such as bubbles, pixel saturation, organ specularity and debris, all of which pose substantial challenges for any quantitative analysis. Consequently, the potential for improved clinical outcomes through quantitative assessment of abnormal mucosal surface observed in endoscopy videos is presently not realized accurately. The EAD challenge promotes awareness of and addresses this key bottleneck problem by investigating methods that can accurately classify, localize and segment artefacts in endoscopy frames as critical prerequisite tasks. Using a diverse curated multi-institutional, multi-modality, multi-organ dataset of video frames, the accuracy and performance of 23 algorithms were objectively ranked for artefact detection and segmentation. The ability of methods to generalize to unseen datasets was also evaluated. The best performing methods (top 15%) propose deep learning strategies to reconcile variabilities in artefact appearance with respect to size, modality, occurrence and organ type. However, no single method outperformed across all tasks. Detailed analyses reveal the

show abstract

mentioning

confidence: 99%

An objective comparison of detection and segmentation algorithms for artefacts in clinical endoscopy

Ali

Zhou

Braden

et al. 2020

Sci Rep

View full text Add to dashboard Cite

show abstract

“…In the last decades, many approaches have been proposed to address this challenging specular highlight removal problem. These existing works can be roughly classified into three categories: dichromatic reflection model-based methods [28,32,23,7,26], inpainting-based methods [27,21,4,18], and deep learningbased methods [8,15,20]. The dichromatic reflection model [24] linearly combines the diffuse and specular reflections, and subsequently many methods are proposed based on this model.…”

Section: Highlight Image Ocr Resultsmentioning

confidence: 99%

“…This kind of methods have limited performance for the large highlight contamination. Considering the complexity of single image specular highlight removal, some recent works [8,15,20] are proposed based on the deep neural networks, e.g., convolutional neural network (CNN) and generative adversarial network (GAN). With the aid of the powerful learning capacity of deep models, these deep learning-based methods usually have better performance compared with traditional optimization-based methods.…”

Section: Highlight Image Ocr Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Text-Aware Single Image Specular Highlight Removal

Hou¹,

Wang²,

Quan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Removing undesirable specular highlight from a single input image is of crucial importance to many computer vision and graphics tasks. Existing methods typically remove specular highlight for medical images and specific-object images, however, they cannot handle the images with text. In addition, the impact of specular highlight on text recognition is rarely studied by text detection and recognition community. Therefore, in this paper, we first raise and study the text-aware single image specular highlight removal problem. The core goal is to improve the accuracy of text detection and recognition by removing the highlight from text images. To tackle this challenging problem, we first collect three high-quality datasets with fine-grained annotations, which will be appropriately released to facilitate the relevant research. Then, we design a novel two-stage network, which contains a highlight detection network and a highlight removal network. The output of highlight detection network provides additional information about highlight regions to guide the subsequent highlight removal network. Moreover, we suggest a measurement set including the end-to-end text detection and recognition evaluation and auxiliary visual quality evaluation. Extensive experiments on our collected datasets demonstrate the superior performance of the proposed method.

show abstract

“…In machine learning, collecting a proper image database with segmented specularities is a long and tedious process which could be tremendously spedup with a proper specularity geometric model for both specularity synthesis, segmentation and removal applications. Current state of the art relies heavily on synthetic image databases as shown in [30], or build a database from real images using specularity detection system such as [25], [39]. However, Dual JOLIMAS is limited to uniformly curved surfaces and fails for changes in the surface curvature.…”

Section: Potential Applicationsmentioning

confidence: 99%

A multiple-view geometric model of specularities on non-uniformly curved surfaces

Morgand

Tamaazousti

Bartoli

2017

Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology

Self Cite

View full text Add to dashboard Cite

Specularity prediction is essential to many computer vision applications. In Augmented Reality (AR), it improves the realism of virtual objects inserted in a live video stream by providing a coherent shape, appearance and motion to specularities. Specularity prediction also gives important visual cues that could be used in Simultaneous Localisation and Mapping (SLAM), 3D reconstruction and material modeling, thus improving scene understanding. However, specularity prediction is a challenging task requiring numerous information from the scene including the camera pose, the geometry of the scene, the light sources and the material properties. Our previous work have addressed this task by creating an explicit model using an ellipsoid whose projection fits the specularity image contours for a given camera pose. These ellipsoid-based approaches belong to a family of models called JOint-LIght MAterial Specularity (JOLIMAS), where we have attempted to gradually remove assumptions on the scene such as the geometry of the specular surfaces to provide real-time specularity prediction. However, our most recent approach is still limited to uniformly curved surfaces. This paper builds upon these methods by generalising JOLIMAS to any surface geometry while improving the quality of specularity prediction, without sacrificing computation performances. The proposed method establishes a link between surface curvature and specularity shape in order to lift the geometric assumptions from previous work. Contrary to previous work, our new model is built from a physics-based local illumination model namely Torrance-Sparrow, providing a better model reconstruction. Specularity prediction using our new model is tested against the most recent JOLIMAS version on both synthetic and real sequences with objects of varying shape curvatures, reconstructed using a CAD model or the Kinect v2 depth camera. Our method outperforms previous approaches in specularity prediction, including the real-time setup, as shown in the supplementary material using videos.

show abstract

Deep Multi-class Adversarial Specularity Removal

Cited by 29 publications

References 30 publications

An objective comparison of detection and segmentation algorithms for artefacts in clinical endoscopy

An objective comparison of detection and segmentation algorithms for artefacts in clinical endoscopy

Text-Aware Single Image Specular Highlight Removal

A multiple-view geometric model of specularities on non-uniformly curved surfaces

Contact Info

Product

Resources

About