Deep Learning for Paint Loss Detection with a multiscale, translation invariant network

Meeus, Laurens; Devolder, Bart; Dubois, Hèléne; Martens, Maximiliaan; Pižurica, Aleksandra

doi:10.1109/ispa.2019.8868659

Cited by 7 publications

(5 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to improve the quality of the prediction with limited data, we introduced earlier a translation invariant UNet (TI-UNet). 8 This TI-UNet combines dilated convolutions with the typical U-Net without any down-and upsampling layers. As there is no reduction in spatial resolution, the information through the layers becomes more dense, which improved the training capacity when working with restricted amount of data.…”

Section: Methodsmentioning

confidence: 99%

“…However, this method is computationally intensive, which makes processing of larger images very challenging, especially in cases where user feedback is desirable. In our previous work 8 we proposed a novel neural network architecture for image segmentation that was optimized to employ the spatial context and multimodal data. Since this is a supervised deep learning approach, it requires a large amount of annotated data samples, which may be impractical.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Assisting classical paintings restoration: efﬁcient paint loss detection and descriptor-based inpainting using shared pretraining

Meeus

Žižakić

Chen

et al. 2020

Optics, Photonics and Digital Technologies for Imaging Applications VI

Self Cite

View full text Add to dashboard Cite

In the restoration process of classical paintings, one of the tasks is to map paint loss for documentation and analysing purposes. Because this is such a sizable and tedious job automatic techniques are highly on demand. The currently available tools allow only rough mapping of the paint loss areas while still requiring considerable manual work. We develop here a learning method for paint loss detection that makes use of multimodal image acquisitions and we apply it within the current restoration of the Ghent Altarpiece. Our neural network architecture is inspired by a multiscale convolutional neural network known as U-Net. In our proposed model, the downsampling of the pooling layers is omitted to enforce translation invariance and the convolutional layers are replaced with dilated convolutions. The dilated convolutions lead to denser computations and improved classification accuracy. Moreover, the proposed method is designed such to make use of multimodal data, which are nowadays routinely acquired during the restoration of master paintings, and which allow more accurate detection of features of interest, including paint losses. Our focus is on developing a robust approach with minimal user-interference. Adequate transfer learning is here crucial in order to extend the applicability of pre-trained models to the paintings that were not included in the training set, with only modest additional retraining. We introduce a pre-training strategy based on a multimodal, convolutional autoencoder and we fine-tune the model when applying it to other paintings. We evaluate the results by comparing the detected paint loss maps to manual expert annotations and also by running virtual inpainting based on the detected paint losses and comparing the virtually inpainted results with the actual physical restorations. The results indicate clearly the efficacy of the proposed method and its potential to assist in the art conservation and restoration processes.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Assisting classical paintings restoration: efﬁcient paint loss detection and descriptor-based inpainting using shared pretraining

Meeus

Žižakić

Chen

et al. 2020

Optics, Photonics and Digital Technologies for Imaging Applications VI

Self Cite

View full text Add to dashboard Cite

show abstract

“…As a case study, we use images from the panels of the Ghent Altarpiece [24], [25]. The paint-loss areas to be inpainted are detected with the algorithm from [57].…”

Section: Inpainting -Proof Of Concept For the Proposed Architecturementioning

confidence: 99%

Efficient Local Image Descriptors Learned With Autoencoders

Žižakić

Pižurica

2022

IEEE Access

Self Cite

View full text Add to dashboard Cite

Local image descriptors play a crucial role in many image processing tasks, such as object tracking, object recognition, panorama stitching, and image retrieval. In this paper, we focus on learning local image descriptors in an unsupervised way, using autoencoders and variational autoencoders. We perform a thorough comparative analysis of these two approaches along with an in-depth analysis of the most relevant hyperparameters to guide their optimal selection. In addition to this analysis, we give insights into the difficulties and the importance of selecting right evaluation techniques during the unsupervised learning of the local image descriptors. We explore the extent to which a simple perceptual metric during training can predict the performance on tasks such as patch matching, retrieval and verification. Finally, we propose an improvement to the encoder architecture that yields significant savings in memory complexity, especially in single-image tasks. As a proof of concept, we integrate our descriptor into an inpainting algorithm and illustrate its results when applied to the virtual restoration of master paintings. The source code required to reproduce the presented results has been made available as a repository on GitHub (https://github.com/nimpy/local-img-descr-ae).INDEX TERMS Local image descriptor, autoencoder, variational autoencoder, inpainting, unsupervised deep learning.

show abstract

“…13). Continued collaboration within the interdisciplinary team of art conservators and image processing and machine learning specialists led to further advances in automatic paint loss detection, also based on deep learning [36], [37].…”

Section: B Paint Loss Detection Using Sparse Codingmentioning

confidence: 99%

Revealing and Reconstructing Hidden or Lost Features in Art Investigation

Sober

Bucklow

Daly

et al. 2022

IEEE BITS Inform. Theory Mag.

View full text Add to dashboard Cite

In recent decades, cultural heritage research -and in particular art investigation -has been undergoing a digital revolution. This is due both to improvements in the digitization and the acquisition of artefact's images generated using traditional 2D imaging methods as well as the growing adoption of a range of more recently introduced spectroscopic imaging techniques.A number of these imaging modalities use wavelengths of electromagnetic radiation that can penetrate surface layers thus yielding information from hidden features non-invasively. Different techniques are often used in combination to provide evidence of construction, condition and past treatment. These can also be used to characterize the materials used, how they were combined and map their distribution, giving insight into an artist's working method and the means to understand changes that have occurred over time. This wealth of data calls for the development of algorithmic approaches in order to handle and fully explore and interpret it. The questions one seeks B. Sober is with the

show abstract

Deep Learning for Paint Loss Detection with a multiscale, translation invariant network

Cited by 7 publications

References 8 publications

Assisting classical paintings restoration: efﬁcient paint loss detection and descriptor-based inpainting using shared pretraining

Assisting classical paintings restoration: efﬁcient paint loss detection and descriptor-based inpainting using shared pretraining

Efficient Local Image Descriptors Learned With Autoencoders

Revealing and Reconstructing Hidden or Lost Features in Art Investigation

Contact Info

Product

Resources

About