State‐of‐the‐art in Multi‐Light Image Collections for Surface Visualization and Analysis

Abstract

Multi‐Light Image Collections (MLICs), i.e., stacks of photos of a scene acquired with a fixed viewpoint and a varying surface illumination, provide large amounts of visual and geometric information. In this survey, we provide an up‐to‐date integrative view of MLICs as a mean to gain insight on objects through the analysis and visualization of the acquired data. After a general overview of MLICs capturing and storage, we focus on the main approaches to produce representations usable for visualization and analysis. In this context, we first discuss methods for direct exploration of the raw data. We then summarize approaches that strive to emphasize shape and material details by fusing all acquisitions in a single enhanced image. Subsequently, we focus on approaches that produce relightable images through intermediate representations. This can be done both by fitting various analytic forms of the light transform function, or by locally estimating the parameters of physically plausible models of shape and reflectance and using them for visualization and analysis. We finally review techniques that improve object understanding by using illustrative approaches to enhance relightable models, or by extracting features and derived maps. We also review how these methods are applied in several, main application domains, and what are the available tools to perform MLIC visualization and analysis. We finally point out relevant research issues, analyze research trends, and offer guidelines for practical applications.

A novel approach for exploring annotated data with interactive lenses

“…Stratigraphic and relightable models While the techniques presented in this work are of general usage, and can be applied to various multi‐faceted 2D exploration tasks, we have focused our implementation on the special case of stratigraphic relightable model. In the last decade, a wide variety of tools have been proposed for targeting either static exploration of multi‐faceted image data (e.g., multi‐spectral or stratigraphic data [MAD∗18, PSK09] or multi‐light image collections [VHW∗18, Mac15]), or dynamic exploration through relighting [PDC∗19]. In this context, lens‐based interaction with such models has been used previously by Jaspe et al [JPGG19, JAP∗21] solely with the purpose of letting users see different layers inside or outside the lens.…”

A Comprehensive Introduction to Photometric 3D-Reconstruction

“…Also, let us mention Reflectance Transform Imaging (RTI) techniques, which are based on PS principles, and allow one to interactively relight the pictured surfaces. Such an approach finds a natural application in the field of cultural heritage, see the recent survey [88] for an overview. Finally, Chapter 7 in the present volume addresses a novel application, which is the estimation of facial aging.…”

Neural reflectance transformation imaging

“…According to a recent survey [14] on the state of the art of RTI processing and relighting of multi-light image collections, practical applications, especially in the cultural heritage domain, rely on the classic polynomial texture mapping (PTM) [9] or on Hemispherical Harmonics (HSH) [10] coefficients to store a compact representation of the original data and interactively relightable images.…”

“…This dataset is composed of 12 multi-light Fig. 6 Sample images of the RealRTI dataset representing the 12 different surfaces captured image collections (cropped and resized to allow a fast processing/evaluation) acquired with light domes or handheld RTI protocols [14] on surfaces with different shape and material complexity. The items imaged are: (1) a wooden painted door (handheld acquisition, 60 light directions), (2) a fresco (dome acquisition, 47 lights), (3,4) two painted icons (handheld 63 and 72 lights), (5,6) two paintings on canvas (handheld, 49 lights and dome, 48 lights), impressions on plaster of a leaf (7) and a shell (8) (light dome, 48 lights), (9,10) two coins (both with light dome, 48 lights), and (11,12) two metallic statues (dome, 48 lights and handheld, 54 lights).…”

“…These sets are often referred to also as multi-light image collections (MLIC). A recent survey [14] shows that there is a large number of applications exploiting interactive relighting and feature extraction from this kind of data, in different fields like cultural heritage, material science, archaeology, quality control, and natural sciences.…”