Nonlinear Spectral Image Fusion

Abstract

Abstract.In this paper we demonstrate that the framework of nonlinear spectral decompositions based on total variation (TV) regularization is very well suited for image fusion as well as more general image manipulation tasks. The well-localized and edge-preserving spectral TV decomposition allows to select frequencies of a certain image to transfer particular features, such as wrinkles in a face, from one image to another. We illustrate the effectiveness of the proposed approach in several numerical experiments, including a comparison to the competing techniques of Poisson image editing, linear osmosis, wavelet fusion and Laplacian pyramid fusion. We conclude that the proposed spectral TV image decomposition framework is a valuable tool for semi-and fullyautomatic image editing and fusion.

Modern regularization methods for inverse problems

“…Nevertheless, the spectral image fusion still works if registration and segmentation are carried out manually, as can be seen in Figure 16. For more information on the nonlinear spectral image fusion we refer to [35]. Figure 16: Image fusion using the nonlinear spectral TV decomposition on the challenging example of fusing a banknote with a picture of Gauß and a painting of Newton.…”

Variational Osmosis for Non-Linear Image Fusion

“…The use of nonlinear models for image fusion has recently been proposed in [5], where the nonlinear spectral decomposition of the Total Variation (TV) regularisation functional is used to improve texture details fusion in a multiscale scheme. The good performance of such approach is, however, somehow limited by the manual pre-processing (e.g.…”

“…The solution of the model is computed as a minimising pair (u, v) of the energy functional where O denotes an osmosis-based term promoting image fusion, D is a fidelity term forcing u to stay close to f in the foreground region and R acts as a regulariser enhancing the salient information of the structural image v. These terms are weighted with non negative parameters γ and η and detailed in Section III. Our model allows for more flexibility than [5] as it can take into account possible prior information available on the structures to-be-fused (such as, for instance piece-wise constant images) encoded in v. We use non-convex optimisation algorithms for computing efficiently a numerical solution of the problem. Numerical results show that the proposed model outperforms state-of-the-art linear and nonlinear methods for several face fusion, colour transfer and heritage imaging applications in terms of chromaticity errors and texture preservation.…”

“…They are based on the computation of the (suitably defined) spectral decomposition of one-homogeneous functionals (such as the Total Variation semi-norm) and on their use as a basis capturing multi-scale (texture) details. Such property is indeed exploited in [5] to deal also with related imaging problems such as image insertion and artistic image transformation. One of the main drawbacks of this approach is the precise (and often heuristic) selection of the eigenvectors extracted from the images to fuse required to combine information with enough precision.…”

Theoretical Analysis of Flows Estimating Eigenfunctions of One-Homogeneous Functionals

“…The method was later generalized to one-homogeneous functionals in [19] where certain properties, like orthogonality of the decomposed components, were shown in specific settings. Applications related to denoising [34], texture manipulation [29,10] and segmentation of medical data [42] were suggested.…”