Guidefill: GPU Accelerated, Artist Guided Geometric Inpainting for 3D Conversion of Film

Abstract

The conversion of traditional film into stereo 3D has become an important problem in the past decade. One of the main bottlenecks is a disocclusion step, which in commercial 3D conversion is usually done by teams of artists armed with a toolbox of inpainting algorithms. A current difficulty in this is that most available algorithms are either too slow for interactive use, or provide no intuitive means for users to tweak the output.In this paper we present a new fast inpainting algorithm based on transporting along automatically detected splines, which the user may edit. Our algorithm is implemented on the GPU and fills the inpainting domain in successive shells that adapt their shape on the fly. In order to allocate GPU resources as efficiently as possible, we propose a parallel algorithm to track the inpainting interface as it evolves, ensuring that no resources are wasted on pixels that are not currently being worked on. Theoretical analysis of the time and processor complexity of our algorithm without and with tracking (as well as numerous numerical experiments) demonstrate the merits of the latter.Our transport mechanism is similar to the one used in coherence transport [7,27], but improves upon it by correcting a "kinking" phenomenon whereby extrapolated isophotes may bend at the boundary of the inpainting domain. Theoretical results explaining this phenomena and its resolution are presented.Although our method ignores texture, in many cases this is not a problem due to the thin inpainting domains in 3D conversion. Experimental results show that our method can achieve a visual quality that is competitive with the state-of-the-art while maintaining interactive speeds and providing the user with an intuitive interface to tweak the results.

Unveiling the invisible: mathematical methods for restoring and interpreting illuminated manuscripts

“…Here we briefly go over the 3D conversion pipeline used in this paper. For more details, please see [27] or [23,Ch. 9.4].…”

“…Reliable digital restoration for illuminated manuscripts requires a multi-disciplinary collaboration as the current work is based on. In what follows we discuss a range of new adaptive, semi-automated restoration methods that (a) reconstruct imagestructures using partial differential equations [17,18,19,20,21,22,23], (b) mimic the human-expert behaviour, using texture-and structure patches sampled from the intact part of the illuminated manuscript at hand and integrating them in exemplar-based inpainting approaches [24,25], (c) exploit infrared imaging data, correlating the visible image content with its traces in the hidden layers of paint [26,16], and (d) create new 3D interpretations of illuminated manuscripts through a new 3D conversion pipeline [27].…”

Analysis of Artifacts in Shell-Based Image Inpainting: Why They Occur and How to Eliminate Them

“…The region to be filled is subsequently referred to as the inpainting domain. Since the seminal work of Bertalmio et al [6], image inpainting has become increasingly important, with applications ranging from removing an undesirable or occluding object from a photograph, to painting out a wire in an action sequence, to 3D conversion of film [27], as well as 3D TV and novel viewpoint construction [12,17,29] in more recent years. See [21] for a recent survey of the field.…”

“…These approaches are typically iterative and convergence is often slow (due to either the high order of the underlying PDE model or the difficult nature of the variational problem). On the other hand, Telea's Algorithm [38], coherence transport [7,31], and our previous work Guidefill [27] are based on the simple idea of filling the inpainting domain in successive shells from the boundary inward, setting the color of pixels on the current boundary equal to a weighted average of their already filled neighbors- Fig. 1 illustrates this process.…”

“…1 illustrates this process. In general, "neighbors" may include "ghost pixels" lying between pixel centers and defined using bilinear interpolation-this concept was introduced in [27]. By choosing the weights, fill order, and averaging neighborhood carefully, image isophotes may be extrapolated into the inpainting domain.…”