Interactive Shadow Removal and Ground Truth for Variable Scene Categories

Abstract

Shadows are ubiquitous in image and video data, and their removal is of interest in both Computer Vision and Graphics. We present an interactive, robust and high quality method for fast shadow removal. To perform detection we use an on-the-fly learning approach guided by two rough user inputs for the pixels of the shadow and the lit area. From this we derive a fusion image that magnifies shadow boundary intensity change due to illumination variation. After detection, we perform shadow removal by registering the penumbra to a normalised frame which allows us to efficiently estimate non-uniform shadow illumination changes, resulting in accurate and robust removal. We also present a reliable, validated and multi-scene category ground truth for shadow removal algorithms which overcomes issues such as inconsistencies between shadow and shadowfree images and limited variations in shadows. Using our data, we perform the most thorough comparison of state of the art shadow removal methods to date. Our algorithm outperforms the state of the art, and we supply our code and evaluation data and scripts to encourage future open comparisons. Shadow removal ground truth The first public data set was supplied in [2]. In our work, we propose a new data set that introduces multiple shadow categories, and overcomes potential environmental illumination and registration errors between the shadow and ground truth images. An example of comparison is shown in Fig. 1. Our new data set avoids these issues using a careful capture setup and a quantitative test for rejecting unavoidable capture failures due to environmental effects. Our images are also categorised according to 4 different attributes. . An example from our data without these properties is shown in (c).Our algorithm consists of 3 steps (see Fig. 2): 1) Pre-processing We detect an initial shadow mask ( Fig. 2(b)) using a KNN classifier trained from data from two rough user inputs (e.g. Fig. 2(a)). We generate a fusion image, which magnifies illumination discontinuities around shadow boundaries, by fusing channels of YCrCb colour space and suppressing texture (Fig. 2(c)).2) Penumbra unwrapping Based on the detected shadow mask and fusion image, we sample the pixel intensities of sampling lines perpendicular to the shadow boundary ( Fig. 2(d)), remove noisy ones and store the remaining as columns for the initial penumbra strip (Fig. 2(e)). We align the initial columns' illumination changes using its intensity conversion image ( Fig. 2(f)). This results in an aligned penumbra strip (Fig. 2(g)) whose conversion image (Fig. 2(h)) exhibits a stabler profile.3) Estimation of shadow scale and relighting Unlike previous work [1, 2], we do not assume a constrained model of illumination change. The columns of penumbra strip are first clustered into a few small groups. A unified sample can be synthesised by averaging the samples of each group (e.g. Fig. 2(i)). Our shadow scale is adaptively and quickly derived from the unified samples which cancel texture noise. The derived sparse scales f...

Interactive Shadow Editing from Single Images

“…The resulting boundaries should exclude the boundaries of the image border. A typical solution to generate a shadow matte is to re-generate the sparse scales of sampling lines perpendicular to shadow boundaries and in-paint for the other unknown shadow scales in image like the algorithm to form a dense shadow scale field from sparse scales described in [1]. However, this can be computationally costly for interactive performance as the in-painting process is comparatively slow and every modification of shadow revokes the in-painting process.…”

“…Approaches to shadow removal from single images can be categorised as either automatic [2][3][4] or user-aided [5][6][7][8][9][10]1]. The difference between these is whether the shadow detection process is automatic or userguided.…”

“…To extract shadow from a single image, a state-of-the-art shadow removal method [1] is applied. This shadow removal method gives a shadow-free image, a shadow mask, and many shadow scale attenuation profiles perpendicular to the shadow boundary.…”

“…Automatic shadow removal methods generally rely on intrinsic image decomposition [2,3] or shadow feature learning [4]. User-aided methods generally achieve more accurate and reliable shadow detection results at the practical cost of varying degrees of user input, such as quad map [5], shadow boundary [6,8], and sample of intensity [9,10,1]. As for the removal phase, some of them [2][3][4][5]7] adopt pixel-wise optimisation while others [11,6,[8][9][10]1] rely on the analysis of intensity profiles passing through shadow boundary.…”

“…Based on a state-ofthe-art shadow removal dataset [1], we have both visually and quantitatively tested our system on various shadow scenes, which includes scenes with strong texture background, broken, soft and coloured shadows.…”