Automatic Registration of Images to Untextured Geometry Using Average Shading Gradients

Abstract

Many existing approaches for image-to-geometry registration assume that either a textured 3D model or a good initial guess of the 3D pose is available to bootstrap the registration process. In this paper we consider the registration of photographs to 3D models even when no texture information is available. This is very challenging as we cannot rely on texture gradients, and even shading gradients are hard to estimate since the lighting conditions are unknown. To that end, we propose average shading gradients, a rendering technique that estimates the average gradient magnitude over all lighting directions under Lambertian shading. We use this gradient representation as the building block of a registration pipeline based on matching sparse features. To cope with inevitable false matches due to the missing texture information and to increase robustness, the pose of the 3D model is estimated in two stages. Coarse pose hypotheses are first obtained from a single correct match each, subsequently refined using SIFT flow, and finally verified. We apply our algorithm to registering images of real-world objects to untextured 3D meshes of limited accuracy. Moreover, we show that registration can be performed even for paintings despite lacking photo-realism.

Automatic Co-Registration of Aerial Imagery and Untextured Model Data Utilizing Average Shading Gradients

“…The exact calculation of ||∇I|| is very computation-intensive due to the complex integrand. Therefore, approximations are proposed by Plötz and Roth (2017) that allow the estimation of ||∇I|| in closed form: (4), we refer to (Plötz and Roth, 2017). The represented form enables an efficient calculation of the desired gradient image, which is based exclusively on the convolution of the normal map with derivation filters.…”

“…Our algorithm is based on the concept for automatic registration of images to untextured geometry, which has been proposed by Plötz and Roth (2017). Assuming an initial camera pose, a given untextured 2.5D or 3D model and a camera image, our algorithm estimates the intrinsic and extrinsic camera parameters based on 2D-3D correspondences between pixels in the input image and object points in the model.…”

“…Since we deal with an untextured model, the calculation of texture gradients on rendered images generated from the model is not possible. To this end, we rely only on gradients of shadings by applying the so-called Average Shading Gradients (ASG) proposed by Plötz and Roth (2017). This is a rendering technique in which observable shading gradients are averaged over all possible lighting directions of the 3D scene under the assumption of a local illumination model.…”

Using Curvilinear Features in Focus for Registering a Single Image to a 3D Object

“…Other methods solve the problem by distinguishing two subproblems: to choose the common representation of the data and, then, to find the correspondences. These methods transforms the initial 2D/3D registration problem to a 2D/2D matching problem by rendering multiple 2D images of 3D models from different viewpoints, such as [8,9,10]. Consequently, the first task of 2D/3D registration is to find an appropriate representation of 3D models in which reliable features can be extracted in 2D and 3D data.…”

“…Since the depth and the intensity surfaces have a different order of representation, the two surfaces can not be directly matched. Thus, bringing both rendered depth images and photographs into a common representation, such as gradient and edge representation, allows to establish a robust sparse 2D-to-3D matching [10]. We propose to extract gradient-based features corresponding to object's shapes in both depth and intensity images regardless of illumination and texture changes.…”