Two Cloud-Based Cues for Estimating Scene Structure and Camera Calibration

Abstract

Abstract-We describe algorithms that use cloud shadows as a form of stochastically structured light to support 3D scene geometry estimation. Taking video captured from a static outdoor camera as input, we use the relationship of the time series of intensity values between pairs of pixels as the primary input to our algorithms. We describe two cues that relate the 3D distance between a pair of points to the pair of intensity time series. The first cue results from the fact that two pixels that are nearby in the world are more likely to be under a cloud at the same time than two distant clouds. We describe methods for using this cue to estimate focal length and scene structure. The second cue is based on the motion of shadow clouds across the scene; this cue results in a set of linear constraints on scene structure. These constraints have an inherent ambiguity, which we show how to overcome by combining the cloud motion cue with the spatial cue. We evaluate our method on several time lapses of real outdoor scenes.

Scene geometry from several partly cloudy days

“…Several cues and approaches have been proposed for estimating shape information from video of an outdoor scene captured by a static camera [1], [2], [6], [9], [11]. These approaches all rely on strong assumptions about the scene, including clear sky conditions [1], [2], [9] or constant albedo [1], [9].…”

“…Methods that use moving cloud shadows to estimate scene geometry [6], [7] provide direct constraints on scene depth. This previous work introduces the delay-based constraint, which relates the distance between pairs of pixels to the rate of cloud motion and the time of transit (i.e., "distance equals rate times time").…”

“…We now formulate a constraint that describes the relationship between distance, rate (cloud velocity) and time [6]. Given two 3D points, x p , x q , directly in line with the cloud motion vector w d , we define the following relation:…”

“…This means there is a fixed temporal delay for each pair of pixels. For each video, we compute the temporal delay between all pairs of pixels, Λ d , by comparing the pixel intensity time series for each pair using a two-stage process [6]. The first stage measures delay using pairwise cross correlation to obtain an initial, integer estimate of λ dpq and correlation, ρ dpq .…”

“…In this domain, most approaches use either photometric cues from changes in the sun position [1], [3] to provide estimates of surface orientation or geometric cues based on cloud motion [6] to give direct constraints on scene shape. In this work, we focus on the cloud cue and generalize the problem setting to incorporate constraints provided by multiple days (with independent cloud motion directions).…”