Flexible Synthesis of Video Frames Based on Motion Hints

Abstract

In this paper, we propose the use of "motion hints" to produce interframe predictions. A motion hint is a loose and global description of motion that can be communicated using metadata; it describes a continuous and invertible motion model over multiple frames, spatially overlapping other motion hints. A motion hint provides a reasonably accurate description of motion but only a loose description of where it is applicable; it is the task of the client to identify the exact locations where this motion model is applicable. The focus of this paper is a probabilistic multiscale approach to identifying these locations of applicability; the method is robust to noise, quantization, and contrast changes. The proposed approach employs the Laplacian pyramid; it generates motion hint probabilities from observations at each scale of the pyramid. These probabilities are then combined across the scales of the pyramid starting from the coarsest scale. The computational cost of the approach is reasonable, and only the neighborhood of a pixel is employed to determine a motion hint probability, which makes parallel implementation feasible. This paper also elaborates on how motion hint probabilities are exploited in generating interframe predictions. The scheme of this paper is applicable to closed-loop prediction, but it is more useful in open-loop prediction scenarios, such as using prediction in conjunction with remote browsing of surveillance footage, communicated by a JPEG2000 Interactive Protocol (JPIP) server. We show that the interframe predictions obtained using the proposed approach are good both visually and in terms of PSNR.

Overlapping motion hints with polynomial motion for video communication

“…In practice, the multi-resolution decomposition needs to be performed only where domains overlap. For completeness, the probabilistic multiscale framework, taken from [18], is outlined briefly below.…”

“…All these observations are evaluated on a 7 × 7-pixel neighborhood of k. The structural measure is a novel operator that can distinguish unstructured (noise-like) regions from highly structured image features; this turns out to be very important in identifying the usefulness of the information derived within each detail band. The reader is referred to [17,18,20] for more on how these observations are evaluated.…”

“…This table is obtained by training the algorithm using matching patches for the numerator, and mismatching patches for the denominator; these patches are obtained from the Kodak test image set with different levels of noise and spatial shift. The reader is referred to [18] for a more detailed account on how these probability density functions are obtained.…”

“…In recent works [16][17][18], we propose the use of motion hints to communicate motion. A motion hint communicates reasonably accurate motion information that is applicable in only a region within a domain associated with the motion hint; it accurately describes motion inside the region of applicability (ROA).…”

“…In [17], we propose a multi-scale approach that is suitable for the case of two motion hints, one foreground and one background, using affine motion models. In [18], we introduce a more rigorous multi-scale framework for identifying the ROA that is suitable for multiple overlapping motion hints; however, the performance of the approach is demonstrated only with affine motion models and two overlapping motion hints, as in [17]. This work focuses on demonstrating the effectiveness of motion hints with higher order motion models and real video sequences, such as the "foreman" sequence, as well as the applicability of the ROA approach in cases where there are several overlapping motion hints.…”