Motion Estimation for Fisheye Video With an Application to Temporal Resolution Enhancement

Abstract

Surveying wide areas with only one camera is a typical scenario in surveillance and automotive applications. Ultra wide-angle fisheye cameras employed to that end produce video data with characteristics that differ significantly from conventional rectilinear imagery as obtained by perspective pinhole cameras. Those characteristics are not considered in typical image and video processing algorithms such as motion estimation, where translation is assumed to be the predominant kind of motion. This contribution introduces an adapted technique for use in block-based motion estimation that takes into account the projection function of fisheye cameras and thus compensates for the non-perspective properties of fisheye videos. By including suitable projections, the translational motion model that would otherwise only hold for perspective material is exploited, leading to improved motion estimation results without altering the source material. In addition, we discuss extensions that allow for a better prediction of the peripheral image areas, where motion estimation falters due to spatial constraints, and further include calibration information to account for lens properties deviating from the theoretical function. Simulations and experiments are conducted on synthetic as well as real-world fisheye video sequences that are part of a data set created in the context of this work. Average synthetic and real-world gains of 1.45 and 1.51 dB in luminance PSNR are achieved compared against conventional block matching. Furthermore, the proposed fisheye motion estimation method is successfully applied to motion compensated temporal resolution enhancement, where average gains amount to 0.79 and 0.76 dB.

Hemispherical Photographs: A Review of Acquisition Methods and Applications in the Context of Urban Energy and Environment Assessments

“…This is because, in addition to matching the mechanical fixture, the size of the image and the sensor should also be matched. If the image size is bigger than the sensor's dimensions, an undesirable cropped photograph will be generated [107]. If the image is smaller than the sensor, image masking is required as an additional task, in which the unwanted area of the photograph is carefully removed [108].…”

A Novel Viewport-Adaptive Motion Compensation Technique for Fisheye Video

“…The projection-based approach from [12] adapts the block matching procedure according to the fisheye projection as follows. While the overall procedure remains the same, the motion vectors are interpreted in the perspective domain instead of the fisheye domain directly.…”

“…We propose an extension of the projection-based approach [12] that projects the coordinates into different viewports in the perspective domain in order to realize differently oriented motion planes in 3D space. In this context, a viewport describes the orientation of a virtual perspective camera in 3D space.…”

“…back) lies on the virtual image plane. In [12], a so-called ultra wide-angle compensation is performed to account for the virtual image plane. However, it does not provide exact reprojections into the spherical domain as it performs the compensation based on the reprojected radius, and hence, does not consider the nonlinearity of the projection functions.…”

“…In the following, three methods for blockbased motion estimation and compensation employing a translatory motion model are tested. The traditional motion compensation method employing a translatory motion model is thereby called Translatory Motion Compensation (TMC) [8], the projection-based approach is called Projection-based Translatory Motion Compensation (PTMC) [12], and the novel viewport-adaptive approach is called Viewport-Adaptive Projection-based Translatory Motion Compensation (VA-PTMC).…”