Real‐time mosaic of multiple fisheye surveillance videos based on geo‐registration and rectification

Abstract

A distributed fisheye video surveillance system (DFVSS) can monitor a wide area without blind spots, but it is often affected by the viewpoint discontinuity and space inconsistency of multiple videos in the area. This paper proposes a novel real‐time fisheye video mosaic algorithm for wide‐area surveillance. First, by extending the line photogrammetry theory under central projection to spherical projection, a fisheye video geo‐registration model is established and estimated using orthogonal parallel lines on the ground, so that all videos of DFVSS are in the unified reference system to eliminate the space inconsistency between them. Second, by combining the photogrammetry orthorectification technique with thin‐plate spline transformation, a fisheye video rectification model is established to eliminate serious distortion in geo‐registered fisheye videos and align them accurately. Third, the viewport‐dependent video selection strategy and video look‐up table computation technique are adopted to create a high‐resolution panorama from input fisheye videos in real time. A parking lot of about 0.4 km2 monitored by eight fisheye cameras was selected as the test area. The experimental result shows the line re‐projection error in fisheye videos is about 0.5 pixels, and the overall efficiency, including panorama creation and mapping to the ground as texture, is not <30 fps. It indicates that the proposed algorithm can achieve a good balance between the limitation of video transmission bandwidth and the smooth observation requirement of computer equipment for the panorama, which is of great value for the construction and application of DFVSS.

Real‐time generation of spherical panoramic video using an omnidirectional multi‐camera system

“…If we consider (x, y) as a representative control point, a total of 2n + 6 TPS parameters can be systematically derived. This is achieved by satisfying six boundary constraints and the Euclidean distances between P i (x i , y i ) and P j (x j , y j ) when n exceeds three, and j is not equal to i (Gao et al, 2023).…”

“…Gao et al (2023) introduced a real-time fisheye video mosaic algorithm aimed at addressing challenges arising from viewpoint disparities and spatial incongruities within a distributed fisheye video surveillance system that incorporates multiple videos. This was accomplished by extending line photogrammetry theory and orthorectification techniques.…”