Reference Picture Synthesis for Video Sequences Captured with a Monocular Moving Camera

3D Geometry-Based Global Motion Compensation For VVC

“…Unlike [13,[15][16][17], in which a tedious 3D reconstruction has to be done at both encoder and decoder sides, we estimate the 3D geometry here only at the encoder side from raw/uncompressed frames. Using raw frames instead of encoded frames (at the decoder side) can lead to a higher quality 3D geometry estimation.…”

“…This 3D data is then employed to synthesize additional virtual RPs. Later, two motion compensation schemes based on un-textured 3D meshes [15][16] and point-clouds [17] were introduced. These methods sped up the method in [13] and increased the coding gains by integrating Depth Image-Based Rendering (DIBR) [18] into HEVC.…”

“…These methods sped up the method in [13] and increased the coding gains by integrating Depth Image-Based Rendering (DIBR) [18] into HEVC. Especially [16] presented decent coding gains: around 6% and 15% bitrate reduction for scenes with and without moving objects, respectively (Anchor: HM16.7).…”

“…Although [15][16][17] achieved impressive coding gains, they have some drawbacks: (1) using HEVC as the anchor, which is not state-of-the-art anymore, and (2) forcing the decoder to tediously reconstruct 3D geometry from encoded frames resulting in computational complexity at the decoder. In this paper, we propose the following modifications of [15][16]: We take VVC as the anchor, estimate 3D geometry from "raw" data at the encoder side, and send it to the decoder. The bitstream of the 3D geometry is considered as overhead.…”

“…This overhead should be minimized not to eat up the whole coding gain, especially when a super-efficient codec like VVC is used as the anchor. We have updated our mesh reconstruction algorithm proposed in [16] and added mesh processing, mesh decimation, and mesh compression blocks to minimize the overhead and running-time. The rest of the paper is structured as follows: Section 2 reviews the proposed algorithm.…”