Computationally Efficient Light Field Image Compression Using a Multiview HEVC Framework

Abstract

The acquisition of the spatial and angular information of a scene using light field (LF) technologies supplement a wide range of post-processing applications, such as scene reconstruction, refocusing, virtual view synthesis, and so forth. The additional angular information possessed by LF data increases the size of the overall data captured while offering the same spatial resolution. The main contributor to the size of captured data (i.e., angular information) contains a high correlation that is exploited by state-of-the-art video encoders by treating the LF as a pseudo video sequence (PVS). The interpretation of LF as a single PVS restricts the encoding scheme to only utilize a single-dimensional angular correlation present in the LF data. In this paper, we present an LF compression framework that efficiently exploits the spatial and angular correlation using a multiview extension of high-efficiency video coding (MV-HEVC). The input LF views are converted into multiple PVSs and are organized hierarchically. The rate-allocation scheme takes into account the assigned organization of frames and distributes quality/bits among them accordingly. Subsequently, the reference picture selection scheme prioritizes the reference frames based on the assigned quality. The proposed compression scheme is evaluated by following the common test conditions set by JPEG Pleno. The proposed scheme performs 0.75 dB better compared to state-of-the-art compression schemes and 2.5 dB better compared to the x265-based JPEG Pleno anchor scheme. Moreover, an optimized motionsearch scheme is proposed in the framework that reduces the computational complexity (in terms of the sum of absolute difference [SAD] computations) of motion estimation by up to 87% with a negligible loss in visual quality (approximately 0.05 dB).INDEX TERMS Compression, light field, MV-HEVC, plenoptic.

The JPEG Pleno Light Field Coding Standard 4D-Transform Mode: How to Design an Efficient 4D-Native Codec

“…Virtual reference frames predicted from reconstructed neighbouring frames, using a deep neural network, are used as additional reference frames in a modified MV-HEVC hierarchical coding structure. In [28], MV-HEVC is modified by introducing a hierarchical organization of the input LF views, where each frame is located to a specific level based on its location in the 2D view (PI) matrix.…”

“…• HEVC anchor (HEVC), which corresponds to the pseudo-video like HEVC coding of the 13×13 views in a IPPP serpentine scanning pattern (left-right, topdown) using the x265 [54] implementation [11]; this is the solution taken as reference for all BD-Rate values in this paper. • 4D-Prediction mode (WaSP), which corresponds to the other LF coding mode specified in the JPEG Pleno Light Field Coding standard [10], using the JPEG Pleno VM2.1 Verification Model software [51]; • HEVC-HR, which corresponds to the LF codec specified in [23] and reviewed in Section II; • MV-HEVC-based, which corresponds to the LF codec specified in [28] and reviewed in Section II; • VVC-serpentine, which corresponds to the pseudovideo-like VVC coding of the 13×13 views in a IPPP serpentine (left-right, top-down) scanning pattern [11]; while more optimized scannings could be used, serpentine scanning is used here to follow the JPEG Pleno CTC recommendations as for the HEVC anchor. Due to constraints of the VVC reference software [55], and to allow a fairer comparison, the LF views were cropped to 624× 432.…”

Local Low Rank Approximation With a Parametric Disparity Model for Light Field Compression

“…Another category of approaches consists in first extracting sub-aperture images (or views) from the raw plenoptic data, that are then coded as a pseudo-video sequence using HEVC inter coding [11], the latest JEM coder [20], or multi-view HEVC based coding scheme [21] [4]. The authors in [22], propose a coding order and a prediction structure inspired from those used in the multi-view coding (MVC) standard.…”

“…In this paper, we focus on the problem of compression of dense light fields, as those captured by plenoptic cameras, which represent very large volumes of highly redundant data. While a number of methods have already been published in the literature aiming at adapting standardized solutions (in particular HEVC) to light field data as in [1]- [4], here we focus on the problem of reducing the angular dimensions of light fields with a low rank approximation method. A homographybased low-rank approximation method called HLRA has been shown to give very good light field compression performances in [5].…”

A Flexible Coding Scheme Based on Block Krylov Subspace Approximation for Light Field Displays with Stacked Multiplicative Layers

“…The majority of existing light field coding approaches are not directly applicable for multi-layered displays. Several coding approaches extract the SAIs of the light field and interpret them as a pseudo video sequence [ 23 , 24 , 25 , 26 , 27 , 28 ]. They utilize existing video encoders, like HEVC [ 29 ] or MV-HEVC, for inter and intra-frame hybrid prediction.…”