A Unified Framework for Compression and Compressed Sensing of Light Fields and Light Field Videos

Miandji, Ehsan; Hajisharif, Saghi; Unger, Jonas

doi:10.1145/3269980

Cited by 28 publications

(35 citation statements)

References 82 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has been shown that data driven dictionary learning algorithms outperform analytical basis functions in terms of compression ratio and reconstruction quality [AEB06, MBPS09, MKU13, SPRE18, APF17]. Miandji et al [MHU19] presented a dictionary learning method for compression and compressed sensing of light field videos. The training procedure leads to an Aggregate Multidimensional Dictionary Ensemble (AMDE).…”

Section: Light Field Pipeline Design and Overviewmentioning

confidence: 99%

“…natural images). Finally, the size of AMDE is orders of magnitude smaller than overcomplete dictionaries such as K‐SVD and its variants [MHU19]. Based on the above arguments, we use AMDE as the underlying representation for light field video data points.…”

Section: Light Field Pipeline Design and Overviewmentioning

confidence: 99%

“…In particular, we will briefly explain the training and encoding stages of AMDE in sections 3.1 and 3.2, respectively. The reader is referred to [MHU19] for further details about the AMDE algorithm. Section 3.3 will present an algorithm for reducing the number of dictionaries in AMDE by a user defined factor with negligible reduction in reconstruction quality, which leads to faster encoding using the GPU.…”

Section: Sparse Representation Of Light Field Videosmentioning

confidence: 99%

“…Afterwards, we introduce sparsity by nullifying elements of each 𝒮 ( i,k ) until a user‐defined threshold, δt, on the representation error is reached. We also set a user‐defined upper‐bound on sparsity, τ t , so that the data points in the test set that are not intrinsically sparse get a fixed number of nonzero coefficients, see [MHU19] for more details. Once the sparse coefficients of a data point for all dictionaries in AMDE are computed, we pick the dictionary corresponding to the most sparse coefficients with the least amount of error.…”

Section: Sparse Representation Of Light Field Videosmentioning

confidence: 99%

“…A key aspect in our system is that we encode both color information and depth in the same dictionary representation, leading to efficient reconstruction algorithms supporting parallel computation and GPU implementation. We introduce the concept of dictionary pruning leading to a significantly more efficient encoding algorithm as compared to [MHU19]. Finally, we perform a study investigating how image noise in the light fields affect the dictionary training and encoding and propose a denoising scheme to increase the compression ratio and reconstruction quality during rendering.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Light Field Video Compression and Real Time Rendering

Hajisharif

Miandji

Larsson

et al. 2019

Computer Graphics Forum

Self Cite

View full text Add to dashboard Cite

Light field imaging is rapidly becoming an established method for generating flexible image based description of scene appearances. Compared to classical 2D imaging techniques, the angular information included in light fields enables effects such as post‐capture refocusing and the exploration of the scene from different vantage points. In this paper, we describe a novel GPU pipeline for compression and real‐time rendering of light field videos with full parallax. To achieve this, we employ a dictionary learning approach and train an ensemble of dictionaries capable of efficiently representing light field video data using highly sparse coefficient sets. A novel, key element in our representation is that we simultaneously compress both image data (pixel colors) and the auxiliary information (depth, disparity, or optical flow) required for view interpolation. During playback, the coefficients are streamed to the GPU where the light field and the auxiliary information are reconstructed using the dictionary ensemble and view interpolation is performed. In order to realize the pipeline we present several technical contributions including a denoising scheme enhancing the sparsity in the dataset which enables higher compression ratios, and a novel pruning strategy which reduces the size of the dictionary ensemble and leads to significant reductions in computational complexity during the encoding of a light field. Our approach is independent of the light field parameterization and can be used with data from any light field video capture system. To demonstrate the usefulness of our pipeline, we utilize various publicly available light field video datasets and discuss the medical application of documenting heart surgery.

show abstract