ELF-VC: Efficient Learned Flexible-Rate Video Coding

Rippel, Oren; Anderson, Alexander G.; Tatwawadi, Kedar; Nair, Sanjay; Lytle, Craig; Bourdev, Lubomir

doi:10.1109/iccv48922.2021.01421

Cited by 64 publications

(36 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While research on general neural video compression already features a rich body of literature (e.g. [21,8,3,13,9]), there is only a handful of works on neural face video compression [11,6,17]. Oquab et al [11] study the suitability of different talking head synthesis approaches for compression, targeting a mobile low-resource scenario.…”

Section: Related Workmentioning

confidence: 99%

Neural Face Video Compression using Multiple Views

Volokitin¹,

Brugger²,

Ali³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent advances in deep generative models led to the development of neural face video compression codecs that use an order of magnitude less bandwidth than engineered codecs. These neural codecs reconstruct the current frame by warping a source frame and using a generative model to compensate for imperfections in the warped source frame. Thereby, the warp is encoded and transmitted using a small number of keypoints rather than a dense flow field, which leads to massive savings compared to traditional codecs. However, by relying on a single source frame only, these methods lead to inaccurate reconstructions (e.g. one side of the head becomes unoccluded when turning the head and has to be synthesized). Here, we aim to tackle this issue by relying on multiple source frames (views of the face) and present encouraging results.

show abstract

Section: Related Workmentioning

confidence: 99%

Neural Face Video Compression using Multiple Views

Volokitin¹,

Brugger²,

Ali³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To reduce motion overhead, Lin et al [24] use predictive motion coding by extrapolating a flow map predictor from the decoded flow maps. Rippel et al [32] use the flow map predictor for motion compensation and signal an incremental flow map between the resulting motion-compensated frame and the target frame. Hu et al [14] adapt, either locally or globally, the resolution of the flow map features.…”

Section: Learned Video Compressionmentioning

confidence: 99%

“…The arrival of deep learning spurs a new wave of developments in end-to-end learned image and video compression [30,9,28,15,23,32]. The seminal work [4] by Ballé et al connects for the first time the learning of an image compression system to learning a variational generative model, known as the variational autoencoder (VAE) [19].…”

Section: Introductionmentioning

confidence: 99%

“…Lately, there have been few attempts at introducing normalizing flow models [28,13] to learned image compression Inspired by the success of learned image compression, research on learned video compression is catching up quickly. However, most end-to-end learned video compression systems [26,27,24,14,32] were developed based primarily on the traditional, hybrid-based coding architecture, replacing key components, such as inter-frame prediction and residual coding, with neural networks. The idea of residual coding is to encode a target frame x t by coding the prediction residual r t = x t − x c between x t and its motion-compensated reference frame x c .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CANF-VC: Conditional Augmented Normalizing Flows for Video Compression

Ho¹,

Chang²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper presents an end-to-end learning-based video compression system, termed CANF-VC, based on conditional augmented normalizing flows (CANF). Most learned video compression systems adopt the same hybrid-based coding architecture as the traditional codecs. Recent research on conditional coding has shown the sub-optimality of the hybrid-based coding and opens up opportunities for deep generative models to take a key role in creating new coding frameworks. CANF-VC represents a new attempt that leverages the conditional ANF to learn a video generative model for conditional inter-frame coding. We choose ANF because it is a special type of generative model, which includes variational autoencoder as a special case and is able to achieve better expressiveness. CANF-VC also extends the idea of conditional coding to motion coding, forming a purely conditional coding framework. Extensive experimental results on commonly used datasets confirm the superiority of CANF-VC to the state-of-the-art methods. The source code of CANF-VC is available at https://github.com/NYCU-MAPL/CANF-VC.

show abstract

“…The vast majority of published video compression research papers provide results using the objective metrics Peak signal-to-noise ratio (PSNR) [2], structural similarity index measure (SSIM) [3], multi-scale SSIM (MS-SSIM) [4], and Video Multi-Method Assessment Fusion (VMAF) [5], e.g., [6,7,8]. However, it has been shown that PSNR, SSIM, and MS-SSIM are not well correlated to subjective opinion [5,9].…”

Section: Introductionmentioning

confidence: 99%

A crowdsourced implementation of ITU-T P.910

Naderi¹,

Cutler²

2022

Preprint

View full text Add to dashboard Cite

The gold standard for measuring video quality is the subjective test, and the most prevalent is the ITU-T P.910, a labbased subjective standard in use for the past two decades. However, in practice using P.910 is slow, expensive, and requires a lab, which all create barriers to usage. As a result, most research papers that need to measure video quality don't use P.910 but rather metrics that are not well correlated to subjective opinion. We provide an open-source extension of P.910 based on crowdsourcing principles which address the speed, usage cost, and barrier to usage issues. We implement Absolute Category Rating (ACR), ACR with hidden reference (ACR-HR), and Degradation Category Rating (DCR). It includes rater, environment, hardware, and network qualifications, as well as gold and trapping questions to ensure quality. We have validated that the implementation is both accurate and highly reproducible compared to existing P.910 lab studies.

show abstract

ELF-VC: Efficient Learned Flexible-Rate Video Coding

Cited by 64 publications

References 19 publications

Neural Face Video Compression using Multiple Views

Neural Face Video Compression using Multiple Views

CANF-VC: Conditional Augmented Normalizing Flows for Video Compression

A crowdsourced implementation of ITU-T P.910

Contact Info

Product

Resources

About