Efficient Volumetric Video Streaming Through Super Resolution

Zhang, Anlan; Wang, Chendong; Han, Bo; Qian, Feng

doi:10.1145/3446382.3448663

Cited by 22 publications

(3 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…b. Super-Resolution [24)]: GAN models employed to enhance the resolution of video frames, enabling better visual quality and detail during streaming. Super-resolution is a technique used to enhance the resolution of images or video frames, enabling better visual quality, sharper details, and improved perception of fine textures.…”

Section: Video Compression and Quality Enhancementmentioning

confidence: 99%

A Taxonomy for Generative Adversarial Networks in Dynamic Adaptive Streaming Over HTTP

Khan¹

2023

IJMRA

View full text Add to dashboard Cite

Generative Adversarial Networks (GANs) have emerged as a powerful tool in the field of Dynamic Adaptive Streaming over HTTP (DASH) to enhance various aspects of video streaming. This paper presents a taxonomy that categorizes the applications and techniques of GANs in the context of DASH. The taxonomy covers several key dimensions, including video generation, compression, quality enhancement, bandwidth adaptation, dynamic bitrate streaming, and cross-modal applications. Within each dimension, specific subcategories are identified to capture the diverse applications of GANs in DASH. Additionally, evaluation metrics for assessing the quality and effectiveness of GAN-based approaches are discussed. The taxonomy serves as a comprehensive framework to understand and organize the different ways in which GANs can be utilized to improve the streaming experience in DASH. By providing an organized structure, this taxonomy facilitates better understanding, comparison, and exploration of GAN-based approaches in DASH and enables researchers and practitioners to identify areas for further research and development.

show abstract

Section: Video Compression and Quality Enhancementmentioning

confidence: 99%

A Taxonomy for Generative Adversarial Networks in Dynamic Adaptive Streaming Over HTTP

Khan¹

2023

IJMRA

View full text Add to dashboard Cite

show abstract

“…The final restored video holds comparable visual experience as the original version, thus can be applied to the content distribution in different downstream tasks. Recently, optimizing the NVS pipeline has become a hot topic, including improving video encoding efficiency [9,14], reducing steaming protocol latency [62], designing adaptive bit-rate delivery [9] and optimizing SR enhancement [38,59,67]. Overall, the core objective of an efficient NVS paradigm is to improve the rate-distortion trade-off for video delivery, i.e., reducing the streaming bitrate while restricting the quality drop of restored videos.…”

Section: Neural-enhanced Video Streamingmentioning

confidence: 99%

CaDM: Codec-aware Diffusion Modeling for Neural-enhanced Video Streaming

Zhou¹,

Li²,

Guo³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent years have witnessed the dramatic growth of Internet video traffic, where the video bitstreams are often compressed and delivered in low quality to fit the streamer's uplink bandwidth. To alleviate the quality degradation, it comes the rise of Neural-enhanced Video Streaming (NVS), which shows great prospects to recover low-quality videos by mostly deploying neural super-resolution (SR) on the media server. Despite its benefit, we reveal that current mainstream works with SR enhancement have not achieved the desired rate-distortion trade-off between bitrate saving and quality restoration, due to: (1) overemphasizing the enhancement on the decoder side while omitting the codesign of encoder, (2) inherent limited restoration capacity to generate high-fidelity perceptual details, and (3) optimizing the compression-and-restoration pipeline from the resolution perspective solely, without considering color bitdepth. Aiming at overcoming these limitations, we are the first to conduct the encoder-decoder (i.e., codec) synergy by leveraging the visual-synthesis genius of diffusion models. Specifically, we present the Codec-aware Diffusion Modeling (CaDM), a novel NVS paradigm to significantly reduce streaming delivery bitrate while holding pretty higher restoration capacity over existing methods. First, CaDM improves the encoder's compression efficiency by simultaneously reducing resolution and color bit-depth of video frames. Second, CaDM provides the decoder with perfect quality enhancement by making the denoising diffusion restoration aware of encoder's resolution-color conditions. Evaluation on public cloud services with Open-MMLab benchmarks shows that CaDM significantly saves streaming bitrate by a nearly 100× reduction over vanilla H.264 and achieves much better recovery quality (e.g., FID of 0.61) over state-of-the-art neural-enhancing methods.

show abstract

“…Recently, optimizing the pipeline of NeVS has become a hot topic, such as improving the encoder-decoder (i.e., codec) efficiency (Du et al 2022;Dasari et al 2022), reducing steaming latency ) and elaborating SR enhancement (Zhang et al 2021;Nguyen et al 2022). However, its security vulnerability has not been well explored.…”

Section: Introductionmentioning

confidence: 99%

On the Robustness of Neural-Enhanced Video Streaming against Adversarial Attacks

Zhou,

Guo,

Guo

et al. 2024

AAAI

View full text Add to dashboard Cite

The explosive growth of video traffic on today's Internet promotes the rise of Neural-enhanced Video Streaming (NeVS), which effectively improves the rate-distortion trade-off by employing a cheap neural super-resolution model for quality enhancement on the receiver side. Missing by existing work, we reveal that the NeVS pipeline may suffer from a practical threat, where the crucial codec component (i.e., encoder for compression and decoder for restoration) can trigger adversarial attacks in a man-in-the-middle manner to significantly destroy video recovery performance and finally incurs the malfunction of downstream video perception tasks. In this paper, we are the first attempt to inspect the vulnerability of NeVS and discover a novel adversarial attack, called codec hijacking, where the injected invisible perturbation conspires with the malicious encoding matrix by reorganizing the spatial-temporal bit allocation within the bitstream size budget. Such a zero-day vulnerability makes our attack hard to defend because there is no visual distortion on the recovered videos until the attack happens. More seriously, this attack can be extended to diverse enhancement models, thus exposing a wide range of video perception tasks under threat. Evaluation based on state-of-the-art video codec benchmark illustrates that our attack significantly degrades the recovery performance of NeVS over previous attack methods. The damaged video quality finally leads to obvious malfunction of downstream tasks with over 75% success rate. We hope to arouse public attention on codec hijacking and its defence.

show abstract

Efficient Volumetric Video Streaming Through Super Resolution

Cited by 22 publications

References 25 publications

A Taxonomy for Generative Adversarial Networks in Dynamic Adaptive Streaming Over HTTP

A Taxonomy for Generative Adversarial Networks in Dynamic Adaptive Streaming Over HTTP

CaDM: Codec-aware Diffusion Modeling for Neural-enhanced Video Streaming

On the Robustness of Neural-Enhanced Video Streaming against Adversarial Attacks

Contact Info

Product

Resources

About