Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression

Li, Jiahao; Li, Bin; Lu, Yan

doi:10.1145/3503161.3547845

Cited by 41 publications

(71 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Li et al proposed learning feature domain contexts as condition. Its following works [29,50] adopt feature propagation to boost performance.…”

Section: Neural Video Compressionmentioning

confidence: 99%

“…At the same time, F t is also generated and propagated to the next frame. It is noted that, our framework is based on [29]. When compared with [29], this paper redesigns the modules to exploit diverse contexts from both temporal (Section 3.2 and 3.3) and spatial (Section 3.4) dimensions.…”

Section: Overviewmentioning

confidence: 99%

“…Inspired by the success in traditional codec, we are thinking whether we can equip NVC with the hierarchical quality structure and let NVC also enjoy the benefits. Considering the recent neural codecs [11,29] also support variable bitrate in single model, one straightforward solution is following the traditional codec and directly assigning hierarchical QPs during the inference of NVC. However, not like traditional codec uses well-defined rules to perform motion estimation and motion compensation (MEMC), NVC uses neural networks and MEMC is often in feature domain.…”

Section: Hierarchical Quality Structurementioning

confidence: 99%

“…Experiments show that our codec achieves much higher efficiency over previous SOTA NVC and pushes the compression ratio to a new height. When compared with DCVC MM22 [29], 23.5 % bitrate saving is achieved while MACs (multiply-accumulate operations) are reduced by 19.4%. Better yet, besides H.266-VTM 17.0, our codec also already outperforms ECM-5.0 (its best compression ratio configuration for low delay coding is used) in both RGB and YUV420 colorspaces, as shown in Fig.…”

Section: Introductionmentioning

confidence: 99%

“…Besides from temporal dimension, this paper also proposes increasing the spatial context diversity when encoding the latent representation. Based on recent checkerboard model [19] and dual spatial model [29], we design a quadtree-based partition to improve the distribution estimation. When compared with [19,29], the types of correlation modelling are more diverse hence the model has a larger chance to find more relevant context.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Neural Video Compression with Diverse Contexts

Li¹,

Li²,

Lü³

2023

Preprint

View full text Add to dashboard Cite

For any video codecs, the coding efficiency highly relies on whether the current signal to be encoded can find the relevant contexts from the previous reconstructed signals. Traditional codec has verified more contexts bring substantial coding gain, but in a time-consuming manner. However, for the emerging neural video codec (NVC), its contexts are still limited, leading to low compression ratio. To boost NVC, this paper proposes increasing the context diversity in both temporal and spatial dimensions. First, we guide the model to learn hierarchical quality patterns across frames, which enriches long-term and yet highquality temporal contexts. Furthermore, to tap the potential of optical flow-based coding framework, we introduce a group-based offset diversity where the cross-group interaction is proposed for better context mining. In addition, this paper also adopts a quadtree-based partition to increase spatial context diversity when encoding the latent representation in parallel. Experiments show that our codec obtains 23.5% bitrate saving over previous SOTA NVC. Better yet, our codec has surpassed the under-developing next generation traditional codec/ECM in both RGB and YUV420 colorspaces, in terms of PSNR. The codes are at https://github.com/microsoft/DCVC.

show abstract

“…Li et al proposed learning feature domain contexts as condition. Its following works [29,50] adopt feature propagation to boost performance.…”

Section: Neural Video Compressionmentioning

confidence: 99%

Section: Overviewmentioning

confidence: 99%

Section: Hierarchical Quality Structurementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%