Auxiliary stream for optimizing memory access of video decoders

Abstract

Due to the ever increasing resolution and frame rate of mainstream video sequences, memory access has become the main performance bottleneck of video decoding. To reduce the required off-chip memory, many decoders employ on-chip cache. However, they cannot distinguish whether a data block is reusable due to the lack of the information of undecoded Macro Blocks (MBs), thus often evicting reusable data from the cache and preserving non-reusable data in the cache, which will lead to a waste of off-chip memory bandwidth. In this paper, we manage to make full use of cache from a novel perspective, i.e., auxiliary bitstream. Concretely speaking, since the memory access behavior of video decoding is determined in video encoding, the encoder can pack the memory access behaviors of video decoding as auxiliary bitstream, which can inform the decoder whether a data block will be reused by future MBs. Hence, such an auxiliary stream can enable optimal management of cache. To effectively reduce the size of auxiliary bitstream, we propose an Auxiliary Prior Information Coding (APIC) method complying with the current video standards. For future video standards, we introduce a Super Block scan Order (SBO) for MB organization to further reduce the bitrate overhead of auxiliary bitstream. The above ideas are evaluated on a number of representative video sequences. The additional prior information can reduce the required off-chip memory bandwidth for motion compensation by over 35% (for a 60 kB cache), while only causing less than 2.3% bitrate increase for high definition (HD) videos.