MFQE 2.0: A New Approach for Multi-Frame Quality Enhancement on Compressed Video

Abstract

The past few years have witnessed great success in applying deep learning to enhance the quality of compressed image/video. The existing approaches mainly focus on enhancing the quality of a single frame, not considering the similarity between consecutive frames. Since heavy fluctuation exists across compressed video frames as investigated in this paper, frame similarity can be utilized for quality enhancement of low-quality frames given their neighboring high-quality frames. This task is Multi-Frame Quality Enhancement (MFQE). Accordingly, this paper proposes an MFQE approach for compressed video, as the first attempt in this direction. In our approach, we firstly develop a Bidirectional Long Short-Term Memory (BiLSTM) based detector to locate Peak Quality Frames (PQFs) in compressed video. Then, a novel Multi-Frame Convolutional Neural Network (MF-CNN) is designed to enhance the quality of compressed video, in which the non-PQF and its nearest two PQFs are the input. In MF-CNN, motion between the non-PQF and PQFs is compensated by a motion compensation subnet. Subsequently, a quality enhancement subnet fuses the non-PQF and compensated PQFs, and then reduces the compression artifacts of the non-PQF. Also, PQF quality is enhanced in the same way. Finally, experiments validate the effectiveness and generalization ability of our MFQE approach in advancing the state-of-the-art quality enhancement of compressed video.

Non-Local ConvLSTM for Video Compression Artifact Reduction

“…Lu et al [26] further incorporated quantized prediction residual in compressed code streams as strong prior knowledge, and proposed a deep Kalman filter network (DKFN) to utilize the spatiotemporal information from the preceding frames of the target frame. In addition, considering that quality of nearby compressed frames fluctuates dramatically, [13,45] proposed multi-frame quality enhancement (MFQE) and utilized motion compensation of two nearest PQFs to enhance lowquality frames. Comparing with DKFN [26], MFQE is a post-processing method and uses less prior knowledge of the compression codec, but still achieves state-of-the-art performance on HEVC compressed videos.…”

“…Quality fluctuation. Quality fluctuation is an index to evaluate the quality of a whole video [13,45]. Drastic quality fluctuation often leads to severe temporal inconsistency and degradation of QoE.…”

“…Drastic quality fluctuation often leads to severe temporal inconsistency and degradation of QoE. We evaluate fluctuation by Standard Deviation (STD) and Peak-Valley Difference (PVD) 5 of the PSNR/SSIM curves for each sequence as in [13,45]. Here, we present only the STDs and PVDs of HEVC baseline, MFQE and our method for simplicity.…”

MRS-Net: Multi-Scale Recurrent Scalable Network for Face Quality Enhancement of Compressed Videos

“…The past years have witnessed the great success of applying deep learning in quality enhancement of compressed videos [11,14,19,32,41,[48][49][50]. Among them, [11,19,32,41,48,49] use a single frame as input, whereas [14,50] take advantage of neighboring frames to enhance the current frame. Unfortunately, there is no work on face quality enhancement of compressed videos, despite [24,30] being recently proposed for enhancing the quality of compressed face images.…”

JsrNet: A Joint Sampling–Reconstruction Framework for Distributed Compressive Video Sensing

“…To reduce blocking artifacts, instead of utilizing post-processing [22], a novel network in which all measurements of blocks from one image are used simultaneously to reconstruct the full image was proposed in [25]. A multi-frame quality enhancement (MFQE) [26] approach based on LSTM networks was proposed, which enhances the quality of low-quality frames by using their neighboring high-quality frames.…”