FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling

Abstract

Current deep video quality assessment (VQA) methods are usually with high computational costs when evaluating high-resolution videos. This cost hinders them from learning better video-quality-related representations via end-to-end training. Existing approaches usually consider naive sampling to reduce the computational cost, such as resizing and cropping. However, they obviously corrupt quality-related information in videos and are thus not optimal to learn good representations for VQA. Therefore, there is an eager need to design a new quality-retained sampling scheme for VQA. In this paper, we propose Grid Mini-patch Sampling (GMS), which allows consideration of local quality by sampling patches at their raw resolution and covers global quality with contextual relations via mini-patches sampled in uniform grids. These mini-patches are spliced and aligned temporally, named as fragments. We further build the Fragment Attention Network (FANet) specially designed to accommodate fragments as inputs. Consisting of fragments and FANet, the proposed FrAgment Sample Transformer for VQA (FAST-VQA) enables efficient end-to-end deep VQA and learns effective video-qualityrelated representations. It improves state-of-the-art accuracy by around 10% while reducing 99.5% FLOPs on 1080P high-resolution videos. The newly learned video-quality-related representations can also be transferred into smaller VQA datasets and boost the performance on these scenarios. Extensive experiments show that FAST-VQA has good performance on inputs of various resolutions while retaining high efficiency. We publish our code at https://github.com/timothyhtimothy/FAST-VQA.

Exploring the Effectiveness of Video Perceptual Representation in Blind Video Quality Assessment

“…RAPIQUE [45] combines and exploits the advantages of both quality-aware scene statistics features and semantics-aware deep convolutional features, designing a general and efficient spatial and temporal bandpass statistical model for VQA. Instead of extracting handcraft features, deep VQA methods [4,19,48,49,56,58] use CNNs to extract rich semantic features and run regression on the extracted features to predict video quality. For example, MLSP-FF [8] extracts frame-wise features with Inception-ResNetv2 model [42] and some works [48,56,58] introduce 3D-CNN instead of 2D-CNN to extract more efficient temporal features.…”