NDNetGaming - development of a no-reference deep CNN for gaming video quality prediction

Abstract

Gaming video streaming services are growing rapidly due to new services such as passive video streaming of gaming content, e.g. Twitch.tv, as well as cloud gaming, e.g. Nvidia GeForce NOW and Google Stadia. In contrast to traditional video content, gaming content has special characteristics such as extremely high and special motion patterns, synthetic content and repetitive content, which poses new opportunities for the design of machine learning-based models to outperform the state-of-the-art video and image quality approaches for this special computer generated content. In this paper, we train a Convolutional Neural Network (CNN) based on an objective quality model, VMAF, as ground truth and fine-tuned it based on subjective image quality ratings. In addition, we propose a new temporal pooling method to predict gaming video quality based on frame-level predictions. Finally, the paper also describes how an appropriate CNN architecture can be chosen and how well the model performs on different contents. Our result shows that among four popular network architectures that we investigated, DenseNet performs best for image quality assessment based on the training dataset. By training the last 57 convolutional layers of DenseNet based on VMAF values, we obtained a high performance model to predict VMAF of distorted frames of video games with a Spearman's Rank correlation (SRCC) of 0.945 and Root Mean Score Error (RMSE) of 7.07 on the image level, while achieving a higher performance on the video level leading to a SRCC of 0.967 and RMSE of 5.47 for the KUGVD dataset. Furthermore, we fine-tuned the model based on subjective quality ratings of images from gaming content which resulted in a SRCC of 0.93 and RMSE of 0.46 using one-hold-out cross validation. Finally, on the video level, using the proposed pooling method, the model achieves a very good performance indicated by a SRCC of 0.968 and RMSE of 0.30 for the used gaming video dataset.

DEMI: Deep Video Quality Estimation Model using Perceptual Video Quality Dimensions

“…using VMAF as groundtruth) and performed well with a Pearson Correlation (PCC) of 0.97 with VMAF. Utke et al [8] proposed a deep learning based gaming video quality metrics which outperforms the existing signal based video quality metrics.…”

“…and hence, requires an anchor dataset to deal with this bias which is missing in such cases. For training, we use the annotated frames using an objective, full-reference video quality metric called VMAF, as was done in [8]. The selection of VMAF is due to high performance of the metric for different types of content (including gaming content [16] when compression artifacts are present.…”

“…As the underlying CNN architecture, we chose the lightweight, DenseNET-121 architecture [17], which has been shown to perform well for image quality estimation tasks [8]. Selection of the DenseNET-121 is also considering the fact that the model is of very low complexity, with almost 8 million parameters (e.g.…”

“…Selection of the DenseNET-121 is also considering the fact that the model is of very low complexity, with almost 8 million parameters (e.g. compared to ResNet50 with 25 million parameters), while reaching high accuracy for quality prediction problems [8]. In order to let DenseNET-121 learn a regression task (instead of the originally trained classification task), the fully connected layer at the end of the CNN was removed.…”

“…For DenseNET-121, we used the implementation available in [18]. For training the model, we crop nine non-overlapping patches, each of size 299 × 299 instead of default DenseNet patch size of 224 × 224, as recommended in [8], from each frame for training the model based on the VMAF. In its entirety, we used over 200k frames and their respective VMAF scores as the target.…”