CVD2014—A Database for Evaluating No-Reference Video Quality Assessment Algorithms

Abstract

In this paper, we present a new video database: CVD2014-Camera Video Database. In contrast to previous video databases, this database uses real cameras rather than introducing distortions via post-processing, which results in a complex distortion space in regard to the video acquisition process. CVD2014 contains a total of 234 videos that are recorded using 78 different cameras. Moreover, this database contains the observer-specific quality evaluation scores rather than only providing mean opinion scores. We have also collected open-ended quality descriptions that are provided by the observers. These descriptions were used to define the quality dimensions for the videos in CVD2014. The dimensions included sharpness, graininess, color balance, darkness, and jerkiness. At the end of this paper, a performance study of image and video quality algorithms for predicting the subjective video quality is reported. For this performance study, we proposed a new performance measure that accounts for observer variance. The performance study revealed that there is room for improvement regarding the video quality assessment algorithms. The CVD2014 video database has been made publicly available for the research community. All video sequences and corresponding subjective ratings can be obtained from the CVD2014 project page (http://www.helsinki.fi/psychology/groups/visualcognition/).

Quality Assessment of In-the-Wild Videos

“…Four relevant databases have been constructed and corresponding subjective studies have been conducted. Overall, CVD2014 [31], KoNViD-1k [12], and LIVE-Qualcomm [10] are publicly available, while LIVE-VQC [42] will be available soon. Due to the fact that we cannot access the pristine reference videos in this situation, only NR-VQA methods are applicable.…”

“…Due to the fact that we cannot access the pristine reference videos in this situation, only NR-VQA methods are applicable. Unfortunately, the evaluation of current state-of-the-art NR-VQA methods [28,35] on these video databases shows a poor performance [10,23,31,42]. Existing deep learning-based VQA models are unfeasible in our problem since they either need the reference information [15,55,56] or only suit for compression artifacts [20].…”

“…Databases. There are four databases constructed for our problem: LIVE Video Quality Challenge Database (LIVE-VQC) [42], Konstanz Natural Video Database (KoNViD-1k) [12], LIVE-Qualcomm Mobile In-Capture Video Quality Database (LIVE-Qualcomm) [10], and Camera Video Database (CVD2014) [31]. The latter three are now publicly available, while the first one is not accessible now.…”

“…There is an essential difference between in-the-wild videos and synthetically-distorted videos, i.e., the former contains a mass of content and may suffer from complex mixed real-world distortions that are temporally heterogeneous. On account of this, current state-of-the-art video quality assessment (VQA) methods (e.g., VBLIINDS [35] and VIIDEO [28]) validated on traditional synthetic VQA databases [30,38] fail in predicting the quality of in-the-wild videos [10,23,31,42].…”

“…To demonstrate the performance of our method, we conduct experiments on three publicly available databases, i.e., KoNViD-1k [12], LIVE-Qualcomm [10] and CVD2014 [31]. Our method is compared with five state-of-the-art methods, and its superior performance is proved by the experimental results.…”