Mimicking Individual Media Quality Perception with Neural Network based Artificial Observers

Abstract

The media quality assessment research community has traditionally been focusing on developing objective algorithms to predict the result of a typical subjective experiment in terms of Mean Opinion Score (MOS) value. However, the MOS, being a single value, is insufficient to model the complexity and diversity of human opinions encountered in an actual subjective experiment. In this work we propose a complementary approach for objective media quality assessment that attempts to more closely model what happens in a subjective experiment in terms of single observers and, at the same time, we perform a qualitative analysis of the proposed approach while highlighting its suitability. More precisely, we propose to model, using neural networks (NNs) , the way single observers perceive media quality. Once trained, these NNs, one for each observer, are expected to mimic the corresponding observer in terms of quality perception. Then, similarly to a subjective experiment, such NNs can be used to simulate the users’ single opinions, which can be later aggregated by means of different statistical indicators such as average, standard deviation, quantiles, etc. Unlike previous approaches that consider subjective experiments as a black box providing reliable ground truth data for training, the proposed approach is able to consider human factors by analyzing and weighting individual observers. Such a model may therefore implicitly account for users’ expectations and tendencies, that have been shown in many studies to significantly correlate with visual quality perception. Furthermore, our proposal also introduces and investigates an index measuring how much inconsistency there would be if an observer was asked to rate many times the same stimulus. Simulation experiments conducted on several datasets demonstrate that the proposed approach can be effectively implemented in practice and thus yielding a more complete objective assessment of end users’ quality of experience.

Predicting Individual Quality Ratings of Compressed Images Through Deep Cnns-Based Artificial Observers

“…Despite the usefulness of individual opinion scores, the design of objective approaches that allow to measure the quality at the single subject level is still in its early stage [10,11]. This work aims at advancing the state-of-the-art in this direction.…”

“…Even though the output of a subjective experiment provides more information than the simple Mean Opinion Score (MOS), in particular the ratings of each individual observer, the use of ML has been mainly restricted to predicting the MOS. Only very recently, ML approaches have been applied to the problem of modeling and predicting individual opinion scores in media quality assessment [10,11].…”

“…Designing and training an ML-based model that can later be used as a substitute of an individual observer in order to automatically reproduce his/her perception of quality is a recent and very promising research direction [11]. Such a model is referred to as an Artificial Intelligence-based Observer (AIO) in the rest of this work.…”

“…Such a model is referred to as an Artificial Intelligence-based Observer (AIO) in the rest of this work. [11]. Once many AIOs (one for each subject with specific characteristics) have been trained, they allow to perform an objective assessment of the quality that resembles more to a subjective test.…”

“…In a recent journal paper [11], to train the AIOs, we employed neural networks (NNs) with a very simple architecture in order to not overfit the few available training samples and thus not to learn the noise that affects the individual opinion scores. We trained a feedforward NN with no more than three hidden layers and few neurons for each observer.…”