The Influence of Subjects and Environment on Audiovisual Subjective Tests: An International Study

Abstract

International audienceTraditionally, audio quality and video quality are evaluated separately in subjective tests. Best practices within the quality assessment community were developed before many modern mobile audiovisual devices and services came into use, such as internet video, smart phones, tablets and connected televisions. These devices and services raise unique questions that require jointly evaluating both the audio and the video within a subjective test. However, audiovisual subjective testing is a relatively under-explored field. In this paper, we address the question of determining the most suitable way to conduct audiovisual subjective testing on a wide range of audiovisual quality. Six laboratories from four countries conducted a systematic study of audiovisual subjective testing. The stimuli and scale were held constant across experiments and labs; only the environment of the subjective test was varied. Some subjective tests were conducted in controlled environments and some in public environments (a cafeteria, patio or hallway). The audiovisual stimuli spanned a wide range of quality. Results show that these audiovisual subjective tests were highly repeatable from one laboratory and environment to the next. The number of subjects was the most important factor. Based on this experiment, 24 or more subjects are recommended for Absolute Category Rating (ACR) tests. In public environments, 35 subjects were required to obtain the same Student.s t-test sensitivity. The second most important variable was individual differences between subjects. Other environmental factors had minimal impact, such as language, country, lighting, background noise, wall color, and monitor calibration. Analyses indicate that Mean Opinion Scores (MOS) are relative rather than absolute. Our analyses show that the results of experiments done in pristine, laboratory environments are highly representative of those devices in actual use, in a typical user environment

On Some Biases Encountered in Modern Audio Quality Listening Tests (Part 2): Selected Graphical Examples and Discussion

“…This conclusion is in line with the observations made by psychologists [2] [17]. The above assertion is also in line with the observation made by other researchers working in a similar discipline [57]. This conclusion is also consistent with the recently issued ITU-T Recommendation P.800.2 [32] regarding the interpretation of MOS results.…”

“…The observation that the effect of the labels on the nonlinearity of the scale is much smaller than initially assumed is in line with the reports from the international multi-laboratory experiments. Despite a great likelihood of nonlinear bias in such tests, for example due to translation issues, the data obtained for the same stimuli from various laboratories are typically collinearly related [57] [58]. Interestingly, in one of the pioneering studies from the field of speech quality assessment, some researchers used the label "unsatisfactory" instead of the standard term "bad."…”

Context-Aware QoE Modelling, Measurement, and Prediction in Mobile Computing Systems

“…There were 26 male and 3 female participants working at the Department of Computer Science, Electrical and Space Engineering at Luleå University of Technology, Sweden. The choice of the number of participants for the subjective study was based on studies presented in the state-of-theart [33], [9], [41]. For example, Wu et al [41] considered sixteen users for subjective and objective tests.…”

Data Analysis in Multimedia Quality Assessment: Revisiting the Statistical Tests

“…For instance, in the context of subjective quality assessment, Ref. [14] recommends a minimum of 15 subjects while the authors in [15] suggested using at least 24 subjects for audiovisual quality measurement. Because the sampling distribution of mean is directly or indirectly used in computing the test statistics such as t, F etc., there are no requirements of normality (or any other distribution) on the data to be analyzed.…”