To quantify the excellence of multimedia quality, subjective evaluation experiments are conducted. In these experiments, the tradition of quantitative assessment is the most dominating, but it disregards the understanding of participants' interpretations, descriptions, and the evaluation criteria of quality. The goal of this paper is to present a new multimedia quality evaluation method called Open Profiling of Quality (OPQ) as a tool for building a deeper understanding on subjective quality. OPQ is a mixed method combining a conventional quantitative psychoperceptual evaluation and qualitative descriptive quality evaluation based on the individual's own vocabulary. OPQ is targeted for naïve participants applicable to experiments with heterogeneous and multimodal stimulus material. The paper presents the theoretical basis of the development of OPQ and overviews the methods for audiovisual quality research. We present three extensive quality evaluation studies where OPQ has been used with 120 participants. Finally, we conclude further recommendations of use of the method in quality evaluation research.
Perceptual quality evaluation experiments are used to assess the excellence of multimedia quality. However, these studies disregard qualitative experiential descriptions, interpretations, and impressions of quality. The goal of this paper is to identify general descriptive characteristics of experienced quality of 3D video on mobile devices. We conducted five studies in which descriptive data was collected after the psychoperceptual quality evaluation experiment. Qualitative semi-structured interviews and written attribute description tasks were conducted with over 90 naïve participants. The experiments contained an extensive and heterogeneous set of produced quality by varying content, level of depth, compression and transmission parameters, and audio and display factors for 3D. The results showed that quality of experience is constructed from four main components, 1) visual quality, 2) viewing experience, 3) content, and 4) quality of other modalities and their interactions.
To guide the practitioner's work in choosing between assessment methods, defined criteria to compare their benefits, costs, and limitations are needed. The goal of this paper is two-fold. Firstly, we develop an extensive comparison model to guide betweenmethod comparisons based on a literature review. The model is composed of four main criteria, called economy, excellence, implementation and assessment with a total of 24 sub-criteria. Secondly, we conduct a comparison study between two mixed methods in which a subset of criteria of the model is examined. We compare Open Profiling of Quality utilizing individuals' own vocabulary and Conventional Profiling utilizing fixed vocabulary in their descriptive evaluation. The study is conducted with naïve participants with varying 3D video qualities on a mobile device. The results compare both methods on a subset of comparison criteria and show that operationalization of the developed comparison model can provide a tool for holistic methods comparison.
The Open Profiling of Quality (OPQ) is a mixed methods approach combining a conventional quantitative psychoperceptual evaluation and qualitative descriptive quality evaluation based on naïve participants' individual vocabulary. The method targets evaluation of heterogeneous and multimodal stimulus material. The current OPQ data collection procedure provides a rich pool of data, but full benefit of it has neither been taken in the analysis to build up completeness in understanding the phenomenon under the study nor has the procedure in the analysis been probed with alternative methods. The goal of this paper is to extend the original OPQ method with advanced research methods that have become popular in related research and the component model to be able to generalize individual attributes into a terminology of Quality of Experience. We conduct an extensive subjective quality evaluation study for 3D video on mobile device with heterogeneous stimuli. We vary factors on content, media (coding, concealments, and slice modes), and transmission levels (channel loss rate). The results showed that advanced procedures in the analysis cannot only complement each other but also draw deeper understanding on Quality of Experience.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.