In vitro vs in vivo: does the study’s interface design influence crowdsourced video QoE?

Abstract

Evaluating the Quality of Experience (QoE) of video streaming and its influence factors has become paramount for streaming providers, as they want to maintain high satisfaction for their customers. In this context, crowdsourced user studies became a valuable tool to evaluate different factors which can affect the perceived user experience on a large scale. In general, most of these crowdsourcing studies either use, what we refer to, as an in vivo or an in vitro interface design. In vivo design means that the study participant has to rate the QoE of a video that is embedded in an application similar to a real streaming service, e.g., YouTube or Netflix. In vitro design refers to a setting, in which the video stream is separated from a specific service and thus, the video plays on a plain background. Although these interface designs vary widely, the results are often compared and generalized. In this work, we use a crowdsourcing study to investigate the influence of three interface design alternatives, an in vitro and two in vivo designs with different levels of interactiveness, on the perceived video QoE. Contrary to our expectations, the results indicate that there is no significant influence of the study’s interface design in general on the video experience. Furthermore, we found that the in vivo design does not reduce the test takers’ attentiveness. However, we observed that participants who interacted with the test interface reported a higher video QoE than other groups.

Impact of Alignment Edits on the Quality of Experience of 360° Videos

“…The participants rated three attributes of each video: overall experience, discomfort, and presence. Table 1 presents the questions and the specific scoring scales used for each of the three attributes [51], [55].…”

Towards the Effects of Alignment Edits on the Quality of Experience of 360 Videos

“…Subjective responses conformed into a 5-point scale single-item questionnaire for QoE, discomfort [64], and presence [65]. The questionnaire design was useful to avoid misinterpretations, fitting the number of stimuli in the 30minute tolerance limit [63], and facilitating the execution of reliability checks in the data [15]. Table I present questions and scales answered by participants.…”

One-to-Many Negotiation QoE Management Mechanism for End-User Satisfaction

“…More recently, several works have started to rely on crowd-sourcing to measure and assess QoE [29]- [31]. For instance, in [29] the authors discuss how crowd-sourcing can support vendors, operators, and regulators to determine the QoE in the context of the new 5G networks that enable various new applications and network architectures.…”