Subjective Video Quality Assessment of HTTP Adaptive Streaming Technologies

Abstract

HTTP adaptive steaming (HAS) is becoming ubiquitous as a reliable method of delivering video content over the open Internet to a variety of devices fromBy introducing various network impairments, we were able to demonstrate that video quality gracefully declined until network conditions became too harsh. As a baseline, we compared them with the Microsoft Mediaroom* Internet Protocol television (IPTV) solution currently deployed by service providers. The HD content was encoded at 6.0 Mb/s.The second approach was to create several clips in which exaggerated errors (such as visual artifacts) typical of HAS were introduced on purpose. These were shown to a much wider audience than we could bring into our test labs. Specifically, we used 150 volunteers from the Alcatel-Lucent Youth Lab to offer their subjective assessment of several pieces of content.From these experiments, we conclude that HAS is suitable for delivery of quality video over unmanaged IntroductionHTTP adaptive steaming (HAS) is becoming ubiquitous as a reliable method of streaming video content over unmanaged networks to a sizable number of classes of devices including smartphones, tablets, laptops and desktop personal computers (PCs) connected to small and medium-sized screens. It achieves this by dynamically adjusting the video quality to match the available bandwidth. This paper reports on two approaches to end user subjective testing of response to these changes in quality.The first approach was to invite a panel of viewers into our test lab facility. They were asked to rate high definition (HD) video content displayed by commercially available HTTP adaptive streaming solutions including Microsoft Smooth Streaming, Apple HTTP Live Streaming, and Adobe Dynamic Streaming.

On subjective quality assessment of adaptive video streaming via crowdsourcing and laboratory based experiments

“…Robinson et al [30] conducted a subjective study to evaluate user experience on HAS based video streaming under constrains of varying bandwith, latency, and video-data losses. Various observations were made including the preference of constant bitrate over frequently changing bitrate and a slow drop to the lower quality over oscillating bitrates.…”

Quality of experience and HTTP adaptive streaming: A review of subjective studies

“…Switching frequency also impacts QoE, as highly frequent quality variations may be annoying to the user [8][9][10][11][12][13]. Additionally, there is an interaction between frequency and amplitude of switches.…”

“…Many studies have addressed the "to switch or not to switch" dilemma [8][9][10][11][12][13][14][15][16], with the result that constant quality is usually preferred to varying quality. In particular: short-term spikes may severely degrade the perceived quality [12]; constant (even lower) quality is preferred to decreasing quality (from higher to lower) [10]; constant or nearly constant quality is preferable to frequently varying quality, even if mean quality is lower [9,13]. In general, providing a bitrate as high as possible does not necessarily lead to the highest QoE [11].…”

Quality of Experience of adaptive video streaming: Investigation in service parameters and subjective quality assessment methodology

“…Studied presented in [3,8] revealed that the effect of spatial and temporal switching varies depending on the content type. This was found to be even true for switches with the same amplitude so that it is difficult to spot quality oscillation when there are frequent scene changes while in steady shots they are more noticeable.…”

“…The idea behind designing this method was to simulate realistic viewing conditions by using longer sequences so the observers become more engaged to the content as they would be in real life, rather than focusing on detecting impairments, which can happen using traditional methodologies with the short and less entertaining test videos. In this sense, another study was presented in [8], where the effect of different switching strategies were investigated, such that each individual adaptation event was evaluated in a sequence of around 2 min.…”