A quest for an Internet video quality-of-experience metric

Abstract

An imminent challenge that content providers, CDNs, thirdparty analytics and optimization services, and video player designers in the Internet video ecosystem face is the lack of a single "gold standard" to evaluate different competing solutions. Existing techniques that describe the quality of the encoded signal or controlled studies to measure opinion scores do not translate directly into user experience at scale. Recent work shows that measurable performance metrics such as buffering, startup time, bitrate, and number of bitrate switches impact user experience. However, converting these observations into a quantitative quality-of-experience metric turns out to be challenging since these metrics are interrelated in complex and sometimes counter-intuitive ways, and their relationship to user experience can be unpredictable. To further complicate things, many confounding factors are introduced by the nature of the content itself (e.g., user interest, genre). We believe that the issue of interdependency can be addressed by casting this as a machine learning problem to build a suitable predictive model from empirical observations. We also show that setting up the problem based on domain-specific and measurement-driven insights can minimize the impact of the various confounding factors to improve the prediction performance.

Understanding the impact of network dynamics on mobile video user engagement

“…Our results show that it can predict whether a video streaming session is abandoned or skipped with more than 87% accuracy by observing only the initial 10 seconds. Our model achieves significantly better accuracy than prior models that require video service provider logs [7,8], while only using standard radio network statistics and/or TCP/IP headers readily available to network operators. Paper Organization: The rest of this paper is organized as follows.…”

“…However, after the 5% completion mark, the distribution is fairly uniform until the 80% completion mark. The initial modality in the distribution indicates abandonment that is likely either because users tend to sample videos [7] or due to longer join times [9]. The later modality in the distribution (excluding the 100% completion mark) indicates abandonment that is likely either because users lose interest in the content (e.g., due to video closing credits) or because shorter videos achieve higher download completion due to aggressive initial buffering.…”

“…They quantified the impact of video streaming quality metrics on user engagement using quasi-experimental designs. In [7,8], Balachandran et al developed a QoE model using various user-side video quality metrics. Specifically, they developed a decision tree based machine learning model to predict the extent of the video watched by users.…”

“…Recent seminal work [7][8][9]15] investigated how video streaming quality influences important user engagement metrics, such as video abandonment rate. However, these studies rely on client-side instrumentation to measure video quality metrics such as buffering, startup delay, and bitrate.…”

Developing a predictive model of quality of experience for internet video

“…The reason is that Internet video introduces new effects with respect to both quality and experience. First, traditional quality indices (e.g., Peak Signal-to-Noise Ratio (PSNR) [7]) are now replaced by metrics that capture delivery-related effects such as rate of buffering, bitrate delivered, bitrate switching, and join time [3,15,20,28,36]. Second, traditional methods of quantifying experience through user opinion scores are replaced by new measurable engagement measures such as viewing time and number of visits that more directly impact content providers' business objectives [3,36].…”