Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

Abstract

Rules-of-thumb for noticeable and detrimental asynchrony between audio and video streams have long since been established from the contributions of several studies. Although these studies share similar findings, none have made any discernible assumptions regarding audio and video quality. Considering the use of active adaptation in present and upcoming streaming systems, audio and video will continue to be delivered in separate streams; consequently, the assumption that the rules-of-thumb hold independent of quality needs to be challenged. To put this assumption to the test, we focus on the detection, not the appraisal, of asynchrony at different levels of distortion. Cognitive psychologists use the term temporal integration to describe the failure to detect asynchrony. The term refers to a perceptual process with an inherent buffer for short asynchronies, where corresponding auditory and visual signals are merged into one experience. Accordingly, this paper discusses relevant causes and concerns with regards to asynchrony, it introduces research on audiovisual perception, and it moves on to explore the impact of audio and video quality on the temporal integration of different audiovisual events. Three content types are explored, speech from a news broadcast, music presented by a drummer, and physical action in the Multimed Tools Appl (2015) 74:345-365 form of a chess game. Within these contexts, we found temporal integration to be very robust to quality discrepancies between the two modalities. In fact, asynchrony detection thresholds varied considerably more between the different content than they did between distortion levels. Nevertheless, our findings indicate that the assumption concerning the independence of asynchrony and audiovisual quality may have to be reconsidered.

Tolerance for audiovisual asynchrony is enhanced by the spectrotemporal fidelity of the speaker’s mouth movements and speech

“…To our knowledge, there is only one report (Eg et al, 2015) that examined the influence of acoustic speech quality on perception of AV synchrony and three prior reports, all recent, that examined the effect of blurring (Eg & Behne, 2015; Eg et al, 2015; Magnotti et al, 2013). The results from these studies are mixed, but we choose to discuss two reports (Eg et al, 2015; Magnotti et al, 2013), because they provided evidence that may seem inconsistent with our hypothesis.…”

“…The results from these studies are mixed, but we choose to discuss two reports (Eg et al, 2015; Magnotti et al, 2013), because they provided evidence that may seem inconsistent with our hypothesis. We partly address them by providing theoretical explanations of the inconsistencies.…”

“…Additionally, it is related to the place and manner of articulation of consonants and the salience of tongue and lip dynamics (front/back tongue, round/unrounded lips) of vowels (Vatakis et al, 2012). One consistent finding is that individuals tolerate greater AVOA when visual events lead auditory events as opposed to the reverse (Bhat et al, 2015; Conrey & Pisoni, 2006; Eg, Griwodz, Halvorsen, & Behne, 2015; Magnotti, Ma, & Beauchamp, 2013; Miller & D’Esposito, 2005; Stevenson et al, 2013), although this effect tend to be less asymmetrical for speech stimuli (Stevenson et al, 2013). Moreover, this asymmetry effect is not observed in single-modality scenarios (auditory-auditory).…”

Subjective video quality evaluation of different content types under different impairments

“…In addition, our population also presents a heterogeneous group, with different age ranges, profiles and experience with video, while other studies tend to only take into account a biased population (with the same age range or the same profile (Eg et al, 2015)). Moreover, the large number of participants provides us with a significant sample of each group, making the results in each group statistically significant.…”

“…These factors include different problems related to the transmission process and the playback of the contents. Nevertheless, from an end-to-end point of view, the impact of different impairments on user experience is analyzed in some works, but only taking into account specific impairments (Eg, Griwodz, Halvorsen, & Behne, 2015) or with a small population (Arslan, Usman, & Shin, 2015). In this sense, many issues are assumed, such as audio preference versus video (Beerends & De Caluwe, 1999).…”