Temporal precision and accuracy of audio-visual stimuli in mixed reality systems

Abstract

Mixed Reality (MR) techniques, such as Virtual (VR) and Augmented Reality (AR), are gaining popularity as a new methodology for neuroscience and psychology research. In studies involving audiovisual stimuli, it is crucial to have MR systems that can deliver these bimodal stimuli with controlled timing between the onset of each modality. However, the extent to which modern MR setups can achieve the necessary precision and accuracy of audiovisual stimulus onset asynchronies (SOAs) remains largely unknown. The objective of this study is to systematically evaluate the lag and variability between the auditory and visual onset of audiovisual stimuli produced on popular modern MR head-mounted displays (HMDs) from Meta, Microsoft, HTC, and Varjo in conjunction with commonly used development environments such as Unity and the Unreal Engine. To accomplish this, we developed a low-cost measurement system that enabled us to measure the actual SOA and its associated jitter. Our findings revealed that certain MR systems exhibited significant SOAs, with one case averaging 156.63 ms, along with jitter of up to ±11.82 ms. Using our methodology, we successfully conducted experimental calibration of a headset, achieving SOAs of −3.89 ± 1.56 ms. This paper aims to raise awareness among neuroscience researchers regarding the limitations of MR systems in delivering audiovisual stimuli without prior calibration. Furthermore, we present cost-effective methods to calibrate these systems, thereby facilitating the replication of future results.