Two common issues in synchronized multimodal recordings with EEG: Jitter and Latency

Abstract

Multimodal recording using electroencephalogram (EEG) and other biological signals (e.g., electromyograms, eye movement, pupil information, or limb kinematics) is ubiquitous in human neuroscience research. However, the precise time alignment of data from heterogeneous sources is limited due to variable recording parameters of commercially available research devices and experimental setups. Here, we introduced the versatility of a Lab Streaming Layer (LSL)-based application for multimodal recordings of high-density EEG and other devices such as eye trackers or hand kinematics. To introduce the benefit of recording multiple devices in a time-synchronized manner, we discuss two common issues in measuring multimodal data: jitter and latency. The LSL-based system can be used for research on precise time-alignment of datasets, such as detecting stimulus-induced transient neural responses and testing hypotheses well-formulated in time by leveraging the millisecond time resolution of the system.

An ecological study protocol for the multimodal investigation of the neurophysiological underpinnings of dyadic joint action

“…Given the network-based infrastructure, LSL can be used both in wired and wireless setups. Although LSL is an open-source networking middleware, not all vendors or manufacturers support LSL for their products, and the existing vendor’s implementations often show considerable differences and limitations in stream format timing (i.e., time stamp), accuracy, temporal resolution, and signal latency ( Iwama et al, 2022 ).…”

A method for synchronized use of EEG and eye tracking in fully immersive VR

“…Although hybrid XR headsets with integrated eye-tracking and EEG are being developed, as Neurospec's DSI-VR300 and OpenBCI's Galea (AG, 2023 ; OpenBCI, 2023 ), their cost and lack of flexibility, such as lack of support to not manufacturer-endorsed applications or high-density EEG, render them at times ill-suited for a variety of research applications. Thus, various open-source methods were developed recently to allow for synchronized multimodal recordings of research-grade and commercial products, which sometimes lack the capability of hardware synchronization via, e.g., transistor-transistor logic (TTL) (Iwama et al, 2022 ). LabStreamingLayer (LSL) is an often-used middleware ecosystem that enables the streaming and synchronizing of multiple data streams via the network (LabStreamingLayer, 2023 ).…”

“…However, LSL can not take the hardware's intrinsic and the data transfer's delay and jitter into account. Thus, in cases where high temporal accuracy is required, those delays must be evaluated before the experiment (Artoni et al, 2018 ; Iwama et al, 2022 ). This study provides a method for measuring and understanding hardware offset and eventual limitations tailored toward an increasingly relevant combination of EEG, VR, and Eye tracking.…”

CLET: Computation of Latencies in Event-related potential Triggers using photodiode on virtual reality apparatuses

“…Many applications of electroencephalography (EEG) and event-related potentials (ERP) ( Luck, 2012 ; Nidal and Malik, 2014 ) require the use of triggers (or tagging) ( Wang et al, 2016 ; Cattan et al, 2018 ) to indicate the exact onset of presented stimuli (mostly visual or auditory events) ( Miyakoshi et al, 2021 ; Ignatious et al, 2023 ) so that the recorded physiological data can be synchronized. However, variability in various hardware and software typically lead to differences in latencies between the transmission and reception of triggers ( Wang et al, 2016 ; Cattan et al, 2021 ; Iwama et al, 2022 ). It is of importance that these latencies in triggers should not be confused with another type of latency that is present in the neural markers, such as N170, P250, N400, etc.…”

“…While past studies ( Cattan et al, 2018 ; Iwama et al, 2022 ) have highlighted shifting of ERP epochs based on computed average latencies, other studies illustrate details about setting up the triggers to overcome latency differences ( Wang et al, 2016 ; Cattan et al, 2018 ). To the best of our knowledge, only a few studies exist which describe the importance and considerations for computing the latencies in triggers using immersive VR systems ( Wang et al, 2016 ; Cattan et al, 2021 ; Iwama et al, 2022 ). In the literature ( Wang et al, 2016 ; Lees et al, 2018 ), Rapid Serial Visual Paradigm (RSVP) ( Wang et al, 2016 ; Lees et al, 2018 ) is one of the common, simple, yet effective approaches for sending triggers.…”