An LSTM Based Architecture to Relate Speech Stimulus to Eeg

Monesi, Mohammad Jalilpour; Accou, Bernd; Montoya-Martínez, Jair; Francart, Tom; hamme, Hugo Van

doi:10.1109/icassp40776.2020.9054000

Cited by 36 publications

(64 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The EEG data analysed in this paper is part of a large existing dataset (Accou et al, 2020;Monesi et al, 2020) that contains EEG responses to continuous Flemish speech for young normal hearing participants (not publicly available).…”

Section: Dataset and Subjectsmentioning

confidence: 99%

Neural tracking of the fundamental frequency of the voice: the effect of voice characteristics

Canneyt

Wouters

Francart

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Traditional electrophysiological methods to study temporal auditory processing of the fundamental frequency of the voice (f0) often use unnaturally repetitive stimuli. In this study, we investigate f0 processing of meaningful continuous speech. EEG responses evoked by stories in quiet were analysed with a novel ’f0-tracking’ method that uses linear models to characterize both the strength and the spatiotemporal properties of the f0 response. Different samples of continuous speech (six stories by four speakers: two male and two female) were used to investigate stimulus effects on the f0 response. The results indicated stronger f0-tracking for the male-narrated stories compared to the female-narrated stories, for which many responses were not significant. Moreover, response strength was inversely related to the f0 of the speaker and the rate of f0 change throughout the story. These effects likely occur because phase-locking to the f0 is more challenging when the f0 is higher and/or more variable. From additional experiments, we conclude that female-narrated stories can be successfully used for continuous f0 responses but only if the rate of f0-change is low and the harmonic content is strong. Most natural female voices do not have these characteristics, so careful stimulus selection is required. The spatio-temporal analysis revealed a centrally-located peak response with latency between 7 and 12 ms for the female-narrated stories, suggestive of a brainstem source. In contrast, for male-narrated stories the analysis indicated at least two sources: one source with a latency of 13-15 ms, likely in the brainstem, and one right-dominant posterior temporal source with a latency of 23-25 ms, suggestive of the right primary auditory cortex. This adds to the growing evidence that auditory responses to frequencies in the lower range of the f0 (e.g. most male voices) do not originate solely from the brainstem but have additional cortical contributions. In conclusion, the novel f0-tracking method was successfully applied to a range of continuous speech samples and the findings on temporal processing of the f0 are in line with earlier findings from repetitive stimulus paradigms. The novel method is recommended over traditional paradigms because continuous speech is more relevant for day-to-day conversation and more engaging for the subjects, but also because the method allows for extensive analysis of both response strength and spatio-temporal characteristics.

show abstract

Section: Dataset and Subjectsmentioning

confidence: 99%

Neural tracking of the fundamental frequency of the voice: the effect of voice characteristics

Canneyt

Wouters

Francart

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The neural responses analysed in this study are part of an existing data set (Accou et al, 2020;Monesi et al, 2020) that was also used in our previous work (Van Canneyt et al, 2020b). EEG responses to continuous speech were measured for 34 young normal hearing participants, who were native Flemish (or Dutch) speakers (31 females, 3 males), with ages ranging between 18 and 24 years old (mean = 22.4 years, standard deviation = 1.4 years).…”

Section: Datasetmentioning

confidence: 99%

Enhanced neural tracking of the fundamental frequency of the voice

Canneyt

Wouters

Francart

2020

Preprint

Self Cite

View full text Add to dashboard Cite

'F0 tracking' is a novel method that investigates the neural processing of the fundamental frequency of the voice (f0) in continuous speech. Through linear modelling, a feature that reflects the stimulus f0 is predicted from the EEG data. Then, the neural response strength is evaluated through the correlation between the predicted and actual f0 feature. The aim of this study was to improve upon this 'f0 tracking' method by optimizing the f0 feature. Specifically, we aimed to design a feature that approximates the expected EEG responses to the f0. We hypothesized that this would improve neural tracking results, because the more similar the feature and the neural response are, the easier it will be to reconstruct the one from the other. Two techniques were explored: a phenomenological model to simulate neural processing in the auditory periphery and a low-pass filter to approximate the effect of more central processing on the f0 response. Since these optimizations target different aspects of the auditory system, they were also applied in a cumulative fashion. Results obtained from EEG evoked by a Flemish story in 34 subjects indicated that both the use of the auditory model and the addition of the low-pass filter significantly improved the correlations between the actual and reconstructed feature. The combination of both strategies almost doubled the mean correlation over subjects, from 0.78 to 0.13. Moreover, canonical correlation analysis with the modelled feature revealed two distinct processes contributing to the f0 response: one driven by the compound activity of auditory nerve fibers with center frequency up to 8 kHz and one driven predominantly by the auditory nerve fibers with center frequency below 1 kHz. The optimized f0 features developed in this study enhance the analysis of f0-tracking responses and facilitate future research and applications.

show abstract

“…Recently, intense activity has been devoted to stimulus-response models to gain insight into perceptual processes for speech or music (Di Liberto et al, 2015;Goossens et al, 2018;O'Sullivan et al, 2019;Broderick et al, 2019;Decruy et al, 2020;Bednar and Lalor, 2020;Zuk et al, 2020), and for BCI applications (Jaeger et al, 2020;Jalilpour Monesi et al, 2020). However, progress is slowed by the lack of reliable comparative evaluation due to the diversity of experimental conditions and data, the absence of state-of-the-art algorithms in the "line-up", and the aforementioned issue of segment mislabeling that hobbles evaluation based on the commonly-used AAD task.…”

mentioning

confidence: 99%

Auditory stimulus-response modeling with a match-mismatch task

et al. 2021

View full text Add to dashboard Cite

The relation between a continuous ongoing stimulus and the brain response that it evokes can be characterized by a stimulus-response model fit to the data. This systems-identification approach offers insight into perceptual processes within the brain, and it is also of potential practical use for devices such as Brain Computer Interfaces (BCI). The quality of the model can be quantified by measuring the fit with a regression problem, or by applying it to a classification task and measuring its performance. Here we focus on a match-mismatch task that entails deciding whether a segment of brain signal matches, via a model, the auditory stimulus that evoked it. The match-mismatch task can be used to compare performance of different stimulus-response models. We show that performance in a matchmismatch task and metrics summarizing regression accuracies can provide complementary insights in the relation between stimulus and response. Importantly, the match-mismatch task provides information about discriminatory power, making it directly applicable to BCI applications. Evaluation is performed on a freely available database, and code is available for scripts and functions to allow scrutiny of our results and facilitate comparative evaluation of future developments. 2.

show abstract

An LSTM Based Architecture to Relate Speech Stimulus to Eeg

Cited by 36 publications

References 20 publications

Neural tracking of the fundamental frequency of the voice: the effect of voice characteristics

Neural tracking of the fundamental frequency of the voice: the effect of voice characteristics

Enhanced neural tracking of the fundamental frequency of the voice

Auditory stimulus-response modeling with a match-mismatch task

Contact Info

Product

Resources

About