Vowel decoding from single‐trial speech‐evoked electrophysiological responses: A feature‐based machine learning approach

Abstract

IntroductionScalp‐recorded electrophysiological responses to complex, periodic auditory signals reflect phase‐locked activity from neural ensembles within the auditory system. These responses, referred to as frequency‐following responses (FFRs), have been widely utilized to index typical and atypical representation of speech signals in the auditory system. One of the major limitations in FFR is the low signal‐to‐noise ratio at the level of single trials. For this reason, the analysis relies on averaging across thousands of trials. The ability to examine the quality of single‐trial FFRs will allow investigation of trial‐by‐trial dynamics of the FFR, which has been impossible due to the averaging approach.MethodsIn a novel, data‐driven approach, we used machine learning principles to decode information related to the speech signal from single trial FFRs. FFRs were collected from participants while they listened to two vowels produced by two speakers. Scalp‐recorded electrophysiological responses were projected onto a low‐dimensional spectral feature space independently derived from the same two vowels produced by 40 speakers, which were not presented to the participants. A novel supervised machine learning classifier was trained to discriminate vowel tokens on a subset of FFRs from each participant, and tested on the remaining subset.ResultsWe demonstrate reliable decoding of speech signals at the level of single‐trials by decomposing the raw FFR based on information‐bearing spectral features in the speech signal that were independently derived.ConclusionsTaken together, the ability to extract interpretable features at the level of single‐trials in a data‐driven manner offers unchartered possibilities in the noninvasive assessment of human auditory function.

Rhythmic Temporal Expectation Boosts Neural Activity by Increasing Neural Gain

“…While recent studies have brought substantial advances in decoding auditory features, studies using discreet stimuli have focused on decoding complex features such as pitch/rate modulation based on spectral information in MEG signals (Herrmann et al, 2013b) or bistable percepts based on evoked MEG responses (Billig et al, 2018). In the domain of speech decoding, speech-evoked responses can be used to decode vowel categories (Yi et al, 2017), but typically a combination of complex spectral features is used to decode the speech envelope (Luo and Poeppel, 2007;Ng et al, 2013;de Cheveigné et al, 2018). Here, robust decoding of pure tone frequency was achieved based on relatively early M/EEG response latencies (Ͻ100 ms) evoked by very brief tones (ϳ33 ms), despite their presentation in gapless streams.…”

Biometric identification of listener identity from frequency following responses to speech

“…1) Greenberg et al, 1987). Previous research has taken advantage of this property of the FFR to decode sound categories from FFR responses (Llanos, Xie, & Chandrasekaran, 2017;Reetzke et al, 2018;Yi et al, 2017). Building upon this finding, we aimed to decode listeners from FFRs as a proxy to assess the biometric specificity of the FFR.…”

Decoding Speech and Music Stimuli from the Frequency Following Response

“…Results show that the spectral amplitude of the FFR in F0 and F1 bands can be used to correctly classify vowels with up to 70–80% accuracy (Sadeghian et al, 2015) and that spectral information related to the F2 band can be used to classify cortical evoked responses to vowels on the basis of single-trial data (Kim et al, 2014). Data acquired with an innovative combination of single-trial classification and machine-learning methods support the notion that the spectral amplitude of the FFR may be used to correctly predict vowel categorization into learned and novel vowel categories (Yi et al, 2017). Moreover, temporal information contained in the phase of theta oscillations (2–9 Hz) could correctly classify eight phonetic categories such that confusion matrices from phase and perceptual responses were not statistically distinguishable from one another (R.…”