Modeling speech imitation and ecological learning of auditory-motor maps

Abstract

Classical models of speech consider an antero-posterior distinction between perceptive and productive functions. However, the selective alteration of neural activity in speech motor centers, via transcranial magnetic stimulation, was shown to affect speech discrimination. On the automatic speech recognition (ASR) side, the recognition systems have classically relied solely on acoustic data, achieving rather good performance in optimal listening conditions. The main limitations of current ASR are mainly evident in the realistic use of such systems. These limitations can be partly reduced by using normalization strategies that minimize inter-speaker variability by either explicitly removing speakers' peculiarities or adapting different speakers to a reference model. In this paper we aim at modeling a motor-based imitation learning mechanism in ASR. We tested the utility of a speaker normalization strategy that uses motor representations of speech and compare it with strategies that ignore the motor domain. Specifically, we first trained a regressor through state-of-the-art machine learning techniques to build an auditory-motor mapping, in a sense mimicking a human learner that tries to reproduce utterances produced by other speakers. This auditory-motor mapping maps the speech acoustics of a speaker into the motor plans of a reference speaker. Since, during recognition, only speech acoustics are available, the mapping is necessary to “recover” motor information. Subsequently, in a phone classification task, we tested the system on either one of the speakers that was used during training or a new one. Results show that in both cases the motor-based speaker normalization strategy slightly but significantly outperforms all other strategies where only acoustics is taken into account.

Listening to speech recruits specific tongue motor synergies as revealed by transcranial magnetic stimulation and tissue-Doppler ultrasound imaging

“…From a computational perspective, the role of the motor system can be that of compensating for the increased ambiguity of the stimuli [29,30]. Subjects that are more impaired by increasing levels of noise should be in principle relying less on a motor compensation strategy.…”

The sensorimotor and social sides of the architecture of speech

“…However, human communities have richer social structures than other animals, which might have favored an open-ended instrumental use of vocal production besides ritualized display. The importance of this skill might have led to a greater investment of parental time in teaching and, we propose, to advanced forms of “tutor learning” (Canevari et al 2013). Of note, a so-called pedagogical learning environment (Csibra & Gergely 2011) might have afforded specialized teaching strategies that could be uniquely human and that greatly improve on imitation and self-teaching learning methods.…”

Integrating articulatory data in deep neural network-based acoustic modeling

“…Other studies have shown that measured articulatory data can improve speaker-independent neural network phone classifiers (Castellini et al, 2011;Canevari et al, 2013b), BN-HMM (Markov et al, 2006) and GMM-HMM phone recognizers. However the utility of measured articulatory data is largely limited by the fact that recording articulatory data is much more difficult than simply recording the audio of a speaker.…”

“…When performing the AAM (learned on msak0) on mngu0 acoustics we try to recover the AFs of a speaker from other's speech acoustics. Table 5 shows that, despite such attempt produces poor reconstruction (Ghosh and Narayanan, 2011;Canevari et al, 2013b), reconstructed AFs can still reduce flPCE and PER. Note that since the phone sets of the two datasets are different we had to map the mngu0 phone set onto the msak0 phone set.…”