The FACTS model of speech motor control: Fusing state estimation and task-based control

Abstract

We present a new computational model of speech motor control: the Feedback-Aware Control of Tasks in Speech or FACTS model. FACTS employs a hierarchical state feedback control architecture to control simulated vocal tract and produce intelligible speech. The model includes higher-level control of speech tasks and lower-level control of speech articulators. The task controller is modeled as a dynamical system governing the creation of desired constrictions in the vocal tract, after Task Dynamics. Both the task and articulatory controllers rely on an internal estimate of the current state of the vocal tract to generate motor commands. This estimate is derived, based on efference copy of applied controls, from a forward model that predicts both the next vocal tract state as well as expected auditory and somatosensory feedback. A comparison between predicted feedback and actual feedback is then used to update the internal state prediction. FACTS is able to qualitatively replicate many characteristics of the human speech system: the model is robust to noise in both the sensory and motor pathways, is relatively unaffected by a loss of auditory feedback but is more significantly impacted by the loss of somatosensory feedback, and responds appropriately to externally-imposed alterations of auditory and somatosensory feedback. The model also replicates previously hypothesized trade-offs between reliance on auditory and somatosensory feedback and shows for the first time how this relationship may be mediated by acuity in each sensory domain. These results have important implications for our understanding of the speech motor control system in humans.

On the timing and coordination of articulatory movements: Historical perspectives and current theoretical challenges

“…But how does this knowledge arise? One possible answer lies with a recent extension of Articulatory Phonology called Feedback‐Aware Control of Tasks (FACTS; Parrell, Ramanarayanan, Nagarajan, & Houde, 2019). According to FACTS, the motor system estimates an internal model of the vocal tract's state, based on the driving signals generated.…”

“…Unlike the dominant planning models reviewed above, DIVA incorporates planning targets defined in articulatory and acoustic spaces. However, the FACTS framework (Parrell et al, 2019) can also account for acoustic perturbation data, despite assuming only articulatory targets. As mentioned earlier, acoustic feedback is used to adjust the motor system's forward model of articulatory posture.…”

“…Adaptation to acoustic perturbation requires that speech plans flexibly respond to acoustic feedback. It remains ambiguous whether this response involves explicit acoustic level goals, as in DIVA (Guenther, 1994, 1995), or whether acoustic feedback is converted to an error signal in articulatory goal space, as in FACTS (Parrell et al, 2019). However, we also reviewed evidence that some participants in the delayed naming task strategically “pause” articulation just before the first acoustic consequence (Kawamoto et al, 2008).…”

Auditory representation of learned sound sequences in motor regions of the macaque brain

“… 11 and 12 for review). Building on early, pioneering work ( 13 – 15 ), studies incorporating optimality principles ( 16 ), such as optimal state estimation ( 17 , 18 ) and state feedback control ( 19 – 22 ), have been successful in capturing important dynamics of human vocal communication. In fact, the emergence of internal models in brain systems for auditory–motor processing could be considered one of the key events during primate evolution that ultimately enabled speech in humans ( 23 , 24 ).…”

Speakers are able to categorize vowels based on tongue somatosensation

“…The sensory correlates of speech production are therefore both auditory (associated with the spectrotemporal characteristics of sounds) and somatosensory (related to the position or shape of the vocal tract articulators and to contacts between articulators and vocal tract boundaries). While the propagation of sounds is the means through which linguistic information passes between speakers and listeners, most recent models of speech motor control [DIVA (1), FACTS (2), HSFC (3), Bayesian GEPPETO (4), and ACT (5)] posit that both auditory and somatosensory information is used during speech production for the planning, monitoring, and correction of movements. The crucial role of auditory information has been documented in experiments using bite blocks or lip tubes (6,7), in which articulation has been shown to be reorganized in order to preserve the acoustical characteristics of speech.…”