Robust Understanding of Robot-Directed Speech Commands Using Sequence to Sequence With Noise Injection

Abstract

This paper describes a new method that enables a service robot to understand spoken commands in a robust manner using off-the-shelf automatic speech recognition (ASR) systems and an encoder-decoder neural network with noise injection. In numerous instances, the understanding of spoken commands in the area of service robotics is modeled as a mapping of speech signals to a sequence of commands that can be understood and performed by a robot. In a conventional approach, speech signals are recognized, and semantic parsing is applied to infer the command sequence from the utterance. However, if errors occur during the process of speech recognition, a conventional semantic parsing method cannot be appropriately applied because most natural language processing methods do not recognize such errors. We propose the use of encoder-decoder neural networks, e.g., sequence to sequence, with noise injection. The noise is injected into phoneme sequences during the training phase of encoder-decoder neural network-based semantic parsing systems. We demonstrate that the use of neural networks with a noise injection can mitigate the negative effects of speech recognition errors in understanding robot-directed speech commands i.e., increase the performance of semantic parsing. We implemented the method and evaluated it using the commands given during a general purpose service robot (GPSR) task, such as a task applied in RoboCup@Home, which is a standard service robot competition for the testing of service robots. The results of the experiment show that the proposed method, namely, sequence to sequence with noise injection (Seq2Seq-NI), outperforms the baseline methods. In addition, Seq2Seq-NI enables a robot to understand a spoken command even when the speech recognition by an off-the-shelf ASR system contains recognition errors. Moreover, in this paper we describe an experiment conducted to evaluate the influence of the injected noise and provide a discussion of the results.

Learning Visual-Audio Representations for Voice-Controlled Robots

“…The usual pipeline of conventional voice-controlled robots consists of an ASR system to transcribe speech to text [13], an NLU system to map text transcripts to speaker intent [2], [14], a grounding module to associate the intent with physical entities [15], [16], and a planner to generate feasible trajectories for robot task execution [1], [17]- [19]. However, the pipeline suffers from several limitations.…”

“…However, the pipeline suffers from several limitations. First, off-the-shelf ASR systems usually operate in a general-purpose setting irrelevant to specific robotic tasks and their outputs could be inevitably out-of-context or erroneous [2], [20], [21]. Typical NLU systems, however, are trained on clean text [4].…”

“…Rule-based systems and methods based on symbolic AI have been used for grounding [22], [23]. However, developing such system requires significant labor and does not scale beyond their programmed domains [2], [3].…”

“…The main contributions of this paper are: (1) We present a novel pipeline for voice-controlled robots and robot language acquisition inspired by sensorimotor theory. (2) We build a novel representation for images and sounds, which is used as intrinsic reward function for RL. (3) Our network exhibits promising results in three robot platforms: a Kuka-IIWA, a TurtleBot3, and a Kinova Gen3.…”

Semiotically adaptive cognition: toward the realization of remotely-operated service robots for the new normal symbiotic society

“…2. Regarding the studies about language understanding of robot-directed commands involving speech recognition errors, please refer to [8] for example. 3.…”