EarCommand

Abstract

Intelligent speech interfaces have been developing vastly to support the growing demands for convenient control and interaction with wearable/earable and portable devices. To avoid privacy leakage during speech interactions and strengthen the resistance to ambient noise, silent speech interfaces have been widely explored to enable people's interaction with mobile/wearable devices without audible sounds. However, most existing silent speech solutions require either restricted background illuminations or hand involvement to hold device or perform gestures. In this study, we propose a novel earphone-based, hand-free silent speech interaction approach, named EarCommand. Our technique discovers the relationship between the deformation of the ear canal and the movements of the articulator and takes advantage of this link to recognize different silent speech commands. Our system can achieve a WER (word error rate) of 10.02% for word-level recognition and 12.33% for sentence-level recognition, when tested in human subjects with 32 word-level commands and 25 sentence-level commands, which indicates the effectiveness of inferring silent speech commands. Moreover, EarCommand shows high reliability and robustness in a variety of configuration settings and environmental conditions. It is anticipated that EarCommand can serve as an efficient, intelligent speech interface for hand-free operation, which could significantly improve the quality and convenience of interactions.

XAIR: A Framework of Explainable AI in Augmented Reality

“…User State. The sensors that could be integrated within future HMDs would empower an AR system to have a rich, instant understanding of user's state, such as activities (IMU [86,219], camera [80,128,194,201], microphone [103,218,229,230]), cognitive load (eye tracking [71,104,238], EEG [20,224]), attention (eye tracking [56,99,204,231], IMU [123], EEG [213]), emotion (facial tracking [233,236], EEG [202,216]) and potential intent (the fusion of multiple sensors and low-level intelligence [14,111,211]). Depending on a user's state, the design of explanations could be different.…”

Decoding silent speech commands from articulatory movements through soft magnetic skin and machine learning

“…Contactless approaches are mainly explored through camera-based visual signals, [9][10][11][12][13][14][15] ultrasound signals. [16][17][18][19][20][21][22][23] Camerabased visual solutions require external video tracking devices, and users must remain within the camera's line of sight. Despite efforts to develop compact shoulder-mounted devices 9 to enhance portability, visual solutions still face challenges in terms of lighting conditions and angles between users and cameras, thereby limiting their practicality.…”

“…As a more portable and user-friendly ultrasound-based solution, the speaker of the cell phone was used to emit ultrasound signals, and the microphone was employed to capture reflected signals from the lips. 16,[18][19][20][21][22] This method is not hands-free and is susceptible to multipath interferences caused by bodily movements and surrounding objects.…”

“…EEG methods [26][27][28] can interpret speech information but are susceptible to interferences, especially when subjects experience cognitive distractions or mental deviations. Silent speech interfaces based on inertial measurement units (IMU) attached to the temporomandibular joint 3,29 or the chin and neck skin, 30 proximity sensors attached to the ear canal, 18,31 and rigid magnets to the tongue and facial skin [32][33][34] can achieve high portability and accuracy. These systems implement rigid components and highly visible interfaces on the skin.…”