TieLent

LipLearner: Customizable Silent Speech Interactions on Mobile Devices

“…Lipreading is a technology that utilizes a camera to visually capture movement around the mouth and interpret speech from the image sequence. HCI researchers have proposed to use devices such as smartphones [45,58] and wearable cameras [6,34,69] to provide mobile silent speech interaction, as well as multimodal approaches such as using silent speech to facilitate eye-gaze-based selection [57].…”

“…Additionally, there is a lack of a practical activating method to initiate silent speech input. Previous methods such as offline segmentation [6,34,69] or trigger buttons [45,52] are not feasible for hands-free real-time interactions, and MOD-based methods can be vulnerable to misactivations [57,58]. We propose a novel few-shot transfer learning paradigm to enable customizable silent speech commands.…”

“…SSI provides seamless and confidential interactions in various situations, especially in those where voice interaction is inappropriate or unavailable. Recent research on SSI has proposed to use various sensing methods such as Electromyography (EMG) [31,43], ultrasound imaging [35], capacitive sensing [40] and video camera (lipreading) [34,45,58] to track the movement of speech articulators and decode silent speech.…”

“…First, the data collection process should minimize the effort for new users to get started with. However, previous approaches to SSIs, not limited to lipreading-based approaches, adopt a train-from-scratch model that requires collecting hundreds of samples from real users [34,57,58,69], leading to excessive mental and physical user burden. Second, such data collected intensively in controlled laboratory environments causes a biased model, which can be sensitive to even minor changes in factors such as lighting, face orientations, and postures, yet there is little discussion on the model's ability to generalize to unseen environments.…”

Decoding silent speech commands from articulatory movements through soft magnetic skin and machine learning

“…7,8 Methods for silent speech recognition can be broadly classified into two categories: contactless approaches and contact-based approaches. Contactless approaches are mainly explored through camera-based visual signals, [9][10][11][12][13][14][15] ultrasound signals. [16][17][18][19][20][21][22][23] Camerabased visual solutions require external video tracking devices, and users must remain within the camera's line of sight.…”