Gaze+Lip: Rapid, Precise and Expressive Interactions Combining Gaze Input and Silent Speech Commands for Hands-free Smart TV Control

LipLearner: Customizable Silent Speech Interactions on Mobile Devices

“…Lipreading is a technology that utilizes a camera to visually capture movement around the mouth and interpret speech from the image sequence. HCI researchers have proposed to use devices such as smartphones [45,58] and wearable cameras [6,34,69] to provide mobile silent speech interaction, as well as multimodal approaches such as using silent speech to facilitate eye-gaze-based selection [57].…”

“…Nonetheless, building a model that can generalize to unseen users remains a huge challenge, as conventional methods either have a small vocabulary [58] or limited accuracy [45,52]. One workaround is to use a context-dependent vocabulary to improve accuracy, but it also limits the number of available commands at a time [57,58]. Furthermore, a common issue in both user-dependent models and off-the-shelf models is that the commands are pre-defined by the researchers.…”

“…Additionally, there is a lack of a practical activating method to initiate silent speech input. Previous methods such as offline segmentation [6,34,69] or trigger buttons [45,52] are not feasible for hands-free real-time interactions, and MOD-based methods can be vulnerable to misactivations [57,58]. We propose a novel few-shot transfer learning paradigm to enable customizable silent speech commands.…”

“…First, the data collection process should minimize the effort for new users to get started with. However, previous approaches to SSIs, not limited to lipreading-based approaches, adopt a train-from-scratch model that requires collecting hundreds of samples from real users [34,57,58,69], leading to excessive mental and physical user burden. Second, such data collected intensively in controlled laboratory environments causes a biased model, which can be sensitive to even minor changes in factors such as lighting, face orientations, and postures, yet there is little discussion on the model's ability to generalize to unseen environments.…”

“…To empower LipLearner with reliable hands-free activation, we propose a visual keyword spotting method that detects the userdefined keyword from lip movements. Most previous lipreading interfaces use the mouth opening degree (MOD) [57,58] as the only cue to trigger silent speech input, which is prone to misactivation. For example, the system can easily respond to the user's unintentional behavior, such as smiling or talking to others.…”