EchoSpeech: Continuous Silent Speech Recognition on Minimally-obtrusive Eyewear Powered by Acoustic Sensing

Zhang, Ruidong; Li, Ke; Hao, Yihong; Wang, Yufan; Lai, Zhengnan; Guimbretière, François; Zhang, Cheng

doi:10.1145/3544548.3580801

Cited by 16 publications

(1 citation statement)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To explore these research questions, we developed EyeEcho using active acoustic sensing to capture the skin deformations on glasses, which we will detail later. We chose acoustic sensing because its sensors are small, lightweight, low-power and have been successfully applied to various tasks on tracking human activities, including health-related activities detection [45,65], novel interaction methods [3,66,77,78], silent speech recognition [61,80,81,83], authentication [11,18,23,40,67], discrete facial expression recognition [74], gaze tracking [33], finger tracking [46,60], hand gesture recognition [31,79], body pose estimation [41], and motion tracking [35,84].…”

Section: Principle Of Operationmentioning

confidence: 99%

EyeEcho: Continuous and Low-power Facial Expression Tracking on Glasses

Li,

Zhang,

Chen

et al. 2024

Proceedings of the CHI Conference on Human Factors in Computing Systems

Self Cite

View full text Add to dashboard Cite

In this paper, we introduce EyeEcho, a minimally-obtrusive acoustic sensing system designed to enable glasses to continuously monitor facial expressions. It utilizes two pairs of speakers and microphones mounted on glasses, to emit encoded inaudible acoustic signals directed towards the face, capturing subtle skin deformations associated with facial expressions. The reflected signals are processed through a customized machine-learning pipeline to estimate full facial movements. EyeEcho samples at 83.3 Hz with a relatively low power consumption of 167𝑚𝑊 . Our user study involving 12 participants demonstrates that, with just four minutes of training data, EyeEcho achieves highly accurate tracking performance across different real-world scenarios, including sitting, walking, and after remounting the devices. Additionally, a semi-in-the-wild study involving 10 participants further validates EyeEcho's performance in naturalistic scenarios while participants engage in various daily activities. Finally, we showcase EyeEcho's potential to be deployed on a commercial-off-the-shelf (COTS) smartphone, offering real-time facial expression tracking. CCS CONCEPTS• Human-centered computing → Ubiquitous and mobile devices; • Hardware → Power and energy.

show abstract