RNN-Based Room Scale Hand Motion Tracking

Mao, Wenguang; Wang, Mei; Sun, Wei; Qiu, Lili; Pradhan, Swadhin; Chen, Yichao

doi:10.1145/3300061.3345439

Cited by 59 publications

(13 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Acoustic Gesture Recognition systems use acoustic signals emitted by speakers and recorded by a microphone to obtain information from the movements of the human body. Examples include SoundWave [12], Dolphin [34], AudioGest [36], SonicOperator [21], Strata [47], FingerIO [27], CAT [23] and the work from Watanabe et al [44] and Mao el al [24]. While these past systems have made significant advances in acoustic sensing, they cannot simultaneously satisfy two key design features of Saving Face: operating entirely with existing smartphones and headsets, and sensing small hand movements in the presence of significant mobility.…”

Section: Acoustic Gesture Recognitionmentioning

confidence: 99%

A Scalable Solution for Signaling Face Touches to Reduce the Spread of Surface-based Pathogens

Rojas

Poulsen

Tuyl

et al. 2021

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

Hand-to-Face transmission has been estimated to be a minority, yet non-negligible, vector of COVID-19 transmission and a major vector for multiple other pathogens. At the same time, as it cannot be effectively addressed with mainstream protection measures, such as wearing masks or tracing contacts, it remains largely untackled. To help address this issue, we have developed Saving Face - an app that alerts users when they are about to touch their faces, by analyzing the distortion patterns in the ultrasound signal emitted by their earphones. The system only relies on pre-existing hardware (a smartphone with generic earphones), which allows it to be rapidly scalable to billions of smartphone users worldwide. This paper describes the design, implementation and evaluation of the system, as well as the results of a user study testing the solution's accuracy, robustness, and user experience during various day-to-day activities (93.7% Sensitivity and 91.5% Precision, N=10). While this paper focuses on the system's application to detecting hand-to-face gestures, the technique can also be applicable to other types of gestures and gesture-based applications.

show abstract

Section: Acoustic Gesture Recognitionmentioning

confidence: 99%

A Scalable Solution for Signaling Face Touches to Reduce the Spread of Surface-based Pathogens

Rojas

Poulsen

Tuyl

et al. 2021

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

show abstract

“…Strata estimates the CIR and measures the phase changes of each channel tap to detect finger motions. In [30], the authors propose a hand motion tracking system that leverages a microphone array and an RNN-enhanced algorithm to track hand motions on a home scale. (iii) Vital sign detection.…”

Section: Acoustic Sensing Systemsmentioning

confidence: 99%

Acoustic-based Upper Facial Action Recognition for Smart Eyewear

Xie

Zhang

2021

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

Smart eyewear (e.g., AR glasses) is considered to be the next big breakthrough for wearable devices. The interaction of state-of-the-art smart eyewear mostly relies on the touchpad which is obtrusive and not user-friendly. In this work, we propose a novel acoustic-based upper facial action (UFA) recognition system that serves as a hands-free interaction mechanism for smart eyewear. The proposed system is a glass-mounted acoustic sensing system with several pairs of commercial speakers and microphones to sense UFAs. There are two main challenges in designing the system. The first challenge is that the system is in a severe multipath environment and the received signal could have large attenuation due to the frequency-selective fading which will degrade the system's performance. To overcome this challenge, we design an Orthogonal Frequency Division Multiplexing (OFDM)-based channel state information (CSI) estimation scheme that is able to measure the phase changes caused by a facial action while mitigating the frequency-selective fading. The second challenge is that because the skin deformation caused by a facial action is tiny, the received signal has very small variations. Thus, it is hard to derive useful information directly from the received signal. To resolve this challenge, we apply a time-frequency analysis to derive the time-frequency domain signal from the CSI. We show that the derived time-frequency domain signal contains distinct patterns for different UFAs. Furthermore, we design a Convolutional Neural Network (CNN) to extract high-level features from the time-frequency patterns and classify the features into six UFAs, namely, cheek-raiser, brow-raiser, brow-lower, wink, blink and neutral. We evaluate the performance of our system through experiments on data collected from 26 subjects. The experimental result shows that our system can recognize the six UFAs with an average F1-score of 0.92.

show abstract

“…CovertBand [33] transmits beacons from an amplified speaker driven by a smartphone, and locates multiple users (even their actions) by analyzing the acoustic reflections received by the phone. Mao et al [30] adopt a similar approach to CovertBand but use a different modulation technique and a microphone array as the receiver. It utilizes 2D-MUSIC [55] and RNN [16] to track multiple users and to identify their gestures in a room scale.…”

Section: Related Workmentioning

confidence: 99%

“…It utilizes 2D-MUSIC [55] and RNN [16] to track multiple users and to identify their gestures in a room scale. However, by mimicking the function of an active sonar (i.e., transmitting some form of sounds to probe the users), both [33] and [30] are not passive localization solutions, hence not applicable to the indoor scenarios we have in mind.…”

Section: Related Workmentioning

confidence: 99%

We Hear Your PACE

Cai

Wang

et al. 2021

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

Indoor localization is crucial to enable context-aware applications, but existing solutions mostly require a user to carry a device, so as to actively sense location-discriminating signals. However, many applications do not prefer user involvement due to, e.g., the cumbersome of carrying a device. Therefore, solutions that track user locations passively can be desirable, yet lack of active user involvement has made passive indoor localization very challenging even for a single person. To this end, we propose Passive Acoustic loCalization of multiple walking pErsons (PACE) as a solution for small-scale indoor scenarios: it passively locates users by pinpointing the positions of their footsteps. In particular, PACE leverages both structure-borne and air-borne footstep impact sounds (FIS); it uses structure-borne FIS for range estimations exploiting their acoustic dispersion nature, and it employs air-borne FIS for Angle-of-Arrival (AoA) estimations and person identifications. To combat the low-SNR nature of FIS, PACE innovatively employs domain adversarial adaptation and spectral weighting to ranging/identification and AoA estimations, respectively. We implement a PACE prototype and extensively evaluate its performance in representative environments. The results demonstrate a promising sub-meter localization accuracy with a median error of 30 cm.

show abstract

RNN-Based Room Scale Hand Motion Tracking

Cited by 59 publications

References 30 publications

A Scalable Solution for Signaling Face Touches to Reduce the Spread of Surface-based Pathogens

A Scalable Solution for Signaling Face Touches to Reduce the Spread of Surface-based Pathogens

Acoustic-based Upper Facial Action Recognition for Smart Eyewear

We Hear Your PACE

Contact Info

Product

Resources

About