Recognizing text using motion data from a smartwatch

Arduser, Luca; Bissig, Pascal; Brandes, Philipp; Wattenhofer, Roger

doi:10.1109/percomw.2016.7457172

Cited by 36 publications

(23 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multiple previous research papers have demonstrated the feasibility of inferring different forms of handwritten information from motion data collected by means of wrist-wearables. We are particularly interested in the following four forms of handwriting scenarios that were evaluated earlier, primarily because they are the most commonly observed in real-life situations: (i) pen(cil) and paper writing [30], (ii) whiteboard writing [6], (iii) finger writing [31], and (iv) airwriting [3][4][5]. In this section, we describe the main strengths and shortcomings of these earlier research efforts, and outline our primary motivation for revisiting the problem of handwriting inference using wrist-wearables.…”

Section: Previous Work and Motivationmentioning

confidence: 99%

“…In the same vein, multiple research efforts have also demonstrated the feasibility of inferring handwritten text using motion sensors (such as accelerometers and gyroscopes) present onboard these wrist-wearables. Some of the initial efforts in this direction showed the feasibility of inferring larger handwriting gestures, such as, writing on a whiteboard [6] or using hand/finger movements to write in the air [4,5,31]. More recent efforts have focused on inferring smaller and more natural handwriting gestures, such as, writing on a paper with pen/pencil [30].…”

mentioning

confidence: 99%

“…Broadly, our goal in this work is to evaluate if existing handwriting inference frameworks are a genuine and realistic threat to a wrist-wearable device user's privacy and security. In order to accomplish this goal in a structured fashion, we first closely replicate the four most notable inference frameworks in this direction, specifically, the ones defined by Xu et al [31], Arduser et al [6], Amma et al [5], and Xia et al [30]. Next, by means of contemporary, consumer-grade wrist-wearables, we collect natural handwriting related motion data from a large number of human subject participants in an unconstrained and non-restrictive setting for a variety of different writing scenarios.…”

mentioning

confidence: 99%

See 2 more Smart Citations

deWristified

Wijewickrama

Maiti

Jadliwala

2019

Proceedings of the 12th Conference on Security and Privacy in Wireless and Mobile Networks

View full text Add to dashboard Cite

Several recent research efforts have shown that privacy of handwritten information is vulnerable to inference threats that employ zeropermission motion sensors commonly found on wrist-wearables (e.g., smart watches and fitness bands) as information side-channels. While the adversary model in these earlier efforts have been reasonable and the proposed inference (or threat) frameworks themselves are practical and have technical merit, the related empirical evaluations suffer from several significant shortcomings, such as, use of specialized sensor hardware and highly constrained or restrictive experimental procedures, to name a few. As a result, it is hard to estimate the practical feasibility of these threats from existing research results in the literature, and thus, the extent to which end-users must be concerned about the possibility of such attacks in real-life. To answer the above question, this paper replicates some of the well-known wrist motion-based handwriting inference frameworks in the literature in order to (re)evaluate their success or accuracy in natural, unrestricted handwriting scenarios and settings by employing commercially available wrist-wearables. The results of these extensive replication and (re)evaluation studies highlight several characteristics in motion data corresponding to natural handwriting scenarios, which were either not observed or ignored by earlier efforts, and contribute to poor inference accuracy of the corresponding frameworks. In summary, accurate and practical handwriting inference using motion data (side-channeled) from consumer-grade wrist-wearables is difficult primarily due to unique and/or inconsistent handwriting behavior observed in natural writing.

show abstract

Section: Previous Work and Motivationmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

deWristified

Wijewickrama

Maiti

Jadliwala

2019

Proceedings of the 12th Conference on Security and Privacy in Wireless and Mobile Networks

View full text Add to dashboard Cite

show abstract

“…Shen et al also show that since wrist position is determined by both the shoulder and the elbow motions [42], the full arm posture can be sensed even by a wristband. Based on this work, several interesting sensing applications such as driving [13, 37], whiteboard writing [11], gaming control [4, 49], and writing or drawing in the air [8, 38, 48] have been developed. Researchers found inertial sensors are capable of capturing subtle hand movements, and sensitive information can be leaked when a user types [21, 32, 36].…”

Section: Related Workmentioning

confidence: 99%

SeleCon

Alanwar

Alzantot

et al. 2017

Proceedings of the Second International Conference on Internet-of-Things Design and Implementation

View full text Add to dashboard Cite

Although different interaction modalities have been proposed in the field of human-computer interface (HCI), only a few of these techniques could reach the end users because of scalability and usability issues. Given the popularity and the growing number of IoT devices, selecting one out of many devices becomes a hurdle in a typical smarthome environment. Therefore, an easy-to-learn, scalable, and non-intrusive interaction modality has to be explored. In this paper, we propose a pointing approach to interact with devices, as pointing is arguably a natural way for device selection. We introduce SeleCon for device selection and control which uses an ultra-wideband (UWB) equipped smartwatch. To interact with a device in our system, people can point to the device to select it then draw a hand gesture in the air to specify a control action. To this end, SeleCon employs inertial sensors for pointing gesture detection and a UWB transceiver for identifying the selected device from ranging measurements. Furthermore, SeleCon supports an alphabet of gestures that can be used for controlling the selected devices. We performed our experiment in a 9m-by-10m lab space with eight deployed devices. The results demonstrate that SeleCon can achieve 84.5% accuracy for device selection and 97% accuracy for hand gesture recognition. We also show that SeleCon is power efficient to sustain daily use by turning off the UWB transceiver, when a user’s wrist is stationary.

show abstract

“…MoLe [42] is a system that analyses motion data from typing movements using smartwatches. A similar system has been proposed by Arduser et al [1], where text written on a whiteboard is inferred from simple acceleration data. Fine finger gestures like pinching, tapping, or rubbing can also be recognized from the built-in motion sensors, as shown by [43].…”

Section: Gestural Interaction With Wearablesmentioning

confidence: 99%

Collocated Multi-user Gestural Interactions with Unmodified Wearable Devices

et al. 2017

View full text Add to dashboard Cite

Many real-life scenarios can benefit from both physical proximity and natural gesture interaction. In this paper, we explore shared collocated interactions on unmodified wearable devices. We introduce an interaction technique which enables a small group of people to interact using natural gestures. The proximity of users and devices is detected through acoustic ranging using inaudible signals, while in-air hand gestures are recognized from threeaxis accelerometers. The underlying wireless communication between the devices is handled over Bluetooth for scalability and extensibility. We present (1) an overview of the interaction technique and (2) an extensive evaluation using unmodified, off-the-shelf, mobile, and wearable devices which show the feasibility of the method. Finally, we demonstrate the resulting design space with three examples of multi-user application scenarios.

show abstract

Recognizing text using motion data from a smartwatch

Cited by 36 publications

References 15 publications

deWristified

deWristified

SeleCon

Collocated Multi-user Gestural Interactions with Unmodified Wearable Devices

Contact Info

Product

Resources

About