Situation awareness via sensor-equipped eyeglasses

Windau, Jens; Itti, Laurent

doi:10.1109/iros.2013.6697178

Cited by 23 publications

(24 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Potential technological applications include: wearable visual technologies (smart glasses like Google Glass), smart displays, adaptive web search, marketing, activity recognition (Albert, Toledo, Shapiro, & Kording, 2012;Fathi, Farhadi, & Rehg, 2011;Pirsiavash & Ramanan, 2012), human-computer interaction, and biometrics. Portable electronic devices such as smartphones, tablets, and smart glasses with cameras are becoming increasingly popular (see Windau & Itti, 2013, for an example study). Enabling eye tracking on these devices could be used to predict the user's intent one step ahead and provide him necessary information in a more efficient and adaptive manner.…”

Section: Discussionmentioning

confidence: 99%

Defending Yarbus: Eye movements reveal observers' task

2014

View full text Add to dashboard Cite

In a very influential yet anecdotal illustration, Yarbus suggested that human eye-movement patterns are modulated top down by different task demands. While the hypothesis that it is possible to decode the observer's task from eye movements has received some support (e.g., Henderson, Shinkareva, Wang, Luke, & Olejarczyk, 2013; Iqbal & Bailey, 2004), Greene, Liu, and Wolfe (2012) argued against it by reporting a failure. In this study, we perform a more systematic investigation of this problem, probing a larger number of experimental factors than previously. Our main goal is to determine the informativeness of eye movements for task and mental state decoding. We perform two experiments. In the first experiment, we reanalyze the data from a previous study by Greene et al. (2012) and contrary to their conclusion, we report that it is possible to decode the observer's task from aggregate eye-movement features slightly but significantly above chance, using a Boosting classifier (34.12% correct vs. 25% chance level; binomial test, p = 1.0722e - 04). In the second experiment, we repeat and extend Yarbus's original experiment by collecting eye movements of 21 observers viewing 15 natural scenes (including Yarbus's scene) under Yarbus's seven questions. We show that task decoding is possible, also moderately but significantly above chance (24.21% vs. 14.29% chance-level; binomial test, p = 2.4535e - 06). We thus conclude that Yarbus's idea is supported by our data and continues to be an inspiration for future computational and experimental eye-movement research. From a broader perspective, we discuss techniques, features, limitations, societal and technological impacts, and future directions in task decoding from eye movements.

show abstract

Section: Discussionmentioning

confidence: 99%

Defending Yarbus: Eye movements reveal observers' task

2014

View full text Add to dashboard Cite

show abstract

“…In [79], the authors used the accelerometer, gyroscope, and camera for the identification of several activities, such as lying,…”

Section: Related Workmentioning

confidence: 99%

“…elevator, using the Naïve Bayes, SVM and HMM methods. For these methods was extracted some features, such as mean and variance for each accelerometer axis, movement intensity, and energy, reporting an accuracy of 81.5% [79].…”

Section: Introductionmentioning

confidence: 99%

Pattern recognition techniques for the identification of Activities of Daily Living using mobile device accelerometer

Pires¹,

García²,

Pombo³

et al. 2019

Preprint

View full text Add to dashboard Cite

This paper focuses on the recognition of Activities of Daily Living (ADL) applying pattern recognition techniques to the data acquired by the accelerometer available in the mobile devices. The recognition of ADL is composed by several stages, including data acquisition, data processing, and artificial intelligence methods. The artificial intelligence methods used are related to pattern recognition, and this study focuses on the use of Artificial Neural Networks (ANN). The data processing includes data cleaning, and the feature extraction techniques to define the inputs for the ANN. Due to the low processing power and memory of the mobile devices, they should be mainly used to acquire the data, applying an ANN previously trained for the identification of the ADL. The main purpose of this paper is to present a new method implemented with ANN for the identification of a defined set of ADL with a reliable accuracy. This paper also presents a comparison of different types of ANN in order to choose the type for the implementation of the final method. Results of this research probes that the best accuracies are achieved with Deep Learning techniques with an accuracy higher than 80%. Following the previous research studies [4][5][6], the recognition of the ADL is composed by several steps, such as the data acquisition, the data processing, composed by data cleaning, data imputation, and feature extraction, the data fusion, and the artificial intelligence methods for the concrete identification of the ADL. However, this study only uses the accelerometer sensor, removing some steps of the proposed architecture. Based on the assumption that the sensor was always acquiring the data, the final steps used are the data acquisition, the data cleaning, and the application of the artificial intelligence methods.During the last years, the recognition of ADL has been studies by several authors [7][8][9][10][11][12], verifying that the Artificial Neural Networks (ANN) are widely used. This paper proposes the creation of a method for the recognition of ADL using accelerometer, comparing three types of ANN, such as Multilayer Perception (MLP) with Backpropagation, Feedforward neural network with Backpropagation, and Deep Learning, in order to verify the method that achieves the best accuracy in the recognition of running, walking, going upstairs, going downstairs, and standing. The datasets are composed with raw accelerometer data acquired by individual with ages ranged between 16 and 60 years old and different lifestyles, with a mobile device in the pocket. For the implementation of these types of ANN are used several datasets with different sets of features, identifying the best features to increase the accuracy of the recognition, and three Java libraries are used for the implementation of the different methods, such as Neuroph [13], Encog [14], and DeepLearning4j [15], achieving the best accuracy with Deep Learning methods.The remaining sections of this paper are organized as follows: Section 2 presents a brief literature review re...

show abstract

“…Then, the data is transformed into the normalized coordinate system. This transformation is necessary to measure two critical types of sensor data during walking independently of the actual head orientation [10].…”

Section: B Steps In Detailmentioning

confidence: 99%

“…This could greatly enhance many device applications, e.g., music streaming with Google Play Music: for different activities (e.g., working out, making food, cleaning the house, or driving a car), the service adapts its music offerings; but currently users have to manually indicate which activity they are engaged in. We showed in previous work that head-mounted sensor systems can successfully distinguish between many daily activities through online analysis and classification of the streaming IMU data, but this was when the head was held pointing straight forward [10]. Head-mounted devices can also be used to track motions with-Jens Windau and Laurent Itti are with the iLab and Computer Science Department at the University of Southern California, Los Angeles, CA, 90089-2520, USA.…”

Section: Introductionmentioning

confidence: 99%

Walking compass with head-mounted IMU sensor

Windau

Itti

2016

2016 IEEE International Conference on Robotics and Automation (ICRA)

Self Cite

View full text Add to dashboard Cite

Emerging wearable technologies offer new sensor placement options on the human body. Particularly, head-mounted glasswear opens up new data capturing possibilities directly from the human head. This allows exploring new cyber-robotics algorithms (robotics sensors and human motor plant). Glass-wear systems, however, require additional compensation for head motions that will affect the captured sensor data. Particularly, pedestrian dead-reckoning (PDR), activity recognition, and other applications are limited or restricted when head-mounted sensors are used, because of possible confusion between head and body movements. Thus, previous PDR approaches typically required to keep the head pointing direction aligned with the walking direction to avoid positional errors. This paper presents a head-mounted orientation system (HOS) that identifies and filters out interfering head motions in 3 steps.Step 1 transforms inertial sensor data into a stable normalized coordinate system (roll/pitch motion compensated).Step 2 compares walking patterns before and after a rotating motion.Step 3 eliminates interfering head motions from sensor data by dynamically adjusting the noise parameters of the extended Kalman filter. HOS has been implemented on a Google Glass platform and achieved high accuracy in tracking a person's path even in the presence of head movements (within 2.5% of traveled distance) when tested in multiple real-world scenarios. By eliminating head motions, HOS not only enables accurate PDR, but also facilitates the task for downstream activity recognition algorithms.

show abstract

Situation awareness via sensor-equipped eyeglasses

Cited by 23 publications

References 11 publications

Defending Yarbus: Eye movements reveal observers' task

Defending Yarbus: Eye movements reveal observers' task

Pattern recognition techniques for the identification of Activities of Daily Living using mobile device accelerometer

Walking compass with head-mounted IMU sensor

Contact Info

Product

Resources

About