Data Fusion on Motion and Magnetic Sensors embedded on Mobile Devices for the Identification of Activities of Daily Living

Abstract

Several types of sensors have been available in off-the-shelf mobile devices, including motion, magnetic, vision, acoustic, and location sensors. This paper focuses on the fusion of the data acquired from motion and magnetic sensors, i.e., accelerometer, gyroscope and magnetometer sensors, for the recognition of Activities of Daily Living (ADL) using pattern recognition techniques. The system developed in this study includes data acquisition, data processing, data fusion, and artificial intelligence methods. Artificial Neural Networks (ANN) are included in artificial intelligence methods, which are used in this study for the recognition of ADL. The purpose of this study is the creation of a new method using ANN for the identification of ADL, comparing three types of ANN, in order to achieve results with a reliable accuracy. The best accuracy was obtained with Deep Learning, which, after the application of the L2 regularization and normalization techniques on the sensors' data, reports an accuracy of 89.51%.[13] that uses only the accelerometer sensor, this study proposes the creation of two different methods for the recognition of ADL using different number of sensors in order to adapt the method according to the number of sensors available. Firstly, it proposes the fusion of the data acquired from the accelerometer, and the magnetometer sensors. Secondly, it proposes the fusion of the data acquired from the accelerometer, gyroscope, and magnetometer sensors. The ADL proposed for the recognition are running, walking, going upstairs, going downstairs, and standing, consisting this research on the analysis of the performance of three types of ANN, such as Multilayer Perception (MLP) with Backpropagation, Feedforward neural network with Backpropagation, and Deep Learning. A dataset used for this research is composed by the sensors' data acquired in several experiments with people aged between 16 and 60 years old, distinct lifestyles, and a mobile device in the front pocket of their pants, performing the proposed ADL. This research was conducted with the use of three Java libraries, such as Neuroph [14], Encog [15], and DeepLearning4j [16], and different datasets of features, in order to identify the best dataset of features and type of ANN for the recognition of ADL, verifying that the best accuracy for the recognition of ADL with the two different methods proposed was achieved with Deep Learning methods.This paragraph concludes the section 1, and this paper is organized as follows: Section 2 summarizes the literature review for the use of data fusion techniques with accelerometer, gyroscope, and magnetometer sensors; Section 3 presents the methods used on each stage of the architecture proposed. The results obtained are presented in the Section 4, presenting the discussion about these results in the Section 5. The conclusions of this study are presented in the Section 6. Related WorkData fusion techniques may be used with the data acquired from motion and magnetic sensors available in the off-the-shelf mobile devic...

User Environment Detection with Acoustic Sensors Embedded on Mobile Devices for the Recognition of Activities of Daily Living

“…In coherence with our previous studies [4,14], based on the data filtered and the most features extracted in the studies available in the literature, the features extracted for the methods using acoustic data for the recognition of environments were the 26 MFCC coefficients, the Standard Deviation of the raw signal, the Average of the raw signal, the Maximum value of the raw signal, the Minimum value of the raw signal, the Variance of the of the raw signal, and the Median of the raw signal. On the other hand, the features extracted from the accelerometer, gyroscope, and magnetometer sensors were the 5 greatest distances between the maximum peaks, the Average of the maximum peaks, the Standard Deviation of the maximum peaks, the Variance of the maximum peaks, the Median of the maximum peaks, the Standard Deviation of the raw signal, the Average of the raw signal, the Maximum value of the raw signal, the Minimum value of the raw signal, the Variance of the of the raw signal, the Median of the raw signal, and the environment recognized.…”

“…In coherence with the methods defined in the previous studies [4,14] for the development of the framework for the recognition of ADL and their environments [5][6][7], the methods developed in this study should be separated in several methods, such as data acquisition, data processing, data fusion, and artificial intelligence methods, where the fusion of the data and the application of artificial intelligence methods are performed at the same time.…”

“…This study proposes the use of the microphone for the recognition of the environment, which is fused with the data acquired from the accelerometer, gyroscope and magnetometer sensors for the recognition of the activities with movement. In continuation of the previous study, available at [4], the main goal of the fusion of the environment recognized with the other sensors' data is to increase the number of ADL recognized using data fusion and artificial intelligence techniques. This study proposes the recognition of ADL, including running, walking, walking on stairs, standing, and sleeping, and the recognition of environments, including bar, classroom, gym, kitchen, library, street, hall, watching TV and bedroom.…”

“…The advantages of recognition of the environments are not limited to the increasing of the number of ADL recognized, but it allows the framework to combine the environments with the ADL recognition returning different results, e.g., the user is walking on the street.The topic related to the recognition of the ADL has some studies available in the literature [8][9][10][11][12][13], but there are no studies that uses all sensors available on the off-the-shelf mobile devices, however the Artificial Neural Networks (ANN) is one of the most used methods in this topic. Based on our previous studies using motion and magnetic sensors for the development of the framework for the recognition of ADL and their environments [4,14], this study proposes the creation of several methods to adapt the framework to the number of sensors available in off-the-shelf mobile devices. Some methods using different combinations of sensors are presented in previous studies [4,14], such as the method using accelerometer data, the method using accelerometer and magnetometer data, and the method using accelerometer, magnetometer and gyroscope sensors.…”

“…Based on our previous studies using motion and magnetic sensors for the development of the framework for the recognition of ADL and their environments [4,14], this study proposes the creation of several methods to adapt the framework to the number of sensors available in off-the-shelf mobile devices. Some methods using different combinations of sensors are presented in previous studies [4,14], such as the method using accelerometer data, the method using accelerometer and magnetometer data, and the method using accelerometer, magnetometer and gyroscope sensors. Thus, this study proposes the creation of the method using acoustic data for the recognition of the environments, as well as, different methods fusing the environment recognized with other data sources, such as the method using accelerometer and environment, the method using accelerometer, magnetometer and environment, and the method using accelerometer, magnetometer, gyroscope and environment.…”