Activity Recognition Using Different Sensor Modalities and Deep Learning

Abstract

In recent years, human activity monitoring and recognition have gained importance in providing valuable information to improve the quality of life. A lack of activity can cause health problems including falling, depression, and decreased mobility. Continuous activity monitoring can be useful to prevent progressive health problems. With this purpose, this study presents a wireless smart insole with four force-sensitive resistors (FSRs) that monitor foot contact states during activities for both indoor and outdoor use. The designed insole is a compact solution and provides walking comfort with a slim and flexible structure. Moreover, the inertial measurement unit (IMU) sensors designed in our previous study were used to collect 3-axis accelerometer and 3-axis gyroscope outputs. Smart insoles were located in the shoe sole for both right and left feet, and two IMU sensors were attached to the thigh area of each leg. The sensor outputs were collected and recorded from forty healthy volunteers for eight different gait-based activities including walking uphill and descending stairs. The obtained datasets were separated into three categories; foot contact states, the combination of acceleration and gyroscope outputs, and a set of all sensor outputs. The dataset for each category was separately fed into deep learning algorithms, namely, convolutional long–short-term memory neural networks. The performance of each neural network for each category type was examined. The results show that the neural network using only foot contact states presents 90.1% accuracy and provides better performance than the combination of acceleration and gyroscope datasets for activity recognition. Moreover, the neural network presents the best results with 93.4% accuracy using a combination of all the data compared with the other two categories.