Context-driven Multi-stream LSTM (M-LSTM) for Recognizing Fine-Grained Activity of Drivers

Abstract

Automatic recognition of in-vehicle activities has significant impact on the next generation intelligent vehicles. In this paper, we present a novel Multi-stream Long Short-Term Memory (M-LSTM) network for recognizing driver activities. We bring together ideas from recent works on LSTMs, transfer learning for object detection and body pose by exploring the use of deep convolutional neural networks (CNN). Recent work has also shown that representations such as hand-object interactions are important cues in characterizing human activities. The proposed M-LSTM integrates these ideas under one framework, where two streams focus on appearance information with two different levels of abstractions. The other two streams analyze the contextual information involving configuration of body parts and body-object interactions. The proposed contextual descriptor is built to be semantically rich and meaningful, and even when coupled with appearance features it is turned out to be highly discriminating. We validate this on two challenging datasets consisting driver activities.

Temporal convolutional neural (TCN) network for an effective weather forecasting using time-series data from the local weather station

“…These models have the ability to long-term dependencies by incorporating memory units. These memory units allow the network to learn, forget previously hidden states, and update hidden states (Behera et al 2018). Figure 7b depicts the general arrangement of an LSTM memory cell.…”

“…where w x ; b x ; ; i t ; j t ; f t ; o t are weight matrices, biases, element-wise vector product, input gate contributing to Connection solar panels to the weatherboard memory, input moderation gate contributing to memory, forget gate, and output gate as a multiplier between memory gates, respectively. The c t and h t are the two types of hidden layers to allow the LSTM to make complex decisions over a short period of time (Behera et al 2018;Jozefowicz et al 2015). The i t and f t gates are switching each other to selectively consider the current inputs or forget its previous memory.…”

Deep CNN, Body Pose, and Body-Object Interaction Features for Drivers’ Activity Monitoring

“…Martin et al [17] advocate a method to combine multiple streams involving body pose and contextual information in videos to recognise driver's activities. Similarly, Behera et al [13] describe a multi-stream LSTM for recognising driver's activities by combining high-level body pose and body-object interaction with CNN features. These models [16], [15], [17], [13] are similar to video classification methods, which require complete observation and is unsuitable for live activity recognition.…”

“…Similarly, Behera et al [13] describe a multi-stream LSTM for recognising driver's activities by combining high-level body pose and body-object interaction with CNN features. These models [16], [15], [17], [13] are similar to video classification methods, which require complete observation and is unsuitable for live activity recognition. Simialrly Alotaibi and Alotaibi [19] describe an approach that combines the inception module with a residual block and a hierarchical recurrent neural network to enhance the recognition performance of the distracted behaviours of drivers.…”

“…In such scenarios, contextual information (e.g. cues of body-objects interactions) plays a vital role [12], [13]. To improve the recognition accuracy, recently researchers have developed deep models focusing on spatio-temporal structures [14], [12], [15], [16], [17].…”