Spatiotemporal Feature Enhancement Aids the Driving Intention Inference of Intelligent Vehicles

Abstract

In order that fully self-driving vehicles can be realized, it is believed that systems where the driver shares control and authority with the intelligent vehicle offer the most effective solution. An understanding of driving intention is the key to building a collaborative autonomous driving system. In this study, the proposed method incorporates the spatiotemporal features of driver behavior and forward-facing traffic scenes through a feature extraction module; the joint representation was input into an inference module for obtaining driver intentions. The feature extraction module was a two-stream structure that was designed based on a deep three-dimensional convolutional neural network. To accommodate the differences in video data inside and outside the cab, the two-stream network consists of a slow pathway that processes the driver behavior data with low frame rates, along with a fast pathway that processes traffic scene data with high frame rates. Then, a gated recurrent unit, based on a recurrent neural network, and a fully connected layer constitute an intent inference module to estimate the driver’s lane-change and turning intentions. A public dataset, Brain4Cars, was used to validate the proposed method. The results showed that compared with modeling using the data related to driver behaviors, the ability of intention inference is significantly improved after integrating traffic scene information. The overall accuracy of the intention inference of five intents was 84.92% at a time of 1 s prior to the maneuver, indicating that making full use of traffic scene information was an effective way to improve inference performance.

Driver intention prediction based on multi-dimensional cross-modality information interaction

“…Some researchers have utilized 3D Conv to model the temporal information of video frame sequences. Chen et al proposed a two-stream structure based on a deep three-dimensional CNN [16], while Rong et al used 3D-ResNet to extract video spatiotemporal features [10]. Recent studies have demonstrated that CNN-based algorithms achieve state-of-the-art (SOTA) performance on datasets such as Brain4Car [1].…”

“…To address this, many research teams have emphasized the importance of cross-modal information interaction and designed effective multi-modal fusion methods. However, these methods typically focus on multimodal fusion in a single dimension (i.e., either feature extractor or classifier) [1,10,[16][17][18], and often overlook the potential benefits of incorporating GPS information. This is a noteworthy limitation in the field that warrants further exploration.…”

“…The effective interaction of multimodal information during feature extraction is critical for successful cross-modal fusion. However, previous studies have often overlooked this aspect [1,10,[16][17][18]. In this paper, we propose a novel information interaction mechanism that avoids the performance limitations of simply concatenating features or increasing the computational load of the network through excessive use of two-dimensional convolutional layer stacks.…”

“…Deep learning has recently gained extensive development and application in various fields. Due to the principle of using a large number of neurons to simulate human perception, thinking, and other activities, researchers have employed deep learning to address driver intention prediction, with promising outcomes [1,10,[16][17][18]. Generally, deep learning-based driver intention prediction approaches offer advantages such as automatic feature learning, end-to-end learning, and more comprehensive and superior performance of the learned features.…”

“…Generally, deep learning-based driver intention prediction approaches offer advantages such as automatic feature learning, end-to-end learning, and more comprehensive and superior performance of the learned features. However, most studies utilize 3D Conv [10,16,19], optical flow [1], or stacking of LSTM [3,[16][17][18]20] to model the temporal information of video sequences, leading to issues of large network parameters and high algorithm deployment costs. Moreover, it is noteworthy that most studies neglect or do not effectively utilize GPS information.…”