Pedestrian Street-Cross Action Recognition in Monocular Far Infrared Sequences

Abstract

The early recognition and understanding of the actions performed by pedestrians in traffic scenes leads to an anticipation of pedestrian intentions in advance and helps in the process of collision warning and avoidance in the context of autonomous vehicles. An environment with low visibility conditions such as night-time, fog, heavy rain or smoke increases the number of difficult situations in traffic. A complete and original model for assessing if a pedestrian is engaged in a street cross action using only infrared monocular scene perception is proposed in this paper. The assessment of a street cross action is done by the time series analysis of features like: pedestrian motion, position of pedestrians with respect to the drivable area and their distance with respect to the ego-vehicle. The extraction of these features emerges from the combination of a deep learning based pedestrian detector with an original tracking algorithm, a semantic segmentation of the road surface and a time series long-short term memory network based action recognition. In order to validate the proposed method we introduce a new dataset named CROSSIR. It is formed of pedestrian annotations, action annotations and semantic labels for the road. The CROSSIR dataset is suitable for several common computer vision algorithms: (1) pedestrian detection and tracking algorithms because each pedestrian has a unique identifier over the frames in which it appears; (2) pedestrian action recognition; (3) semantic segmentation of the road pixels in the infrared image.

Robust Data Association Using Fusion of Data-Driven and Engineered Features for Real-Time Pedestrian Tracking in Thermal Images

“…Another tracker that was applied on thermal images [ 20 ] used edge features and a 2640-dimensional histogram feature computed from the intensity channel. In [ 21 ] the authors combined a motion and an appearance score for improving the data association process from the tracking framework. The appearance cost, between the track and measurement, was composed of a weighted combination of multiple individual scores obtained via feature engineering.…”

“…We built upon the state of the art by creating a data association solution that efficiently combines the data-driven and feature-engineered costs in order to create a robust data association function useful within the tracking framework. We used the motion and appearance scores presented in [ 21 ] and we added to the appearance score two additional terms. The first term was a feature engineered score that was derived by combining the uniform LBP with HOG features, and the second term was a data-driven term obtained by using a family of Siamese neural networks.…”

“…In this paper, we build upon the solution presented in [ 21 ], which was considered the base solution for our approach. In this section we are providing some details regarding the base solution and in the following subsections we describe the proposed contributions.…”

“…The weights introduced in both Equations (1) and (2) allow us to set the influence of certain parameters. Their value was determined experimentally and can be found in [ 21 ]. The final similarity cost was composed of the sum of the motion and appearance costs.…”

“…The main disadvantages of thermal images are that they do not contain as much information as the color or even the monochrome images, and they usually have a lower resolution, which makes the design of a data association function based on appearance even more difficult. There are two main directions in the literature for addressing this issue of data association: the feature engineering approaches [ 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ] and the data-driven methods (using convolutional neural networks) [ 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 ]. The advantage of designing the data association function using data-driven methods is that after the convolutional neural network architecture is designed, through a learning process the best features are identified.…”