VIENA $$^2$$ : A Driving Anticipation Dataset

Abstract

Action anticipation is critical in scenarios where one needs to react before the action is finalized. This is, for instance, the case in automated driving, where a car needs to, e.g., avoid hitting pedestrians and respect traffic lights. While solutions have been proposed to tackle subsets of the driving anticipation tasks, by making use of diverse, taskspecific sensors, there is no single dataset or framework that addresses them all in a consistent manner. In this paper, we therefore introduce a new, large-scale dataset, called VIENA 2 , covering 5 generic driving scenarios, with a total of 25 distinct action classes. It contains more than 15K full HD, 5s long videos acquired in various driving conditions, weathers, daytimes and environments, complemented with a common and realistic set of sensor measurements. This amounts to more than 2.25M frames, each annotated with an action label, corresponding to 600 samples per action class. We discuss our data acquisition strategy and the statistics of our dataset, and benchmark state-of-the-art action anticipation techniques, including a new multi-modal LSTM architecture with an effective loss function for action anticipation in driving scenarios.

Driving Scene Understanding: How much temporal context and spatial resolution is necessary?

“…We present our results for VIENA 2 dataset in Table 2, where we compare our class-wise accuracy scores (obtained with E r×c,16 ) with that of Aliakbarian et al [9] (obtained on the visual and sensor data, at the end of the 5 th second). In the experiments E r×c, 16 , we found that the class-wise accuracy scores for each Scenario plateaus after the 50 th epoch, with slight fluctuations later (total number of epochs run -64).…”

“…Torstensson et al [8] proposed a Convolutional and LSTM based network to predict the actions of the in-vehicle driver. In a work related to Driving Action Anticipation, Aliakbarian et al [9] introduced a new dataset: VIENA 2 and proposed a multi-modal LSTM based network to forecast driver actions from visual and sensor data.…”

“…For the HDD and VIENA 2 dataset, 1 epoch's training and inference run-time figures (averaged across runs) of the best performing experiment E 108×192,16 are mentioned in Table 1. Since the authors [9][10][11]13] (with whom we compare our results later) do not provide run-time estimates of their experiments, we are unable to present a comparison. Our code is publicly available 1 .…”

Video Action Understanding

“…These create predictions directly from observed features. Examples include F-RNN-EL [226] which uses an exponential loss to bias a multi-modal CNN+LSTM fusion strategy towards the most recent predictions, MS-LSTM [227] which uses two LSTM stages for action-aware and context-aware learning, MM-LSTM [228] which extends MS-LSTM to arbitrarily many modalities, FN [229] which uses a three-stage LSTM approach, and TP-LSTM [230] which uses a temporal pyramid learning structure.…”

Graph-SIM: A Graph-based Spatiotemporal Interaction Modelling for Pedestrian Action Prediction

“…There are a number of datasets that cater to pedestrian crossing prediction [8], [9], [12], [13], [14]. They contain video clips annotated with 2D bounding boxes and behavioural tags for pedestrians.…”