Forecasting Time-to-Collision from Monocular Video: Feasibility, Dataset, and Challenges

Abstract

We explore the possibility of using a single monocular camera to forecast the time to collision between a suitcaseshaped robot being pushed by its user and other nearby pedestrians. We develop a purely image-based deep learning approach that directly estimates the time to collision without the need of relying on explicit geometric depth estimates or velocity information to predict future collisions. While previous work has focused on detecting immediate collision in the context of navigating Unmanned Aerial Vehicles, the detection was limited to a binary variable (i.e., collision or no collision). We propose a more fine-grained approach to collision forecasting by predicting the exact time to collision in terms of milliseconds, which is more helpful for collision avoidance in the context of dynamic path planning. To evaluate our method, we have collected a novel dataset of over 13,000 indoor video segments each showing a trajectory of at least one person ending in a close proximity (a near collision) with the camera mounted on a mobile suitcase-shaped platform. Using this dataset, we do extensive experimentation on different temporal windows as input using an exhaustive list of state-of-the-art convolutional neural networks (CNNs). Our results show that our proposed multi-stream CNN is the best model for predicting time to nearcollision. The average prediction error of our time to nearcollision is 0.75 seconds across the test videos. The project webpage can be found at https://aashi7.github.io/ NearCollision.html.

Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud

“…3D object detection from a single image (monocular vision) is an indispensable part of future autonomous driving [51] and robot vision [28] because a single cheap onboard camera is readily available in most modern cars. Successful modern day methods for 3D object detection heavily rely on 3D sensors, such as a depth camera, a stereo camera or a laser scanner (i.e., LiDAR), which can provide explicit 3D information about the entire scene.…”

Time- and Resource-Efficient Time-to-Collision Forecasting for Indoor Pedestrian Obstacles Avoidance

“…Our method struggles more with predictions for further pedestrians (Time-to-Collision > 3 s), however, produces slightly smoother and more accurate results for closer pedestrians in the Near-Collision test set (Figure 12). [25]. It is a multi-stream that takes 6 consecutive RGB frames as input through the same convolutional network.…”

“…Figure 6. Three different test videos from the Near-Collision dataset with their labeled Time-to-Collision ground truth and their predicted time[25].…”

“…Figure 11. Near-Collision network architecture from[25]. It is a multi-stream that takes 6 consecutive RGB frames as input through the same convolutional network.…”

“…Data Availability Statement: Our generated dataset for testing is not publicly available. The NearCollision [25] dataset and code is available at https://aashi7.github.io/NearCollision.html (accessed on 15 February 2021).…”