4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

Abstract

In many robotics and VR/AR applications, 3D-videos are readily-available sources of input (a continuous sequence of depth images, or LIDAR scans). However, these 3D-videos are processed frame-by-frame either through 2D convnets or 3D perception algorithms in many cases. In this work, we propose 4-dimensional convolutional neural networks for spatio-temporal perception that can directly process such 3D-videos using high-dimensional convolutions. For this, we adopt sparse tensors [8, 9] and propose the generalized sparse convolution which encompasses all discrete convolutions. To implement the generalized sparse convolution, we create an open-source auto-differentiation library for sparse tensors that provides extensive functions for highdimensional convolutional neural networks. 1 We create 4D spatio-temporal convolutional neural networks using the library and validate them on various 3D semantic segmentation benchmarks and proposed 4D datasets for 3D-video perception. To overcome challenges in the high-dimensional 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and the trilateral-stationary conditional random field that enforces spatio-temporal consistency in the 7D space-time-chroma space. Experimentally, we show that convolutional neural networks with only generalized sparse convolutions can outperform 2D or 2D-3D hybrid methods by a large margin. 2 Also, we show that on 3D-videos, 4D spatio-temporal convolutional neural networks are robust to noise, outperform 3D convolutional neural networks and are faster than the 3D counterpart in some cases.

Interactive Gibson Benchmark: A Benchmark for Interactive Navigation in Cluttered Environments

“…These proposals are then refined and further annotated by humans. We use a pretrained Minkowski indoor semantic segmentation model [42] to predict per-voxel semantic labels ( Fig. 3.2).…”

MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequences

“…Fast-and-Furious [15] pro-posed to view the vertical dimension as feature channels and apply 3D convolutions on the remaining three dimensions. MinkowskiNet [4] explicitly used sparse 4D convolution on a 4D occupancy grid. Instead of quantizing the raw point clouds into an occupancy grid, our method directly processes point clouds.…”

“…We used a toy version of two recent grid-based deep architectures for dynamic 3D scenes, FaF [15] and MinkNet [4], as well as our MeteorNet. We only allow three layers of neurons within the three network architectures and convolution kernel sizes of no larger than 3.…”

“…Unlike 2D image videos which can be represented as a regular spatiotemporal 3D grid and learned by 3D convolutional neural networks (CNNs) [27,10,2], dynamic 3D point cloud sequences have an entirely different structure and are more challenging to learn. Recently, deep architectures were proposed for dynamic point cloud sequences that convert the irregular 3D data to a grid representation and leverage 3D or 4D convolutions along the time dimension [15,4]. However, the grid quantization error is inevitable in these methods and can be fatal for robotic applications that require precise localization.…”

“…However, the grid quantization error is inevitable in these methods and can be fatal for robotic applications that require precise localization. Moreover, performing convolution with a shared and fixed kernel everywhere in the 3D or 4D space is either inefficient or requires special engineering efforts such as sparse convolution [4].…”