PVO: Panoptic Visual Odometry

Abstract

We present a novel panoptic visual odometry framework, termed PVO, to achieve a more comprehensive modeling of the scene's motion, geometry, and panoptic segmentation information. PVO models visual odometry (VO) and video panoptic segmentation (VPS) in a unified view, enabling the two tasks to facilitate each other. Specifically, we introduce a panoptic update module into the VO module, which operates on the image panoptic segmentation. This Panoptic-Enhanced VO module can trim the interference of dynamic objects in the camera pose estimation by adjusting the weights of optimized camera poses. On the other hand, the VO-Enhanced VPS module improves the segmentation accuracy by fusing the panoptic segmentation result of the current frame on the fly to the adjacent frames, using geometric information such as camera pose, depth, and optical flow obtained from the VO module. These two modules contribute to each other through a recurrent iterative optimization. Extensive experiments demonstrate that PVO outperforms state-of-the-art methods in both visual odometry and video panoptic segmentation tasks. Code and data are available on the project webpage: https://zju3dv.github.io/pvo/.

iDF-SLAM: End-to-End RGB-D SLAM with Neural Implicit Mapping and Deep Feature Tracking

“…Depending on the constructed maps, conventional RGB-D SLAM systems are commonly categorised into sparse SLAM [1], [2], dense SLAM [3]- [5] and hybrid ones [6], [7]. Later, with advances in deep neural networks (DNNs), much attention has been paid to improve various aspects of SLAM with semantic information extracted with DNNs, such as meaningful mapping [8]- [10], dynamic tracking [11]- [13], relocalisation [14], [15], etc. Although these systems have demonstrated promising performance in terms of both tracking and reconstruction accuracy, they suffer from the huge video memory (VRAM) footprint in storing the reconstructed map and the computational consumption when modifying the map on-the-fly.…”