HeteroFusion: Dense Scene Reconstruction Integrating Multi-Sensors

Yang, Songfan; Li, Beichen; Liu, Minghua; Lai, Yu‐Kun; Kobbelt, Leif; Hu, Shi‐Min

doi:10.1109/tvcg.2019.2919619

Cited by 14 publications

(5 citation statements)

References 46 publications

(96 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yet, these methods study specific sensors and are not inherently equipped with 3D reasoning. Few works consider 3D reconstruction with multiple sensors [57,31,7,76,77,24], but these do not consider the online mapping setting. Conceptually, more closely related to our work is SenFu-Net [58], which is an online mapping method for multisensor depth fusion.…”

Section: Multi-sensor Depth Fusionmentioning

confidence: 99%

UncLe-SLAM: Uncertainty Learning for Dense Neural SLAM

Sandström,

Ta,

Van Gool

et al. 2023

2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

View full text Add to dashboard Cite

We present an uncertainty learning framework for dense neural simultaneous localization and mapping (SLAM). Estimating pixel-wise uncertainties for the depth input of dense SLAM methods allows re-weighing the tracking and mapping losses towards image regions that contain more suitable information that is more reliable for SLAM. To this end, we propose an online framework for sensor uncertainty estimation that can be trained in a self-supervised manner from only 2D input data. We further discuss the advantages of the uncertainty learning for the case of multi-sensor input. Extensive analysis, experimentation, and ablations show that our proposed modeling paradigm improves both mapping and tracking accuracy and often performs better than alternatives that require ground truth depth or 3D. Our experiments show that we achieve a 38% and 27% lower absolute trajectory tracking error (ATE) on the 7-Scenes and TUM-RGBD datasets respectively. On the popular Replica dataset using two types of depth sensors, we report an 11% F1-score improvement on RGBD SLAM compared to the recent state-of-the-art neural implicit approaches.

show abstract

Section: Multi-sensor Depth Fusionmentioning

confidence: 99%

UncLe-SLAM: Uncertainty Learning for Dense Neural SLAM

Sandström,

Ta,

Van Gool

et al. 2023

2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

View full text Add to dashboard Cite

show abstract

“…Therefore, some processing is required to upscale the spatial resolution for the measured depth. This upscaling can be performed by tempo-spatial supersampling of the depth data, with Inertial Measurement Unit (IMU) and position-tracking [29]. However, this is only feasible for static scenes and moving sensors.…”

Section: Depth Upscalingmentioning

confidence: 99%

Camera and Lidar-Based View Generation for Augmented Remote Operation in Mining Applications

Dima

Sjöström

2021

IEEE Access

View full text Add to dashboard Cite

Remote operation of diggers, scalers, and other tunnel-boring machines has significant benefits for worker safety in underground mining. Real-time augmentation of the presented remote views can further improve the operator effectiveness through a more complete presentation of relevant sections of the remote location. In safety-critical applications, such augmentation cannot depend on preconditioned data, nor generate plausible-looking yet inaccurate sections of the view. In this paper, we present a capture and rendering pipeline for real time view augmentation and novel view synthesis that depends only on the inbound data from lidar and camera sensors. We suggest an on-the-fly lidar filtering for reducing point oscillation at no performance cost, and a full rendering process based on lidar depth upscaling and in-view occluder removal from the presented scene. Performance assessments show that the proposed solution is feasible for real-time applications, where per-frame processing fits within the constraints set by the inbound sensor data and within framerate tolerances for enabling effective remote operation.

show abstract

“…Panoramic images have a wider perspective than typical perspective images, contain more comprehensive buildings and a stronger sense of reality, and have richer scene details [3]. Owing to the abundant resources and wide field of view (FOV), panoramic images have considerable advantages over perspective images in large-scale, three-dimensional (3D) scene reconstruction [4,5]. Three-dimensional scene reconstruction [6][7][8][9] has been comprehensively studied and applied to digital cities, virtual reality [10], and SLAM [11,12] because of its significant advantages in terms of model accuracy, realism, and ease of modeling.…”

Section: Introductionmentioning

confidence: 99%

Leveraging CNNs for Panoramic Image Matching Based on Improved Cube Projection Model

et al. 2023

View full text Add to dashboard Cite

Three-dimensional (3D) scene reconstruction plays an important role in digital cities, virtual reality, and simultaneous localization and mapping (SLAM). In contrast to perspective images, a single panoramic image can contain the complete scene information because of the wide field of view. The extraction and matching of image feature points is a critical and difficult part of 3D scene reconstruction using panoramic images. We attempted to solve this problem using convolutional neural networks (CNNs). Compared with traditional feature extraction and matching algorithms, the SuperPoint (SP) and SuperGlue (SG) algorithms have advantages for handling images with distortions. However, the rich content of panoramic images leads to a significant disadvantage of these algorithms with regard to time loss. To address this problem, we introduce the Improved Cube Projection Model: First, the panoramic image is projected into split-frame perspective images with significant overlap in six directions. Second, the SP and SG algorithms are used to process the six split-frame images in parallel for feature extraction and matching. Finally, matching points are mapped back to the panoramic image through coordinate inverse mapping. Experimental results in multiple environments indicated that the algorithm can not only guarantee the number of feature points extracted and the accuracy of feature point extraction but can also significantly reduce the computation time compared to other commonly used algorithms.

show abstract

HeteroFusion: Dense Scene Reconstruction Integrating Multi-Sensors

Cited by 14 publications

References 46 publications

UncLe-SLAM: Uncertainty Learning for Dense Neural SLAM

UncLe-SLAM: Uncertainty Learning for Dense Neural SLAM

Camera and Lidar-Based View Generation for Augmented Remote Operation in Mining Applications

Leveraging CNNs for Panoramic Image Matching Based on Improved Cube Projection Model

Contact Info

Product

Resources

About