MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects

Rünz, Martin; Buffier, Maud; Agapito, Lourdes

doi:10.1109/ismar.2018.00024

Cited by 314 publications

(286 citation statements)

References 50 publications

Supporting

Mentioning

286

Contrasting

Order By: Relevance

“…We will not fail to test and compare these new network structures in the near future. Hybrid approaches [36] will also be another option to speed up the segmentation process. Another direction in our future work will deal with stereoscopic consistency, which we have not addressed yet.…”

Section: Discussionmentioning

confidence: 99%

Using CNNs For Users Segmentation In Video See-Through Augmented Virtuality

Pigny

Dominjon

2019

2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR)

View full text Add to dashboard Cite

In this paper, we present preliminary results on the use of deep learning techniques to integrate the user's selfbody and other participants into a head-mounted video seethrough augmented virtuality scenario. It has been previously shown that seeing user's bodies in such simulations may improve the feeling of both self and social presence in the virtual environment, as well as user performance. We propose to use a convolutional neural network for real time semantic segmentation of users' bodies in the stereoscopic RGB video streams acquired from the perspective of the user. We describe design issues as well as implementation details of the system and demonstrate the feasibility of using such neural networks for merging users' bodies in an augmented virtuality simulation.

show abstract

Section: Discussionmentioning

confidence: 99%

Using CNNs For Users Segmentation In Video See-Through Augmented Virtuality

Pigny

Dominjon

2019

2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR)

View full text Add to dashboard Cite

show abstract

“…Dynamic object reconstruction is closely related to MOT, as to perform reconstruction first tracking needs to be performed. A number of methods [2], [31] have approached this task. Compared to these methods, our method finishes the loop; using these reconstructions to further improve tracking.…”

Section: Related Workmentioning

confidence: 99%

Track to Reconstruct and Reconstruct to Track

Luiten

Fischer

Leibe

2020

IEEE Robot. Autom. Lett.

102

View full text Add to dashboard Cite

Object tracking and 3D reconstruction are often performed together, with tracking used as input for reconstruction. However, the obtained reconstructions also provide useful information for improving tracking. We propose a novel method that closes this loop, first tracking to reconstruct, and then reconstructing to track. Our approach, MOTSFusion (Multi-Object Tracking, Segmentation and dynamic object Fusion), exploits the 3D motion extracted from dynamic object reconstructions to track objects through long periods of complete occlusion and to recover missing detections. Our approach first builds up short tracklets using 2D optical flow, and then fuses these into dynamic 3D object reconstructions. The precise 3D object motion of these reconstructions is used to merge tracklets through occlusion into long-term tracks, and to locate objects when detections are missing. On KITTI, our reconstructionbased tracking reduces the number of ID switches of the initial tracklets by more than 50%, and outperforms all previous approaches for both bounding box and segmentation tracking.

show abstract

“…Bescos et al [1] combine a geometric approach with a deep learning segmentation to enable removal of dynamics in an ORB-SLAM2 system [8]. Similarly, Rünz et al [13] exploit deep learning segmentation, refined with a geometric approach, to detect objects in the environment. In addition, they reconstruct and track all the detected objects independently.…”

Section: Related Workmentioning

confidence: 99%

ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residuals

Palazzolo

Behley

Lottes

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

102

View full text Add to dashboard Cite

Mapping and localization are essential capabilities of robotic systems. Although the majority of mapping systems focus on static environments, the deployment in real-world situations requires them to handle dynamic objects. In this paper, we propose an approach for an RGB-D sensor that is able to consistently map scenes containing multiple dynamic elements. For localization and mapping, we employ an efficient direct tracking on the truncated signed distance function (TSDF) and leverage color information encoded in the TSDF to estimate the pose of the sensor. The TSDF is efficiently represented using voxel hashing, with most computations parallelized on a GPU. For detecting dynamics, we exploit the residuals obtained after an initial registration, together with the explicit modeling of free space in the model. We evaluate our approach on existing datasets, and provide a new dataset showing highly dynamic scenes. These experiments show that our approach often surpass other state-of-the-art dense SLAM methods. We make available our dataset with the ground truth for both the trajectory of the RGB-D sensor obtained by a motion capture system and the model of the static environment using a high-precision terrestrial laser scanner. Finally, we release our approach as open source code.

show abstract

MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects

Cited by 314 publications

References 50 publications

Using CNNs For Users Segmentation In Video See-Through Augmented Virtuality

Using CNNs For Users Segmentation In Video See-Through Augmented Virtuality

Track to Reconstruct and Reconstruct to Track

ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residuals

Contact Info

Product

Resources

About