Collaborative Large-Scale Dense 3D Reconstruction with Online Inter-Agent Pose Optimisation

Abstract

Fig. 1: Globally consistent reconstructions produced by our approach, based on the Flat, House and Lab subsets of our dataset.Abstract-Reconstructing dense, volumetric models of real-world 3D scenes is important for many tasks, but capturing large scenes can take significant time, and the risk of transient changes to the scene goes up as the capture time increases. These are good reasons to want instead to capture several smaller sub-scenes that can be joined to make the whole scene. Achieving this has traditionally been difficult: joining sub-scenes that may never have been viewed from the same angle requires a high-quality camera relocaliser that can cope with novel poses, and tracking drift in each sub-scene can prevent them from being joined to make a consistent overall scene. Recent advances, however, have significantly improved our ability to capture medium-sized sub-scenes with little to no tracking drift: real-time globally consistent reconstruction systems can close loops and re-integrate the scene surface on the fly, whilst new visual-inertial odometry approaches can significantly reduce tracking drift during live reconstruction. Moreover, high-quality regression forest-based relocalisers have recently been made more practical by the introduction of a method to allow them to be trained and used online. In this paper, we leverage these advances to present what to our knowledge is the first system to allow multiple users to collaborate interactively to reconstruct dense, voxel-based models of whole buildings using only consumer-grade hardware, a task that has traditionally been both time-consuming and dependent on the availability of specialised hardware. Using our system, an entire house or lab can be reconstructed in under half an hour and at a far lower cost than was previously possible.Moreover, the risk of transient changes to the scene (e.g. people moving around) goes up as the capture time increases, corrupting the model and forcing the user to restart the capture. There are thus good reasons to want to split the capture into several shorter sequences, which can be captured either over multiple sessions or in parallel (by multiple users) and then joined to make the whole scene.Achieving this has traditionally been difficult: joining the sub-scenes requires the ability to accurately determine the relative transformations between them (a problem that can be expressed as camera relocalisation), even though the areas in which they overlap may never have been viewed from the same angles, and tracking drift in each sub-scene can prevent them from being joined to make a consistent overall scene. Recent advances, however, have significantly improved our ability to capture consistent, medium-sized sub-scenes, e.g. by closing loops and re-integrating the scene surface on the fly [17], which yields accurate poses for individual frames once loops have been closed, or by combining visual and inertial cues using an extended Kalman filter [28] to achieve accurate camera tracking during live reconstruction. Moreo...

Efficient 3D Reconstruction and Streaming for Group-Scale Multi-Client Live Telepresence

“…Furthermore, the replacement of the TSDF representation by a highresolution binary voxel grid has also been considered by Reichl et al [25] to improve the scalability and reduce the memory requirements. Recent extensions include the detection of loop closures [2,11,16] to reduce drift artifacts in camera localization as well as multi-client collaborative acquisition and reconstruction of static scenes [6]. This progress in real-time capturing enabled the development of various telepresence applications.…”

“…Note that this downsampling is only performed during allocation whereas the whole depth image is still used for data fusion to employ TSDF-based regularization. In the context of globally-consistent 3D reconstruction using bundle-adjusted submaps [6,11], our filter improves the compactness of the respective submap into which the RGB-D data Algorithm 1 Our optimized server voxel block data integration Input: Received TSDF voxel block positions P T SDF and voxel data V T SDF Output: Voxel block position list P MC for updating the stream sets 1:…”

“…However, in contrast to previous garbage collection approaches [14,23], we do not remove these voxel blocks but only ignore them. This avoids accidental removal of blocks that might become stable at a future time when this scene part is also partially stored in a different submap or revisited by the user or another client in multiclient acquisition setups [6,11]. Furthermore, by ignoring unstable data, the raycasted view will also be consistent with the exploration client's version of the 3D model.…”

“…As a result, efficiently representing and transmitting the scene to visualization devices is significantly harder. While further work has been spent on parallelized capturing [6], the goal of immersing a large number of people into the same live-captured environment while allowing them to interact with each other for e.g. remote collaboration and exploration scenarios, to the best of our knowledge, has not received a lot of attention so far.…”

The Mobile AR Sensor Logger for Android and iOS devices

“…Images from the front cameras were captured infrequently in the UMDAA datasets [8]. The collaborative SLAM dataset [9] provided color and depth frames captured with an Asus ZenFone AR smartphone at 5Hz. The ADVIO dataset [10] contained sessions lasting up to 7 minutes of camera frames at 60Hz and inertial data at 100Hz captured by an iPhone 6S.…”