Iterative Online 3D Reconstruction from RGB Images

Abstract

3D reconstruction is the computer vision task of reconstructing the 3D shape of an object from multiple 2D images. Most existing algorithms for this task are designed for offline settings, producing a single reconstruction from a batch of images taken from diverse viewpoints. Alongside reconstruction accuracy, additional considerations arise when 3D reconstructions are used in real-time processing pipelines for applications such as robot navigation or manipulation. In these cases, an accurate 3D reconstruction is already required while the data gathering is still in progress. In this paper, we demonstrate how existing batch-based reconstruction algorithms lead to suboptimal reconstruction quality when used for online, iterative 3D reconstruction and propose appropriate modifications to the existing Pix2Vox++ architecture. When additional viewpoints become available at a high rate, e.g., from a camera mounted on a drone, selecting the most informative viewpoints is important in order to mitigate long term memory loss and to reduce the computational footprint. We present qualitative and quantitative results on the optimal selection of viewpoints and show that state-of-the-art reconstruction quality is already obtained with elementary selection algorithms.

DataCurator.jl: efficient, portable and reproducible validation, curation and transformation of large heterogeneous datasets using human-readable recipes compiled into machine-verifiable templates

“…On top of standard file operations (copy, move, rename, delete, match), DataCurator can operate on the content of images, tables (CSV), HDF5, six single molecule localization microscopy formats ( Cardoen, 2022 ), MAT, JSON, SQLite (file-based databases) and common mesh (geometry) formats and can be further extended to include new datatypes. While DataCurator examples currently focus on biomedical images, there are no technical restrictions to the kind of data it can process, with interfaces for processing files abstracted away to enable easy inclusion of new data type operations.…”