Annotated reconstruction of 3D spaces using drones

Abstract

As the fields of robotics and drone technologies are continually advancing, the challenge of teaching these agents to learn and maneuver in the real world becomes increasingly important. A critical component of this is the ability for a robot to map and understand its surrounding unknown environment, both in terms of physical structure and object classification. In this project we tackle the challenge of mapping a 3D space with annotations using only 2D images acquired from a Parrot Drone. In order to make such a system operate efficiently in close to real time, we address a number challenges including (1) creating a optimized version of Faster RCNN that can operate on drone hardware while still being accurate, (2) developing a method to reconstruct 3D spaces from 2D images annotated with bounding boxes, and (3) using generated 3D annotations to complete drone motion planning for unknown space exploration.

Multi-Drone 3D Building Reconstruction Method

“…The papers that were published prior to 2015 were dropped. The resulting list contained 17 papers [3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19], which were filtered by excluding papers that did not comply to at least one of the following requirements: Due to the formulation of the problem task, reviewed papers should be characterized by a set of unified criteria that describe how the main challenges are handled. These criteria can be structured in a form of five questions for each paper:…”

“…Ref. [8] does not cover multi-agent interaction explicitly, but provides that the architecture of using neural networks for annotated 3D reconstruction can be scaled for a multi-agent environment by simply using data from several drones if there were to be a collision avoidance approach.…”

“…All reviewed solutions can be divided to three groups by the localization approach-SLAM-based [6,7,11,15,18,19], machine learning-based [5,8,12] and solutions that rely on any drone built-in technologies, which rely on a visual markup or do not specify any particular approach [3,4,9,10,13,14,16,17].…”

“…At [5] the localization is based on a consensus matrix (generated by a two-layer neural network) that calculates, among other things, the average error of the position. Solution [8] performs drone navigation by using a voxel grid reconstructed using the region-based convolutional neural network (FRCNN) and representing a rough model of a reconstructed environment. For solution [12] a drone is localized relative to the previous captured image by using a pretrained model by Scan-RL algorithm, which uses images as an input.…”

“…The reviewed solutions perform three main strategies for 3D reconstruction-usage of photogrammetry (i.e., one of the most common photogrammetric algorithms-structure from motion(SfM)) [3,4,6,9,10,13,14,[16][17][18], 3D reconstruction as a result of visual SLAM application [15,19] and machine learning approaches [8,12]. The authors of [5,7,11] do not explicitly describe 3D reconstruction methods.…”