SfM ToAEval: Trade-off Aware Evaluation of Feature Extraction Algorithms in Structure from Motion

Abstract

This paper presents SfM ToAEval, a framework for evaluating different feature extraction algorithms in Structure from Motion (SfM) pipelines. SfM ToAEval allows automatically evaluating the effect of using different feature detectors and descriptors combinations on the quality of the 3D reconstruction from a given collection of image sequences. In addition, SfM ToAEval evaluates the 3D reconstruction without the need for ground truth. Moreover, SfM ToAEval is aware of the reconstruction density-accuracy trade-off, and it supports visualizing it in order to allow deciding the "best" reconstruction transparently. Furthermore, SfM ToAEval allows quantifying the quality of each 3D reconstruction compared to others. SfM ToAEval was used to evaluate 98 feature detectors and descriptors combinations on six image sequences, and it was able to identify four promising combinations. Experimental results comparing the proposed combinations with related work are presented in this research. The complete source code of the proposed framework as well as a minimal Jupyter Notebook demonstrating how different functionalities can be used are released under the MIT license.