Towards a Fair Evaluation of Feature Extraction Algorithms Robustness in Structure from Motion

Abstract

Structure from Motion is a pipeline for 3D reconstruction in which the true geometry of an object or a scene is inferred from a sequence of 2D images. As feature extraction is usually the first phase in the pipeline, the reconstruction quality depends on the accuracy of the feature extraction algorithm. Fairly evaluating the robustness of feature extraction algorithms in the absence of reconstruction ground truth is challenging due to the considerable number of parameters that affect the algorithms' sensitivity and the tradeoff between reconstruction size and error. The evaluation methodology proposed in this paper is based on two elements. The first is using constrained 3D reconstruction, in which only fixed numbers of extracted and matched features are passed to subsequent phases. The second is comparing the 3D reconstructions using size-error curves (introduced in this paper) rather than the value of reconstruction size, error, or both. The experimental results show that the proposed methodology is more transparent.

SfM ToAEval: Trade-off Aware Evaluation of Feature Extraction Algorithms in Structure from Motion

“…SfM ToAEval is a stand-alone cross-platform software that is entirely developed using Python 3 and can be easily used on Google Colab or in a Jupyter Notebook on Windows, Linux, or macOS. SfM ToAEval not only automates the process of evaluation but also employs size-error curves proposed by Taha et al in [14] for visualizing the reconstruction densityaccuracy trade-off in order to allow deciding the best reconstruction transparently. As the visualization may become crowded when the number of combinations being evaluated grows as shown in Fig.…”

“…Feature Extraction Feature detection and feature description are automatically performed using OpenCV for all possible combinations of the feature detectors and feature descriptors provided to the proposed framework as illustrated in Figure 3. Out of the features detected by each algorithm, the subsequent stage receives only a fixed number in order to reduce the evaluation time and ensure fairness [14].…”

“…Feature extraction is automatically performed using OpenCV brute force matcher with cross-check enabled and the default norm for each feature descriptor. Out of the features matched by each algorithm, the subsequent stage receives only a fixed number in order to reduce the evaluation time and ensure fairness [14].…”

“…Therefore, a lot of attention has been paid during the past decade to the problem of deciding good feature detectors and descriptors combinations for SfM and it has become a hot area of research [4][5][6][7][8][9][10][11][12]. When using synthetic data (like SfM Flow does in [13]) is not an option, and the ground truth of the real data is not available, evaluation of the 3D reconstruction becomes very challenging as relying on the value of the reconstruction density (number of 3D points in the sparse point cloud) and accuracy (average reprojection error of the 3D points on the images) can be misleading [14]. The recent interest in the evaluation of different feature detectors and descriptors combinations in SfM, the challenge of evaluating 3D reconstruction in the absence of ground truth, and the increasing number of published feature extraction algorithms (for example, as of 2022, OpenCV implements about 30 different feature extraction algorithms [15]) motivated the development of SfM ToAEval.…”