Generation of a Benchmark Dataset Using Historical Photographs for an Automated Evaluation of Different Feature Matching Methods

Abstract

<p><strong>Abstract.</strong> This contribution shows the generation of a benchmark dataset using historical images. The difficulties when working with historical images are pointed out and structured in three categories. Especially large viewpoint differences, image artifacts and radiometric differences lead to weak matching results with classical feature matching approaches. The necessity of publishing an own benchmark dataset is emphasized when comparing to existing datasets which are partly using synthetic data, well-known orientation or strictly categorized image differences. The presented image dataset consists at the moment of 24 images which are oriented in image triples using the properties of the Trifocal Tensor as a more stable image geometry. In the following, three different feature detectors and descriptors that have already been proven well on historical images (MSER, ORB, RIFT) are evaluated using the new benchmark dataset. Then, several outlier removal methods were applied on the detected features. The tests show that for the entirety of image pairs RIFT performs slightly better than the other two methods. Nonetheless, for some image pairs MSER significantly improves the matching score but even so, historical image pairs are difficult to be matched with the presented methods due to challenging outlier removal. Still, the estimated projective relative orientation could be used in an autocalibration approach to place the images in a metric scene.</p>

An automatic workflow for orientation of historical images with large radiometric and geometric differences

“…It has been identified that, in addition to camera calibration, feature extraction and description is considered to be the most difficult step in an SfM workflow when trying to retrieve the orientation of multiple historical images (Ali and Whitehead, 2014; Maiwald, 2019). While there exist many comprehensive surveys (Tuytelaars and Mikolajczyk, 2007; Fan et al, 2015; Csurka and Humenberger, 2018) and benchmarks (Mikolajczyk et al, 2005; Schönberger et al, 2017; Jin et al, 2021) of a large number of feature matching methods, there is very little research on evaluating feature matching on historical images (Maiwald, 2019). Consequently, the second part of this section on related research focuses on feature matching in image pairs with large illumination and/or viewpoint differences since these are quite comparable to pairs of historical photographs.…”

“…Nonetheless, for the presented datasets (see the Datasets section), it has been noticed that SIFT and its variants RootSIFT (Arandjelović and Zisserman, 2012) and DSP‐SIFT (Dong and Soatto, 2015) did not perform well and produced only few, or even no, matching image pairs. Further algorithmic methods that have been evaluated on historical images (Maiwald et al, 2019; Maiwald, 2019) are maximally stable extremal regions (MSER: Matas et al, 2004), radiation‐invariant feature transform (RIFT: J. Li et al, 2020), matching on demand with view synthesis (MODS: Mishkin et al, 2015a) and MODS‐WxBS (Mishkin et al, 2015b). Initially, the MSER method developed for wide baselines between image pairs produced decent results for historical image pairs, but was outperformed by a combination of RIFT and MODS/MODS‐WxBS.…”

“…The authors experienced that historical image pairs are often much more difficult to match than recent images because of the properties already explained in the Introduction section. Since only sparse ground truth for historical terrestrial images is available, a small dataset consisting of 24 real‐world example images (relative orientation of eight image triples) has been created in previous research (Maiwald, 2019). The ground‐truth image orientation between image triples is available in the format of a trifocal tensor (http://dx.doi.org/10.25532/OPARA-24).…”

Historical Photos and Visualizations: Potential for Research

“…The focal length of historical images is especially often unknown, which negatively affects the position accuracy in terms of the view direction of the camera. Further image and object differences hamper automatic image orientation (Maiwald, 2019). We propose visualizing the accuracy of the camera position in the interface using an error ellipsoid centered at the principal point of the camera.…”

Machine Learning Based Analysis of Finnish World War II Photographers

“…Despite the potential, up to this point the use of machine learning has been very scarce in this context. Previous works in the field include applications of face recognition to assist in identifying persons in historical portrait photographs [38], feature matching for geolocalization or target matching in historical repeat photography [2], [35], [36], [61], application of marked point processes on automatic detection of bomb craters in aerial wartime images [27], a rudimentary classification of historical photographs into portraits, landscapes, group photographs, and buildings/architectural photography [12].…”