Architectural Heritage Recognition in Historical Film Footage Using Neural Networks

Abstract

<p><strong>Abstract.</strong> Researching historical archives for material suitable for photogrammetry is essential for the documentation and 3D reconstruction of Cultural Heritage, especially when this heritage has been lost or transformed over time. This research presents an innovative workflow which combines the photogrammetric procedure with Machine Learning for the processing of historical film footage. A Neural Network is trained to automatically detect frames in which architectural heritage appears. These frames are subsequently processed using photogrammetry and finally the resulting model is assessed for metric quality. This paper proposes best practises in training and validation on a Cultural Heritage asset. The algorithm was tested through a case study of the Tour Saint Jacques in Paris for which an entirely new dataset was created. The findings are encouraging both in terms of saving human effort and of improvement of the photogrammetric survey pipeline. This new tool can help researchers to better manage and organize historical information.</p>

A Match-Moving Method Combining Ai and SFM Algorithms in Historical Film Footage

“…Despite the presence of both false positives and false negatives in the final results, the evaluation of the performance of the Neural Networks for the automatic detection of the monuments has been substantially successful in terms of saving time and efficiency for the end user over a manual search. A full explanation and discussion of quantitative performance evaluation is provided in Condorelli et al, 2019. The processing of the videos was performed thanks to the use of the High-Performance Computing resources by CINECA that benefits greatly from the use of Graphical Processing Units (GPUs).…”

“…Object detection is a good solution in applications like monument recognition in film footage because it allows the tracking of the object also when the image is noisy, the camera is not stable and the object is with a complex structure (Parekh et al, 2014). The choice of the Neural Network, the training and the validation phases were widely explained in a previous paper of the authors (Condorelli et al, 2019) in which a workflow capable of automatically detecting architectural heritage in film footage was detailed. This workflow allows to extract the frames containing the architecture but they were subsequently filtered to be processed with photogrammetry in a manual way.…”

“…Only the shots taken from multiple points of view of the same scene are suitable for the photogrammetric process and the tilting and trucking camera motions have demonstrated to be more effective to perform this kind of frame selection (Condorelli and Rinaudo, 2019).…”

“…This step will be very useful during the evaluation of the metric quality of the results. The detection and extraction of these feature points is performed importing in COLMAP a text file in which the image coordinates (x, y) expressed in pixel and the scale and orientation information are indicated (Condorelli et al, 2019).…”

“…The results of the 3D reconstruction of the Heritage were compared with a benchmark specifically created to evaluate the metric quality of the model according to the type of camera motion used (Condorelli and Rinaudo, 2019). The assessment was completed with the scale of the model through the feature points selected during the photogrammetric process and the comparison with existing material from which extract metric information: a point cloud, if present or historical drawings, for example.…”