Detecting Change for Multi-View, Long-Term Surface Inspection

Abstract

We describe a system for the detection of changes in multiple views of a textured surface taken at different times by a moving camera. Our motivation is the development of a non-contact inspection system, summarised in fig. 1, to be used for detecting anomalous visual changes on surfacesin this case on concrete tunnel linings. This application is of increasing social importance as tunnels and other large-scale infrastructure age and more efficient methods for structural inspection are required to allow their continued safe operation.The problem is challenging for several reasons: (i) Size and nature of changes. Changes of interest are often small and subtle -e.g. a fattening in the width of a hairline crack or a patch of discolouration caused by organic growth or surface damage. (ii) Nuisance factors. A sizeable proportion of the observed change over time is caused by nuisance factors, either internal to the acquisition system (such as different image sensors, capture settings or lighting setup) or due to external causes (for example, seasonal changes of temperature and humidity). (iii) Registration error. Achieving the pixel-accurate registration typically required for change detection is challenging because neither the sensor positions nor the tunnel geometry can be reliably determined. Parallax errors are common.We address these challenges by first using a structure-from-motion pipeline to approximately register images from our robotic inspection rig to the reconstructed surface of interest. Given a pair of registered image patches from different times, our main contribution is a novel approach to detect changes between the patches using a two-channel convolutional neural network (CNN). CNNs have recently been shown to be very effective at learning invariance to certain modes of image variability, but require large amounts of labelled image data to train. We create an unlimited source of negative pairs (i.e. patches where no abnormal change has occurred) by taking registered viewpoints from different cameras from the same time. We supplement this with a smaller dataset of negative pairs across the different test times from regions where no changes of interest have occurred. This requires a limited effort in coarsely labelling a small subset of the test data. Together, these negative pairs capture much of the natural variance from nuisance factors and registration error. For the positive (changed) pair generation, we provide randomly sampled pairs as well as synthetically generated changes using a crack model (fig. 2). The homogeneity of the tunnel environment allows a network to generalize well from a manageable amount of labelled ground-truth data.Our approach is similar to [1], who learn a two-channel CNN for the inverse problem of similarity measurement between image patch pairs. A key difference is that we train directly on a mixture of synthetic data and task data generated by our own pipeline, allowing us to learn task-specific invariances for the improved detection of changes.We evaluate our system usi...

Precise deterministic change detection for smooth surfaces

“…tamper detection, medical imaging and industrial inspection [3,5,6,7,10,11]. The problem involves the estimation of a change map between two images or sets of images of a scene, taken at different times.…”

Street-View Change Detection with Deconvolutional Networks

“…Despite having reasonable performance, the method relies on superpixel regularization and sky/ground segmentation to delineate changes accurately. Other works such as [36] propose training change detection networks from scratch on image patches to classify changes for industrial inspection. In contrast to these prior works, we adopt a deconvolutional network approach [28,31] and demonstrate its ability to learn an appropriate, spatially precise similarity function for this challenging outdoor problem.…”

“…The definition of what constitutes a change of interest or a nuisance change varies depending on the task. Changes of interest may be purely geometric, such as the appearance or disappearance of urban structures [33,38,39], or textural, such as changes in billboards or shop-fronts [25] or surface defects [36]. Prototypical nuisance changes which the similarity measure must be invariant to include vegetation (e.g.…”

“…For final experiments, we trained on the full training set without validation, performing early stopping after 150 epochs in all cases to ensure fairness. In addition, we used data augmentation to help prevent overfitting during training, by adding image pairs containing both artificial changes (by adding synthetic changes to existing images) and no changes of interest, in the manner of [36]. These were added in the approximate ratio of 35% real changes to 65% augmented changes.…”

“…Registration is needed because for typical large urban datasets, images are often sparsely sampled and it is common to have a 1 Available from: http://3dvis.ri.cmu.edu/data-sets/localization/ baseline of several metres when comparing one image to the closest matching picture across time (resulting in significant perspective deformations). Registration is typically addressed using Structure from Motion (SfM), followed by either multiview stereo techniques or model-based reconstruction, which allow the generation of surface models that can be used to warp images from one camera viewpoint to another via reprojection [25,36,38].…”