Assessing out-of-domain generalization for robust building damage detection

Abstract

An important step for limiting the negative impact of natural disasters is rapid damage assessment after a disaster occurred. For instance, building damage detection can be automated by applying computer vision techniques to satellite imagery. Such models operate in a multi-domain setting: every disaster is inherently different (new geolocation, unique circumstances), and models must be robust to a shift in distribution between disaster imagery available for training and the images of the new event. Accordingly, estimating real-world performance requires an out-of-domain (OOD) test set. However, building damage detection models have so far been evaluated mostly in the simpler yet unrealistic in-distribution (IID) test setting. Here we argue that future work should focus on the OOD regime instead. We assess OOD performance of two competitive damage detection models and find that existing state-of-the-art models show a substantial generalization gap: their performance drops when evaluated OOD on new disasters not used during training. Moreover, IID performance is not predictive of OOD performance, rendering current benchmarks uninformative about real-world performance. Code and model weights are available at https://github.com/ecker-lab/robust-bdd.

On Transfer Learning for Building Damage Assessment from Satellite Imagery in Emergency Contexts

“…Their in-depth study of per-disaster performance led them to propose a specialized model for each disaster type. Benson and Ecker [40] highlighted the unrealistic test setting in which damage assessment models were developed and proposed a new formulation based on out-of-domain distribution. They experimented with two domain adaptation techniques, multi-domain AdaBN [41] and stochastic weight averaging [42].…”

BLDNet: A Semi-supervised Change Detection Building Damage Framework using Graph Convolutional Networks and Urban Domain Knowledge

“…The works [29,41] concerned with developing methodologies to infer new disasters used model architectures similar to the ones described in the previous paragraph. Other studies extrapolated to different disasters types [32,42,43,44,45]. In many of these cases, transferability is dependent on the similarity between to the train and test regions [29] and is usually degraded by data heterogeneity [29,44].…”

“…In many of these cases, transferability is dependent on the similarity between to the train and test regions [29] and is usually degraded by data heterogeneity [29,44]. Other solutions proposed including a small number of test samples with the training data [41,44], a multi-domain adaptive batch normalization and a stochastic weight averaging [42].…”

Exploiting Position Bias for Robust Aspect Sentiment Classification

“…In terms of O.O.D. robustness, they often use the crossdomain setting to evaluate models (Benson and Ecker, 2020). Previous work mainly focuses on how to minimize the domain discrepancy and how to improve the feature adaptability of models (Rietzler et al, 2020;Ye et al, 2020).…”