Don’t Worry About the Weather: Unsupervised Condition-Dependent Domain Adaptation

Abstract

Modern models that perform system-critical tasks such as segmentation and localization exhibit good performance and robustness under ideal conditions (i.e. daytime, overcast) but performance degrades quickly and often catastrophically when input conditions change. In this work, we present a domain adaptation system that uses light-weight input adapters to pre-processes input images, irrespective of their appearance, in a way that makes them compatible with off-the-shelf computer vision tasks that are trained only on inputs with ideal conditions. No fine-tuning is performed on the off-the-shelf models, and the system is capable of incrementally training new input adapters in a self-supervised fashion, using the computer vision tasks as supervisors, when the input domain differs significantly from previously seen domains. We report large improvements in semantic segmentation and topological localization performance on two popular datasets, RobotCar and BDD.

HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images

“…Other recent works explore model adaptation from daytime to nighttime via an intermediate twilight domain [6], [22]. Following a different approach, works were proposed that conduct unpaired image-to-image translation using generative models to create synthetic nighttime training data [19], [24], [21]. Most similar to our work, in [32], the authors investigate adversarial domain adaptation, where they use a binary classifier to discriminate between daytime and nighttime image features produced by an encoder network.…”

See clearer at night: towards robust nighttime semantic segmentation through day-night image conversion

“…36,37 Unsupervised learning has also been frequently leveraged to pre-process input images, in order to prevent performance from degrading catastrophically when the input domain differs significantly from previously seen domains. 15,38 Specifically, this research line is also highly related to topological localization, 38,39 where modern visual localizers like 40,41 can also benefit from the input adaptation to perform more reliably against variation challenges. More recently, model distillation/imitation were applied to make model behave stable in unseen domains.…”

ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding

“…The normal-to-adverse domain adaptation scenario for semantic segmentation, which is much more relevant for realworld deployment of autonomous cars due to the difficulty of both acquiring and annotating adverse-condition data, has largely been overlooked. In particular, much fewer works consider normal-to-adverse adaptation in their experiments [10,11,32,37,38,39,40] and whenever they do, they either restrict the target adverse domain to a single condition, e.g. nighttime [10,39,40], fog [37,38], or rain [11], or do not include a quantitative evaluation on the real tar-get domain altogether [32].…”

“…In particular, much fewer works consider normal-to-adverse adaptation in their experiments [10,11,32,37,38,39,40] and whenever they do, they either restrict the target adverse domain to a single condition, e.g. nighttime [10,39,40], fog [37,38], or rain [11], or do not include a quantitative evaluation on the real tar-get domain altogether [32]. We attribute this fragmentation of normal-to-adverse adaptation works to the absence of a general large-scale dataset for semantic segmentation that evenly covers the majority of common adverse conditions and provides reliable ground truth for a sound evaluation in such challenging domains.…”