UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation

Abstract

The fusion of multiple sensor modalities, especially through deep learning architectures, has been an active area of study. However, an under-explored aspect of such work is whether the methods can be robust to degradations across their input modalities, especially when they must generalize to degradations not seen during training. In this work, we propose an uncertainty-aware fusion scheme to effectively fuse inputs that might suffer from a range of known and unknown degradations. Specifically, we analyze a number of uncertainty measures, each of which captures a different aspect of uncertainty, and we propose a novel way to fuse degraded inputs by scaling modalityspecific output softmax probabilities. We additionally propose a novel data-dependent spatial temperature scaling method to complement these existing uncertainty measures. Finally, we integrate the uncertainty-scaled output from each modality using a probabilistic noisy-or fusion method. In a photo-realistic simulation environment (AirSim), we show that our method achieves significantly better results on a semantic segmentation task, compared to state-of-art fusion architectures, on a range of degradations (e.g. fog, snow, frost, and various other types of noise), some of which are unknown during training. We specifically improve upon the state-of-art [30] by 28% in mean IoU on various degradations.

Posterior Re-calibration for Imbalanced Datasets

“…Because the rain condition is not encountered during training, the rgb channel failed to cope with the degradation. UNO [19] dynamically shifts the fusion algorithms weight to the depth channel while our imbalance calibration method gives more weights to small objects. The qualitative and quatitative results in Sec 4.4.2 demonstrate the effectiveness of our unifed perspective 3.1 and Alg.…”

“…UNO [19] introduced an uncertainty-aware temperature scaling factor δ(x) which depends on a network's input x. The uncertainty-aware scalar correlates with uncertainty in an input image.…”

“…1 for prior and likelihood shift and apply it to multimodal tasks. We use noisy-or operation as the final fusion layer as in UNO [19]. We demonstrate the performance of the algorithm on a RGB-D semantic segmentation task in Seq.…”

“…1 for prior and likelihood shift, we conduct RGB-D multimodal semantic segmentation experiments on Synthia out-of-distribution splits which consist of seven weather conditions not seen during training. In this experiment, we compare to a simple baseline, SoftAve, which simply adds RGB and Depth predictions and averages the two, and UNO [19] which is designed for unseen degradations. In Table 5, we report both mean IOU and mean accuracy.…”