Overcoming Missing and Incomplete Modalities with Generative Adversarial Networks for Building Footprint Segmentation

Bischke, Benjamin; Helber, Patrick; Koenig, Florian; Borth, Damian; Dengel, Andreas

doi:10.1109/cbmi.2018.8516271

Cited by 18 publications

(17 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The typical generative networks include generative adversarial network(GAN) [8] and variational auto encoder [11]. For modality missing problem, a group of methods [2,3,19,26] utilize GAN or its varieties including cGAN [15] and cycleGAN [33] to generate the data of missing modalities. The CRA [21] use cascaded residual autoencoder adapted from stacked denoising autoencoder [24] to calculate the residual and reconstruct the corrupted multimodal data sequence.…”

Section: Generative Networkmentioning

confidence: 99%

Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis

Yuan

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Improving robustness against data missing has become one of the core challenges in Multimodal Sentiment Analysis (MSA), which aims to judge speaker sentiments from the language, visual, and acoustic signals. In the current research, translation-based methods and tensor regularization methods are proposed for MSA with incomplete modality features. However, both of them fail to cope with random modality feature missing in non-aligned sequences. In this paper, a transformer-based feature reconstruction network (TFR-Net) is proposed to improve the robustness of models for the random missing in non-aligned modality sequences. First, intramodal and inter-modal attention-based extractors are adopted to learn robust representations for each element in modality sequences. Then, a reconstruction module is proposed to generate the missing modality features. With the supervision of SmoothL1Loss between generated and complete sequences, TFR-Net is expected to learn semantic-level features corresponding to missing features. Extensive experiments on two public benchmark datasets show that our model achieves good results against data missing across various missing modality combinations and various missing degrees. CCS CONCEPTS• Information systems → Multimedia and multimodal retrieval; Web log analysis; Web applications; • Computing methodologies → Natural language processing.

show abstract

Section: Generative Networkmentioning

confidence: 99%

Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis

Yuan

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…This may be attributed to the characteristics of the generative model or we have not found how to use GAN in some special applications. In [104], GAN was used to translate the input of RGB image into a synthetic representation of the missing one (synthetic depth), and make an improvement to semantic segmentation of building footprints with missing modalities. In reference [103], to compensate for the poor quality of the point clouds, in generator, the proposed method adds the crossmodal guidance from the side-output features of the RGB stream to the decoder network of depth stream.…”

Section: B Point Clouds and Optical Image Fusionmentioning

confidence: 99%

“…In some applications, the data collection process is likely to generate data with missing, incomplete, or corrupted modalities. In [140], the authors focus on semantic segmentation of building footprints with missing modalities, in which GAN are effectively used to synthesis the missing or incomplete data in the depth map.…”

Section: A Missing Data Reconstructionmentioning

confidence: 99%

A Review on Remote Sensing Data Fusion with Generative Adversarial Networks (GAN)

Liu¹

2021

Preprint

View full text Add to dashboard Cite

In the past decades, remote sensing (RS) data fusion has always been an active research community. A large number of algorithms and models have been developed. Generative Adversarial Networks (GAN), as an important branch of deep learning, show promising performances in variety of RS image fusions. This review provides an introduction to GAN for remote sensing data fusion. We briefly review the frequently-used architecture and characteristics of GAN in data fusion and comprehensively discuss how to use GAN to realize fusion for homogeneous RS data, heterogeneous RS data, and RS and ground observation data. We also analyzed some typical applications with GAN-based RS image fusion. This review takes insight into how to make GAN adapt to different types of fusion tasks and summarizes the advantages and disadvantages of GAN-based RS data fusion. Finally, we discuss the promising future research directions and make a prediction on its trends.

show abstract

“…The issue of a completely missing modality is emphasized in [12], where a hallucination network is used to recreate features from the missing modality using the available modality. Bischke et al [13] extend this idea by using GANs to instead recreate the missing modality and feeding into a segmentation network along with the original modality. [14] uses a different approach to handle missing modality for medical imaging fusion by training with dropout of modalities and using late fusion, which allows for independent training of separate modalities.…”

Section: Missing Modalities In Fusionmentioning

confidence: 99%

Generative Information Fusion

Tran

Sakla

Krim

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In this work, we demonstrate the ability to exploit sensing modalities for mitigating an unrepresented modality or for potentially re-targeting resources. This is tantamount to developing proxy sensing capabilities for multi-modal learning. In classical fusion, multiple sensors are required to capture different information about the same target. Maintaining and collecting samples from multiple sensors can be financially demanding. Additionally, the effort necessary to ensure a logical mapping between the modalities may be prohibitively limiting. We examine the scenario where we have access to all modalities during training, but only a single modality at testing. In our approach, we initialize the parameters of our single modality inference network with weights learned from the fusion of multiple modalities through both classification and GANs losses. Our experiments show that emulating a multimodal system by perturbing a single modality with noise can help us achieve competitive results compared to using multiple modalities.

show abstract

Overcoming Missing and Incomplete Modalities with Generative Adversarial Networks for Building Footprint Segmentation

Cited by 18 publications

References 29 publications

Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis

Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis

A Review on Remote Sensing Data Fusion with Generative Adversarial Networks (GAN)

Generative Information Fusion

Contact Info

Product

Resources

About