Deep multimodal fusion for semantic image segmentation: A survey

Abstract

Recent advances in deep learning have shown excellent performance in various scene understanding tasks. However, in some complex environments or under challenging conditions, it is necessary to employ multiple modalities that provide complementary information on the same scene. A variety of studies have demonstrated that deep multimodal fusion for semantic image segmentation achieves significant performance improvement. These fusion approaches take the benefits of multiple information sources and generate an optimal joint prediction automatically. This paper describes the essential background concepts of deep multimodal fusion and the relevant applications in computer vision. In particular, we provide a systematic survey of multimodal fusion methodologies, multimodal segmentation datasets, and quantitative evaluations on the benchmark datasets. Existing fusion methods are summarized according to a common taxonomy: early fusion, late fusion, and hybrid fusion. Based on their performance, we analyze the strengths and weaknesses of different fusion strategies. Current challenges and design choices are discussed, aiming to provide the reader with a comprehensive and heuristic view of deep multimodal image segmentation.

Multimodal Classification: Current Landscape, Taxonomy and Future Directions

“…While depictions of early and late fusion styles have been relatively consistent across multiple papers [9,17,37,41,75,84,119], there are still cases where other terms have been used. In [31], one network architecture described as multi-view-one-network is essentially early fusion and one-view-one-network could be considered late fusion.…”

“…In addition to the presentation of many different network architectures, it was observed that multimodal 2-D models can perform well on a 3-D task, especially since pre-trained 2-D networks were more mature than 3-D networks. [119] also performed a review of research using multimodal image data such as RGB-D for image segmentation.…”

“…The term intermediate has been used for cross-modality fusion but can also refer to fusion occurring somewhere between feature extraction and classification [16]. Terms like middle [38], joint [41], and hybrid [119] have also been used to describe this kind of fusion. Since our proposed taxonomy is based on the five processing stages, the middle fusion concept can be captured by early or late fusion style architectures.…”

A Machine Learning Model for Improving Building Detection in Informal Areas: A Case Study of Greater Cairo

“…The process of combining images from multiple sources into a single imagery is referred to as multi-source image fusion technology, where the resulted fused image would be more beneficial than any of the input imageries, and it has major importance to the photogrammetry tasks in computer vision [26][27][28][29]. A detailed review can be found in [30].…”

A Multi-Branch U-Net for Steel Surface Defect Type and Severity Segmentation

“…There are many ways of incorporating the parameters into our model [25]. More specifically, we can insert them at the earlier layers, mid-layers, or the last layers of our model.…”