Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects

Abstract

This paper provides an overview of the main challenges in multimodal data fusion across various disciplines and addresses two key issues: ''why we need data fusion'' and ''how we perform it.'' ABSTRACT | In various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or subjects, among others. We use the term ''modality'' for each such acquisition framework. Due to the rich characteristics of natural phenomena, it is rare that a single modality provides complete knowledge of the phenomenon of interest. The increasing availability of several modalities reporting on the same system introduces new degrees of freedom, which raise questions beyond those related to exploiting each modality separately. As we argue, many of these questions, or ''challenges,'' are common to multiple domains. This paper deals with two key issues: ''why we need data fusion'' and ''how we perform it.''The first issue is motivated by numerous examples in science and technology, followed by a mathematical framework that showcases some of the benefits that data fusion provides. In order to address the second issue, ''diversity'' is introduced as a key concept, and a number of data-driven solutions based on matrix and tensor decompositions are discussed, emphasizing how they account for diversity across the data sets. The aim of this paper is to provide the reader, regardless of his or her community of origin, with a taste of the vastness of the field, the prospects, and the opportunities that it holds.

Anomalous behaviour detection based on heterogeneous data and data fusion

“…Data fusion techniques have been widely applied in multisensor environments (Khaleghi et al 2013). The motivations of using data fusion are various (Lahat et al 2015). One of them is to improve the decision-making.…”

“…One of them is to improve the decision-making. Data fusion is a challenging task (Lahat et al 2015). One of the challenges is working with heterogeneous datasets when advantages of each dataset are exploited maximally and the disadvantages suppressed.…”

“…Data imperfection, data association, data correlation, and data dimensionality are challenging factors when applying data fusion (Khaleghi et al 2013). However, the most challenging part of data fusion is working with heterogeneous data (Lahat et al 2015). Heterogeneous data will have multiple features.…”

Cognitive computing for coalition situational understanding

“…[10] These views of SU are intrinsically linked to information fusion in that they involve the collection and processing of data from multiple environmental sources as input to deriving SA and ultimately SU; the data sources feed into the left of Figure 1 and the bottom of Figure 2. Moreover, the sources will commonly span multiple modalities (for example, imagery, acoustic and textual data [11]) requiring NLP and VSP in addition to ML.…”

Addressing Complexities of Machine Learning in Big Data: Principles, Trends and Challenges from Systematical Perspectives

“…Recalling the 5 V characteristics, the volume of data and the velocity of processing techniques have posed great challenges for data storage and processing; consequently, distributed systems have been developed as a plausible framework to address these challenges [57,59]. Regarding the variety of big data, it is best matched through multi-modalities, and its inherent heterogeneity makes it easier to gain a global view of phenomena [60]. The property of multiple sources or cross-domains has contributed to the boom in data fusion techniques [60,61].…”