Early vs Late Fusion in Multimodal Convolutional Neural Networks

Lee

et al. 2022

Preprint

Anthropometric profiles are important indices for the assessment of medical conditions, including malnutrition, obesity, andgrowth disorders. Noncontact methods for estimating those parameters could have considerable value in many practicalsituations, such as the assessment of young, uncooperative infants or children and the prevention of infectious diseasetransmission. The purpose of this study is to investigate the feasibility of obtaining noncontact anthropometric measurementsusing the impulse-radio ultrawideband (IR-UWB) radar sensor technique. A total of 45 healthy adults were enrolled, and aconvolutional neural network (CNN) algorithm was implemented to analyze data extracted from IR-UWB radar. The differences(root-mean-square error, RMSE) between values from the radar and bioelectrical impedance analysis (BIA) as a reference inthe measurement of height, weight, and body mass index (BMI) were 2.78, 5.31, and 2.25, respectively; predicted data fromthe radar highly agreed with those from the BIA. The intraclass correlation coefficients (ICCs) were 0.93, 0.94, and 0.83.In conclusion, IR-UWB radar can provide accurate estimates of anthropometric parameters in a noncontact manner; this studyis thus the first supporting the radar sensor as an applicable method in clinical situations.

Section: Height Estimation Using Radar Signal Processingmentioning

confidence: 99%

Preclinical Trial of Noncontact Anthropometric Measurement Using IR-UWB Radar

Lee

et al. 2022

Preprint

“…The co-existence of diverse modes of input information naturally raises the question of their combination. That is, a framework facilitating the collaboration of the multiple input representations is needed in order to profit from every type of information and provide an enhanced final prediction [19]. Two are the state-of-the-art techniques that can help towards this direction: a) model ensembling through stacked generalization and b) feature concatenation inside a CNN.…”

Section: Modality Fusion Techniquesmentioning

confidence: 99%

“…Feature concatenation [18,19] on the other hand can be performed inside a CNN: that is, two distinct branches, respectively extract features from the tabular data and the images. These features are then concatenated and fed into the CNN's regressor part, which performs the predictions.…”

Section: Modality Fusion Techniquesmentioning

confidence: 99%

“…A final remark is that the earlier the fusion happens, the better the results are. That is, an early integration of the available sources of information leads to the most efficient prediction model [19]. In order to evaluate the contribution of the individual channels to the prediction performance, we have examined the cases where each image was fed to the CNN in the absence of the other two, as well as the cases where two images formed the CNN's input: more specifically, we have converted each of the three grayscale images into colored ones, so as to either fill all three channels from three copies of the same image, or to use one of them for two of the channels and fill the third channel with a second image.…”

Section: Other Bimodal Approachesmentioning

confidence: 99%

See 1 more Smart Citation

Fusing Diverse Input Modalities for Path Loss Prediction: A Deep Learning Approach

et al. 2021

Tabular data and images have been used from machine learning models as two diverse types of inputs, in order to perform path loss predictions in urban areas. Different types of models are applied on these distinct modes of input information. The work at hand tries to incorporate both modes of input data within a single prediction model. It therefore manipulates and transforms the vectors of tabular data into images. Each feature of the tabular data vector is spread into several pixels, corresponding to the calculated importance of the particular feature. The resulting synthetic images are then fused with images representing selected regions of the area's map. Compound pseudoimages, having channels of both map-based and tabular data-based images, are then being used as inputs for a Convolutional Neural Network (CNN), which predicts the path loss value at a specific point of the area of interest. The results are clearly better than those obtained from models based on a single mode of input data, as well as from the results produced by other bimodal-input approaches. This approach could be applied for path loss prediction in urban environments for several state-of-art wireless networks like 5G and Internet of Things (IoT).

“…We combine the two types of features by concatenating the vectors representing each aspect in a single vector, used as input to a supervised classification algorithm. According to [Gadzicki et al 2020], this approach of combining multi-modal features is referred to as early-maturing fusion.…”

Section: Feature Extraction From News and Diffusion Network For Fake News Classificationmentioning

confidence: 99%

Assessing the combination of DistilBERT news representations and difusion topological features to classify fake news

Sáenz

Dias

Becker

2021

jidm

Fake news (FN) have affected people’s lives in unimaginable ways. The automatic classification of FN is a vital tool to prevent their dissemination and support fact-checking. Related work has shown that FN spread faster, deeper, and more broadly than truthful news on social media. Deep learning has produced state-of-the-art solutions in this field, mainly based on textual attributes. In this paper, we propose to combine compact representations of the textual news properties generated using DistilBERT, with topological metrics extracted from their propagation network in social media. Using a dataset related to politics and distinct learning algorithms, we extensively assessed the components of the proposed solution. Regarding the textual attributes, we reached results comparable to stateof-the-art solutions using only the news title and contents, which is useful for FN early detection. We assessed the influential topological metrics, and the effect of their combination with the news textual features. We also explored the use of ensembles. Our results were very promising, revealing the potential of the features proposed and the adoption of ensembles.