DeepFusion: Deep Ensembles for Domain Independent System Fusion

Abstract

While ensemble systems and late fusion mechanisms have proven their effectiveness by achieving state-of-the-art results in various computer vision tasks, current approaches are not exploiting the power of deep neural networks as their primary ensembling algorithm, but only as inducers, i.e., systems that are used as inputs for the primary ensembling algorithm. In this paper, we propose several deep neural network architectures as ensembling algorithms with various network configurations that use dense and attention layers, an input pre-processing algorithm, and a new type of deep neural network layer denoted the Cross-Space-Fusion layer, that further improves the overall results. Experimental validation is carried out on several data sets from various domains (emotional content classification, medical data captioning) and under various evaluation conditions (two-class regression, binary classification, and multi-label classification), proving the efficiency of DeepFusion.

Overview of the ImageCLEF 2022: Multimedia Retrieval in Medical, Social Media and Nature Applications

“…The results are presented in Table 6 for the interestingness task, and Table 7 for the diversification task. In both tasks, the best performance is achieved with a DeepFusion type approach [9], submitted by the AIMultimediaLab team. The best performance for the ImageCLEF-int task is a MAP@10 value of 0.2192, while for the ImageCLEF-div task a F1@20 of 0.6216 and a CR@20 of 0.4916 is achieved.…”

ImageCLEF 2022: Multimedia Retrieval in Medical, Nature, Fusion, and Internet Applications

“…In the current literature late fusion systems are sometimes successfully used even in traditional tasks such as video action recognition [27], and more often in subjective and multimodal tasks like memorability [1], violence detection [7] and media interestingness [30]. Furthermore, latest developments in this field, using deep neural networks as the primary ensembling method show major improvements over traditional ensembling methods, by greatly increasing the performance of individual inducers [5,6,26].…”

Exploring Deep Fusion Ensembling for Automatic Visual Interestingness Prediction

“…While not directly attempting to model human behaviour and understanding of visual interestingness, we believe these models are able to model inducer behaviour and understanding, thus being able to learn the positive and negative biases of inducers towards visual samples. Thus, while the approaches presented here are centered around the prediction of visual interestingness, they are domain-independent and are useful in other tasks as well [31].…”

“…With regards to media interestingness, [8] represents an in-depth literature review of interestingness and covariate concepts, analyzing these concepts and their correlations from psychological, user-centric and computer vision perspectives, while [19] represents a review of the MediaEval Predicting Media Interestingness task, analyzing the best practices, methods, user annotation statistics and the data itself. From an ensembling perspective, three papers introduce some of the deep neural network architectures that we will deploy in this work: [30,19,31]. The code corresponding to the proposed methods we will present is available online , developed in Python 3 using the Keras 2.2.4 and Tensorflow 1.12 libraries.…”