Multimodal Classification: Current Landscape, Taxonomy and Future Directions

Abstract

Multimodal classification research has been gaining popularity in many domains that collect more data from multiple sources including satellite imagery, biometrics, and medicine. However, the lack of consistent terminology and architectural descriptions makes it difficult to compare different existing solutions. We address these challenges by proposing a new taxonomy for describing such systems based on trends found in recent publications on multimodal classification. Many of the most difficult aspects of unimodal classificationhave not yet been fully addressed for multimodal datasets including big data, class imbalance, and instance level difficulty. We also provide a discussion of these challenges and future directions.

Improved Kidney Stone Recognition Through Attention and Multi-View Feature Fusion Strategies

“…Multi-View (MV) classification seeks to combine characteristics from different sources (here the image types). The accuracy of object identification increases due to the diversity of the features extracted from different sources and that are fused [17], [21]. Thus, the performance of a DL-model can be improved by optimizing multiple functions, one per each image type.…”

“…Finally, the baseline model and the modified version with attention were used to train the MV fusion models without, and with attention, respectively. To fuse the features, two late-fusion strategies are explored: feature concatenation, and max-pooling of the individual features obtained by each view response [16], [17].…”

“…This contribution takes inspiration from recent works in multi-view fusion strategies [15]- [17], which seek to combine characteristics from different sources or modalities to further improve machine learning based classification models. The aim of combining/fusing the features extracted from surface and section images is to increase the amount of discriminant information to improve the accuracy of the classification.…”