RAIF: A deep learning‐based architecture for multi‐modal aesthetic biometric system

Abstract

Human aesthetics play a significant role in video game development, emotional‐aware robot design, online recommender systems, digital human, and other domains of research focusing on human‐computer interactions. Social network user recognition based on aesthetic preferences is an emerging research domain. In this paper, a novel deep learning architecture is proposed for multi‐modal audio‐visual person identification that combines audio and visual aesthetic features. A pre‐trained ResNet architecture is utilized to extract high‐level features from a set of user‐preferred audio and image samples. A novel deep learning‐based fusion technique called residual‐aided intermediate fusion (RAIF) is introduced in order to effectively merge the audio and visual features. The proposed RAIF method achieved an accuracy of 98% and a loss of 0.01 on a proprietary multi‐modal dataset, indicating its effectiveness in fusing audio and visual information.

Adaptive information fusion network for multi‐modal personality recognition

“…Liao et al 5 established a reproducible audiovisual benchmark framework, ensuring a fair and consistent evaluation of existing models. Iffath et al 17 proposed a multi-modal audio-visual person recognition deep learning architecture that combines audio-visual features to effectively fuse audio-visual features. Serrano et al 18 proposed a stacked ensemble model, leveraging diverse deep learning algorithms and word embedding techniques to formulate personality recognition models.…”