Attach-SwiNet: Multimodal Attachment Style Classification Model Based on Non-Verbal Signals

Abstract

Attachment systems, representing emotional bonds with significant others, have emerged as an important aspect of psychology that influences self-development and social interactions. Adult attachment studies in psychology have mostly relied on questionnaires and interviews, primarily centered on romantic relationships and parent-child dynamics during childhood. In machine learning-based adult attachment studies, the assessment of objective aspects of non-verbal behaviors is also investigated, but how these behavioral characteristics are associated with attachment remains unexplored. This paper introduces a new multimodal model for attachment style classification, focusing on close relationships established within young adults. Our proposed model integrates representations of emotions derived from non-verbal behaviors alongside subjective responses to classify attachment styles. Leveraging pre-trained Swin Transformers for capturing emotion associations in facial expression videos and pre-trained ResNet50 for analyzing speech responses, we fuse the best emotion representations from both datasets with rating data from the Experiences in Close Relationships -Relationship Structures (ECR-RS) for attachment style classification. The experimental results demonstrate that the proposed model enhances the performance of unimodal behavioral data and subjective questionnaire responses by 1.13%.