A Multimodal Approach for Multiple-Relation Extraction in Videos

Liu, Zihe; Hou, Weiying; Zhang, Jiayi; Cao, Chenyu; Wu, Bin

doi:10.1007/s11042-021-11466-y

Cited by 5 publications

(1 citation statement)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The social relationship recognition accuracy on the ViSR dataset by using C3D, R3D, and I3D are 27.35%, 30.26%, and 33.11% respectively, which are comparable to the standard 3D ConvNet performance published in [47]. This ensures that the social relationship network is efficient, allowing us to extract valuable RVCs.…”

Section: Quantitative Analysismentioning

confidence: 56%

Social Relation Atmosphere Recognition with Relevant Visual Concepts

JI,

WANG,

MORI

et al. 2023

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

Social relationships (e.g., couples, opponents) are the foundational part of society. Social relation atmosphere describes the overall interaction environment between social relationships. Discovering social relation atmosphere can help machines better comprehend human behaviors and improve the performance of social intelligent applications. Most existing research mainly focuses on investigating social relationships, while ignoring the social relation atmosphere. Due to the complexity of the expressions in video data and the uncertainty of the social relation atmosphere, it is even difficult to define and evaluate. In this paper, we innovatively analyze the social relation atmosphere in video data. We introduce a Relevant Visual Concept (RVC) from the social relationship recognition task to facilitate social relation atmosphere recognition, because social relationships contain useful information about human interactions and surrounding environments, which are crucial clues for social relation atmosphere recognition. Our approach consists of two main steps: (1) we first generate a group of visual concepts that preserve the inherent social relationship information by utilizing a 3D explanation module; (2) the extracted relevant visual concepts are used to supplement the social relation atmosphere recognition. In addition, we present a new dataset based on the existing Video Social Relation Dataset. Each video is annotated with four kinds of social relation atmosphere attributes and one social relationship. We evaluate the proposed method on our dataset. Experiments with various 3D ConvNets and fusion methods demonstrate that the proposed method can effectively improve recognition accuracy compared to end-toend ConvNets. The visualization results also indicate that essential information in social relationships can be discovered and used to enhance social relation atmosphere recognition.

show abstract

Section: Quantitative Analysismentioning

confidence: 56%