3D Social Research: Analysis of Social Interaction Using Computer Vision

Abstract

Video data offer important insights into social processes because they enable direct observation of real-life social interaction. Though such data have become abundant and increasingly accessible, they pose challenges to scalability and measurement. Computer vision (CV), i.e., software-based automated analysis of visual material, can help address these challenges, but existing CV tools are not sufficiently tailored to analyze social interactions. We describe our novel approach, “3D social research” (3DSR), which uses CV and 3D camera footage to study kinesics and proxemics, two core elements of social interaction. Using eight videos of a scripted interaction and five real-life street scene videos, we demonstrate how 3DSR expands sociologists’ analytical toolkit by facilitating a range of scalable and precise measurements. We specifically emphasize 3DSR's potential for analyzing physical distance, movement in space, and movement rate – important aspects of kinesics and proxemics in interactions. We also assess data reliability when using 3DSR.

Machine Learning and the Analysis of Culture

“…Research has started to use computer vision for video data analysis to examine movements and interactions using existing video footage (Bernasco et al, 2023) as well as applying additional body keypoint software to images, thus improving the capture of individual bodies across frames (Goldstein, Legewie, & Shiffer-Sebba, 2023). Several software packages are currently available for converting videos into usable, analyzable metrics (Nassauer & Legewie, 2021) which can accurately capture people's walking, gesturing, or interaction patterns, as well as detect individual or group emotions.…”