Controlling Video Stimuli in Sign Language and Gesture Research: The OpenPoseR Package for Analyzing OpenPose Motion-Tracking Data in R

Abstract

Researchers in the fields of sign language and gesture studies frequently present their participants with video stimuli showing actors performing linguistic signs or co-speech gestures. Up to now, such video stimuli have been mostly controlled only for some of the technical aspects of the video material (e.g., duration of clips, encoding, framerate, etc.), leaving open the possibility that systematic differences in video stimulus materials may be concealed in the actual motion properties of the actor’s movements. Computer vision methods such as OpenPose enable the fitting of body-pose models to the consecutive frames of a video clip and thereby make it possible to recover the movements performed by the actor in a particular video clip without the use of a point-based or markerless motion-tracking system during recording. The OpenPoseR package provides a straightforward and reproducible way of working with these body-pose model data extracted from video clips using OpenPose, allowing researchers in the fields of sign language and gesture studies to quantify the amount of motion (velocity and acceleration) pertaining only to the movements performed by the actor in a video clip. These quantitative measures can be used for controlling differences in the movements of an actor in stimulus video clips or, for example, between different conditions of an experiment. In addition, the package also provides a set of functions for generating plots for data visualization, as well as an easy-to-use way of automatically extracting metadata (e.g., duration, framerate, etc.) from large sets of video files.

A Systematic Investigation of Gesture Kinematics in Evolving Manual Languages in the Lab

“…Based on signal processing alone, we have detected systematic changes reflective of a linguistically maturing communication system, from continuous multi‐articulatory kinematics of silent gestures. We applied computer vision techniques to extract kinematics from video data (e.g., Östling, Börstell, & Courtaux, 2018 ; Ripperda et al., 2020 ; Trettenbrein & Zaccarella, 2021 ). We then quantified kinematic relationships between gestural utterances (Pouw & Dixon, 2019 ).…”

A Roadmap for Technological Innovation in Multimodal Communication Research

“…Depending on the system and model, the general pose of the actor can be determined based on predefined key points such as the head, elbows, shoulders, or feet, but also the position and posture of the hands and fingers, facial movements, and gaze directions. These data can then be used to quantify a number of parameters such as the amount of motion of an actor or a particular body part [186]. In animal research, tools such as DeepLabCut [112] or SLEAP, revolutionized the ease with which researchers can track morphologically unique body poses in a wide range of animals.…”

Research on Methods of Physical Aided Education Based on Deep Learning

“…erefore, taking the modified human posture model as reference, this paper created a new deep neural network model to estimate the two-dimensional coordinates of human skeleton points of badminton players in a single frame image. Its architecture is shown in Figure 2 [13].…”