An Automated Mutual Gaze Detection Framework for Social Behavior Assessment in Therapy for Children with Autism

Zhang, Guo; Kim, Kangsoo; Bhat, Anjana; Barmaki, Roghayeh

doi:10.1145/3462244.3479882

Cited by 11 publications

(2 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In future work, we will integrate graph neural networks to generate spatial skeletal representations [10]. We will also investigate other privacy-preserving data modalities, such as gaze data [57], facial landmarks, and emotion clues [58].…”

Section: Discussionmentioning

confidence: 99%

Dyadic Movement Synchrony Estimation Under Privacy-preserving Conditions

Li¹,

Bhat²,

Barmaki³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Movement synchrony refers to the dynamic temporal connection between the motions of interacting people. The applications of movement synchrony are wide and broad. For example, as a measure of coordination between teammates, synchrony scores are often reported in sports. The autism community also identifies movement synchrony as a key indicator of children's social and developmental achievements. In general, raw video recordings are often used for movement synchrony estimation, with the drawback that they may reveal people's identities. Furthermore, such privacy concern also hinders data sharing, one major roadblock to a fair comparison between different approaches in autism research. To address the issue, this paper proposes an ensemble method for movement synchrony estimation, one of the first deep-learning-based methods for automatic movement synchrony assessment under privacypreserving conditions. Our method relies entirely on publicly shareable, identity-agnostic secondary data, such as skeleton data and optical flow. We validate our method on two datasets: (1) PT13 dataset collected from autism therapy interventions and (2) TASD-2 dataset collected from synchronized diving competitions. In this context, our method outperforms its counterpart approaches, both deep neural networks and alternatives.

show abstract

Section: Discussionmentioning

confidence: 99%

Dyadic Movement Synchrony Estimation Under Privacy-preserving Conditions

Li¹,

Bhat²,

Barmaki³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In future work, we plan to consider the frequency and difficulty of an action in synchrony estimation, which we did not emphasize adequately in this paper. We will also investigate other privacy-preserving data modalities, such as gaze data [21,22], facial landmarks, and emotion [36]. Other group-based interaction evaluation approaches such as group activity recognition [19] are also worth investigating.…”

Section: Discussionmentioning

confidence: 99%

Pose Uncertainty Aware Movement Synchrony Estimation via Spatial-Temporal Graph Transformer

Li¹,

Bhat²,

Barmaki³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Movement synchrony reflects the coordination of body movements between interacting dyads. The estimation of movement synchrony has been automated by powerful deep learning models such as transformer networks. However, instead of designing a specialized network for movement synchrony estimation, previous transformer-based works broadly adopted architectures from other tasks such as human activity recognition. Therefore, this paper proposed a skeleton-based graph transformer for movement synchrony estimation.The proposed model applied ST-GCN, a spatial-temporal graph convolutional neural network for skeleton feature extraction, followed by a spatial transformer for spatial feature generation. The spatial transformer is guided by a uniquely designed joint position embedding shared between the same joints of interacting individuals. Besides, we incorporated a temporal similarity matrix in temporal attention computation considering the periodic intrinsic of body movements. In addition, the confidence score associated with each joint reflects the uncertainty of a pose, while previous works on movement synchrony estimation have not sufficiently emphasized this point. We introduced a pose uncertainty awareness mechanism in spatial and temporal feature generations. Since transformer networks demand a significant amount of data to train, we constructed a dataset for movement synchrony estimation using Human3.6M, a benchmark dataset for human activity recognition, and pretrained our model on it using contrastive learning. We further applied knowledge distillation to alleviate information loss introduced by pose detector failure in a privacy-preserving way.We compared our method with representative approaches on PT13, a dataset collected from autism therapy interventions. Our method achieved an overall accuracy of 88.98% and surpassed its counterparts by a wide margin while maintaining data privacy. This work has implications on synchronous movement activity recognition in group settings, with applications in education and sports. CCS Concepts: • Social and professional topics → People with disabilities; • Human-centered computing; • Computing methodologies → Activity recognition and understanding;

show abstract

A Computational Approach for Analysing Autistic Behaviour During Dyadic Interactions

Çeliktutan

Vogeley

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

An Automated Mutual Gaze Detection Framework for Social Behavior Assessment in Therapy for Children with Autism

Cited by 11 publications

References 46 publications

Dyadic Movement Synchrony Estimation Under Privacy-preserving Conditions

Dyadic Movement Synchrony Estimation Under Privacy-preserving Conditions

Pose Uncertainty Aware Movement Synchrony Estimation via Spatial-Temporal Graph Transformer

A Computational Approach for Analysing Autistic Behaviour During Dyadic Interactions

Contact Info

Product

Resources

About