Activity Recognition with Moving Cameras and Few Training Examples: Applications for Detection of Autism-Related Headbanging

Abstract

Activity recognition computer vision algorithms can be used to detect the presence of autism-related behaviors, including what are termed "restricted and repetitive behaviors", or stimming, by diagnostic instruments. Examples of stimming include hand fapping, spinning, and head banging. One of the most signifcant bottlenecks for implementing such classifers is the lack of sufciently large training sets of human behavior specifc to pediatric developmental delays. The data that do exist are usually recorded with a handheld camera which is itself shaky or even moving, posing a challenge for traditional feature representation approaches for activity detection which capture the camera's motion as a feature. To address these issues, we frst document the advantages and limitations of current feature representation techniques for activity recognition when applied to head banging detection. We then propose a feature representation consisting exclusively of head pose keypoints. We create a computer vision classifer for detecting head banging in home videos using a time-distributed convolutional neural network (CNN) in which a single CNN extracts features from each frame in the input sequence, and these extracted features are fed as input to a long short-term memory (LSTM) network. On the binary task of predicting head banging and no head banging within videos from the Self Stimulatory Behaviour Dataset (SSBD), we reach a mean F1-score of 90.77% using 3-fold cross validation (with individual fold F1-scores of 83.3%, 89.0%, and 100.0%) when ensuring that no child who appeared in the train set was in the test set for all folds. This work documents a successful process for training a computer vision classifer which can detect a particular human motion pattern with few training examples and even when the camera recording the source clip is unstable. The process of engineering useful feature This work is licensed under a Creative Commons Attribution International 4.0 License.

Classifying Autism From Crowdsourced Semistructured Speech Recordings: Machine Learning Model Comparison Study

“…Previous work examined the use of crowdsourced annotations for autism, indicating that similar approaches could perhaps be applied through audio [31,[46][47][48][49][50][51]. Audio feature extraction combined with other autism classifiers could be used to create an explainable diagnostic system [52][53][54][55][56][57][58][59][60][61][62][63][64] fit for mobile devices [60]. Previous work investigated using such classifiers to detect autism or approach autism-related tasks like identifying emotion to improve socialization skills; combining computer vision-based quantification of relevant areas of interest, including hand stimming [58], upper limb movement [63], and eye contact [62,64], could possibly result in interpretable models.…”

Detecting A Child’s Stimming Behaviours for Autism Spectrum Disorder Diagnosis using Rgbpose-Slowfast Network

“…[15] and Washington et.al. [16] have modelled arm flapping and head banging actions respectively from Self-Stimulatory Behaviour Dataset (SSBD) dataset. Lakkapragada et.al.…”

“…Washington et.al. [16] used head keypoints and time distributed CNN integrated with LSTM to detect head banging.…”

A Local Community-Based Social Network for Mental Health and Well-being (Quokka): Exploratory Feasibility Study

“…Interventions such as Quokka provide a mechanism for eliciting behavior change from distributed participants. To optimize the provided interventions, direct measurement of behavior changes via machine learning [69][70][71][72][73][74][75][76] along with self-reported questionnaires can generate useful multimodal data sets. Feature selection approaches could be applied to such data streams to identify salient behavioral markers [77][78][79][80][81] of mental health, and classifiers for these could be realized via trustworthy and reliable crowdsourced labeling of the incoming data [82][83][84][85][86][87].…”