Towards Tokenized Human Dynamics Representation

Li, Kenneth; Xiao, Sun; Wu, Zhirong; Wei, Fangyun; Lin, Stephen

doi:10.48550/arxiv.2111.11433

Cited by 1 publication

(1 citation statement)

References 42 publications

(71 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different from these methods, our goal is fine-grained temporal understanding of videos and we treat a long sequence of frames as input data. The most relevant work to ours is [28], which utilizes 3D human keypoints for self-supervised acton discovery in long kinematic videos.…”

Section: Related Workmentioning

confidence: 99%

Frame-wise Action Representations for Long Videos via Sequence Contrastive Learning

Chen¹,

Wei²,

Li³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Prior works on action representation learning mainly focus on designing various architectures to extract the global representations for short video clips. In contrast, many practical applications such as video alignment have strong demand for learning dense representations for long videos. In this paper, we introduce a novel contrastive action representation learning (CARL) framework to learn frame-wise action representations, especially for long videos, in a selfsupervised manner. Concretely, we introduce a simple yet efficient video encoder that considers spatio-temporal context to extract frame-wise representations. Inspired by the recent progress of self-supervised learning, we present a novel sequence contrastive loss (SCL) applied on two correlated views obtained through a series of spatio-temporal data augmentations. SCL optimizes the embedding space by minimizing the KL-divergence between the sequence similarity of two augmented views and a prior Gaussian distribution of timestamp distance. Experiments on FineGym, PennAction and Pouring datasets show that our method outperforms previous state-of-the-art by a large margin for downstream fine-grained action classification. Surprisingly, although without training on paired videos, our approach also shows outstanding performance on video alignment and fine-grained frame retrieval tasks. Code and models are available at https://github.com/ minghchen/CARL_code.

show abstract

Section: Related Workmentioning

confidence: 99%