Counting Out Time: Class Agnostic Video Repetition Counting in the Wild

Dwibedi, Debidatta; Aytar, Yusuf; Tompson, Jonathan; Sermanet, Pierre; Zisserman, Andrew

doi:10.1109/cvpr42600.2020.01040

Cited by 84 publications

(125 citation statements)

References 50 publications

(46 reference statements)

Supporting

Mentioning

125

Contrasting

Order By: Relevance

“…For the n-tuplet and synchronization loss to function correctly, it is required that each cine in our training set contains approximately one cardiac cycle. To enforce the one cycle constraint, we either used the provided average heart rate and frame rate data to approximate a single period, or in cases where this information was not available, we applied the class agnostic video repetition counting method (RepNet) proposed in [15]. In training studies with more than one cycle, all available cycles are considered as individual training samples.…”

Section: A Datasetmentioning

confidence: 99%

Echo-SyncNet: Self-Supervised Cardiac View Synchronization in Echocardiography

Dezaki

Luong

Ginsberg

et al. 2021

IEEE Trans. Med. Imaging

View full text Add to dashboard Cite

In echocardiography (echo), an electrocardiogram (ECG) is conventionally used to temporally align different cardiac views for assessing critical measurements. However, in emergencies or point-of-care situations, acquiring an ECG is often not an option, hence motivating the need for alternative temporal synchronization methods. Here, we propose Echo-SyncNet, a self-supervised learning framework to synchronize various cross-sectional 2D echo series without any human supervision or external inputs. The proposed framework takes advantage of two types of supervisory signals derived from the input data: spatiotemporal patterns found between the frames of a single cine (intra-view self-supervision) and interdependencies between multiple cines (inter-view self-supervision). The combined supervisory signals are used to learn a featurerich and low dimensional embedding space where multiple echo cines can be temporally synchronized. Two intra-view self-supervisions are used, the first is based on the information encoded by the temporal ordering of a cine (temporal intra-view) and the second on the spatial similarities between nearby frames (spatial intra-view). The inter-view self-supervision is used to promote the learning of similar embeddings for frames captured from the same cardiac phase in different echo views. We evaluate the framework with multiple experiments: 1) Using data from 998 patients, Echo-SyncNet shows promising results for synchronizing Manuscript received ***; accepted ***. Date of publication ***; date of current version ***.

show abstract

Section: A Datasetmentioning

confidence: 99%

Echo-SyncNet: Self-Supervised Cardiac View Synchronization in Echocardiography

Dezaki

Luong

Ginsberg

et al. 2021

IEEE Trans. Med. Imaging

View full text Add to dashboard Cite

show abstract

“…It is reliable in the unsupervised situation when the semantic content of the videos, the number of periods and the valid duration of the video are unknown. Debidatta et al [18] introduced a new symmetric matrix that helps achieve up-to-date accuracy. It acts as an intermediate layer to predict the cycle length and valid periodic length.…”

Section: Repetitive Countingmentioning

confidence: 99%

“…Therefore, we synthesize the Rep-Penn dataset by utilizing the single-cycle exercise video frames. Inspired by dataset synthesis methods from [18], we connect the forward video with the rewinding video instead of connecting the original video repeatedly. In this way, continuity of the body movement is guaranteed, and exercise of various periods can be generated for repetition counting.…”

Section: Rep-penn Benchmarkmentioning

confidence: 99%

See 1 more Smart Citation

Deep Learning-Enabled Multitask System for Exercise Recognition and Counting

Wang

Laamarti

et al. 2021

MTI

View full text Add to dashboard Cite

Exercise is a prevailing topic in modern society as more people are pursuing a healthy lifestyle. Physical activities provide significant benefits to human well-being from the inside out. Human pose estimation, action recognition and repetitive counting fields developed rapidly in the past several years. However, few works combined them together to assist people in exercise. In this paper, we propose a multitask system covering the three domains. Different from existing methods, heatmaps, which are the byproducts of 2D human pose estimation models, are adopted for exercise recognition and counting. Recent heatmap processing methods have been proven effective in extracting dynamic body pose information. Inspired by this, we propose a deep-learning multitask model of exercise recognition and repetition counting. To the best of our knowledge, this approach is attempted for the first time. To meet the needs of the multitask model, we create a new dataset Rep-Penn with action, counting and speed labels. Our multitask system can estimate human pose, identify physical activities and count repeated motions. We achieved 95.69% accuracy in exercise recognition on the Rep-Penn dataset. The multitask model also performed well in repetitive counting with 0.004 Mean Average Error (MAE) and 0.997 Off-By-One (OBO) accuracy on the Rep-Penn dataset. Compared with existing frameworks, our method obtained state-of-the-art results.

show abstract

“…Still, counting the repetition of exercise is a less studied field compared with the prosperous-growing field in human activity recognition. Dwibedi et al [10] pointed out that it is strenuous to search for suitable video from large-scale public datasets, as there is no specific keyword and annotations catering to this particular needs. The root cause of this phenomenon is that annotating video or signal across the temporal domain is labor-intensive and monotonous work.…”

Section: Introductionmentioning

confidence: 99%

Viewpoint-Invariant Exercise Repetition Counting

Hsu

Efstratios

Tsui

et al. 2021

Preprint

View full text Add to dashboard Cite

Background Counting the repetition of human exercise and physical rehabilitation is a common task in rehabilitation and exercise training. The existing vision-based repetition counting methods less emphasize the concurrent motions in the same video. Methods This work analyzed the spectrogram of the pose estimation result to count the repetition. Besides from the public datasets. This work also collected exercise videos from 11 adults to verify the proposed method is capable for handling concurrent motion and different view angles. Results The presented method was validated on the University of Idaho Physical Rehabilitation Movements Data Set (UI-PRMD) and MM-fit dataset. The overall mean absolute error (MAE) for MM-fit was 0.06 with off-by-one Accuracy (OBOA) 0.94. As for UI-PRMD dataset, MAE was 0.06 with OBOA 0.95. We have also tested the performance in a variety of camera locations and concurrent motions with 57 skeleton time-series video with overall MAE 0.07 and OBOA 0.91. Conclusion The proposed method provides a view-angle and motion agnostic concurrent motion counting. This method can potentially use in large-scale remote rehabilitation and exercise training with only one camera.

show abstract

Counting Out Time: Class Agnostic Video Repetition Counting in the Wild

Cited by 84 publications

References 50 publications

Echo-SyncNet: Self-Supervised Cardiac View Synchronization in Echocardiography

Echo-SyncNet: Self-Supervised Cardiac View Synchronization in Echocardiography

Deep Learning-Enabled Multitask System for Exercise Recognition and Counting

Viewpoint-Invariant Exercise Repetition Counting

Contact Info

Product

Resources

About