A Perceptual Prediction Framework for Self Supervised Event Segmentation

Aakur, Sathyanarayanan N.; Sarkar, Sudeep

doi:10.1109/cvpr.2019.00129

Cited by 55 publications

(21 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, these narration or subtitles may be inaccurate [82] or even irrelevant to the video as we mention above. For the action segmentation task, Aakur et al [5] presented a self-supervised and predictive learning framework to explore the spatial-temporal dynamics of the videos, while Sener et al [58] proposed a Generalized Mallows Model (GMM) to model the distribution over sub-activity permutations. More recently, Kukleva et al [37] first learned a continuous temporal embedding of frame-based features, and then decoded the videos into coherent action segments according to an ordered clustering of these features.…”

Section: Methods For Instructional Video Analysismentioning

confidence: 99%

See 1 more Smart Citation

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Tang

Lu²,

Zhou³

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Thanks to the substantial and explosively inscreased instructional videos on the Internet, novices are able to acquire knowledge for completing various tasks. Over the past decade, growing efforts have been devoted to investigating the problem on instructional video analysis. However, the most existing datasets in this area have limitations in diversity and scale, which makes them far from many real-world applications where more diverse activities occur. To address this, we present a large-scale dataset named as "COIN" for COmprehensive INstructional video analysis. Organized with a hierarchical structure, the COIN dataset contains 11,827 videos of 180 tasks in 12 domains (e.g., vehicles, gadgets, etc.) related to our daily life. With a new developed toolbox, all the videos are annotated efficiently with a series of step labels and the corresponding temporal boundaries. In order to provide a benchmark for instructional video analysis, we evaluate plenty of approaches on the COIN dataset under five different settings. Furthermore, we exploit two important characteristics (i.e., task-consistency and ordering-dependency) for localizing important steps in instructional videos. Accordingly, we propose two simple yet effective methods, which can be easily plugged into conventional proposal-based action detection models. We believe the introduction of the COIN dataset will promote the future in-depth research on instructional video analysis for the community. Our dataset, annotation toolbox and source code are available at http://coin-dataset.github.io. DomainTaskStep VehiclesHousehold Items Change the Car Tire {unscrew the screws, jack up the car, remove the tire, put on the tire, tighten the screws } {remove the door knob, remove bolt and pin board, install new pin board, install new bolt, install new door knob } Replace the Door Knob

show abstract

Section: Methods For Instructional Video Analysismentioning

confidence: 99%

“…(2) SSN [80]. This is an effective model for action detection, which outputs the same type of results (interval and label for each action 5. We present a table to clarify the goal, metric, and evaluated methods for each task in supplementary material.…”

Section: Evaluation On Step Localizationmentioning

confidence: 99%

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Tang

Lu²,

Zhou³

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Unsupervised learning-based approaches recently received attention [15], [16], [17], [35]. One line of work targets keyframe localization in videos [19], [20].…”

Section: Related Workmentioning

confidence: 99%

“…First, video-level matching [35], [23] only matches the labeled actions with respect to the ground truth actions of that given video; this granularity of matching produces very high performance since it somehow simplifies this task in the sense that it is reluctant to associate actions in any pair of videos even within the same complex activity.…”

Section: Hungarian Matching Hierarchiesmentioning

confidence: 99%

Temporal Action Segmentation with High-level Complex Activity Labels

Ding

Yao

2021

Preprint

View full text Add to dashboard Cite

Over the past few years, the success in action recognition on short trimmed videos has led more investigations towards the temporal segmentation of actions in untrimmed long videos. Recently, supervised approaches have achieved excellent performance in segmenting complex human actions in untrimmed videos. However, besides action labels, such approaches also require the start and end points of each action, which is expensive and tedious to collect.In this paper, we aim to learn the action segments taking only the high-level activity labels as input. Under the setting where no action-level supervision is provided, Hungarian matching is often used to find the mapping between segments and ground truth actions to evaluate the model and report the performance. On the one hand, we show that with the high-level supervision, we are able to generalize the Hungarian matching settings from the current video and activity level to the global level. The extended global-level matching allows for the shared actions across activities. On the other hand, we propose a novel action discovery framework that automatically discovers constituent actions in videos with the activity classification task. Specifically, we define a finite number of prototypes to form a dual representation of a video sequence. These collectively learned prototypes are considered discovered actions. This classification setting endows our approach the capability of discovering potentially shared actions across multiple complex activities. Extensive experiments demonstrate that the discovered actions are helpful in performing temporal action segmentation and activity recognition.

show abstract

“…Previous work mainly divides the problem of video object detection on key frames into sub-problems of key frame recognition [13]- [17] and object detection. They propose techniques such as self-supervised learning [18], semi-supervised learning [19], label propagation [20], registration [21], and temporal cycle-consistency [22]. However, the inherent predictive uncertainties in video landmark measurements are generally ignored in the existing literature.…”

Section: Introductionmentioning

confidence: 99%

U-LanD: Uncertainty-Driven Video Landmark Detection

Jafari¹,

Luong²,

Tsang³

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper presents U-LanD, a framework for joint detection of key frames and landmarks in videos. We tackle a specifically challenging problem, where training labels are noisy and highly sparse. U-LanD builds upon a pivotal observation: a deep Bayesian landmark detector solely trained on key video frames, has significantly lower predictive uncertainty on those frames vs. other frames in videos. We use this observation as an unsupervised signal to automatically recognize key frames on which we detect landmarks. As a test-bed for our framework, we use ultrasound imaging videos of the heart, where sparse and noisy clinical labels are only available for a single frame in each video. Using data from 4,493 patients, we demonstrate that U-LanD can exceedingly outperform the state-of-the-art non-Bayesian counterpart by a noticeable absolute margin of 42% in R 2 score, with almost no overhead imposed on the model size. Our approach is generic and can be potentially applied to other challenging data with noisy and sparse training labels.

show abstract

A Perceptual Prediction Framework for Self Supervised Event Segmentation

Cited by 55 publications

References 39 publications

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Temporal Action Segmentation with High-level Complex Activity Labels

U-LanD: Uncertainty-Driven Video Landmark Detection

Contact Info

Product

Resources

About