Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection

Dai, Rui; Das, Srijan; Sharma, Saurav; Minciullo, Luca; Garattoni, Lorenzo; Brémond, François; Francesca, Gianpiero

doi:10.48550/arxiv.2010.14982

Cited by 3 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Action detection has received a lot of interest in recent years Dai et al (2021a); Huang et al (2020); Dai et al (2019a); Piergiovanni & Ryoo (2018). In this work, we focus on densely labelled action detection for handling videos with additional temporal relationships between different action classes Yeung et al (2018);Sigurdsson et al (2016); Dai et al (2020). Different from the sparsely labelled detection methods which output a sparse set of action snippets Caba Heilbron et al ( 2015 2019) frameworks apply temporal filters over snippet-wise features with a snippet-level classification, therefore, it interprets the sequence of images to a sequence of predictions.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

CTRN: Class-Temporal Relational Network for Action Detection

Dai¹,

Das²,

Brémond³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Action detection is an essential and challenging task, especially for densely labelled datasets of untrimmed videos. There are many real-world challenges in those datasets, such as composite action, co-occurring action, and high temporal variation of instance duration. For handling these challenges, we propose to explore both the class and temporal relations of detected actions. In this work, we introduce an end-to-end network: Class-Temporal Relational Network (CTRN). It contains three key components: (1) The Representation Transform Module filters the class-specific features from the mixed representations to build a graph structured data. (2) The Class-Temporal Module models the class and temporal relations in a sequential manner. (3) G-classifier leverages the privileged knowledge of the snippet-wise co-occurring action pairs to further improve the co-occurring action detection. We evaluate CTRN on three challenging densely labelled datasets and achieve state-of-the-art performance, reflecting the effectiveness and robustness of our method.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Datasets. We evaluate our method on three densely labelled action detection datasets: Charades Sigurdsson et al ( 2017), TSU Dai et al (2020) and MultiTHUMOS Yeung et al (2018). These datasets contain videos of different types: (1) sports and daily living videos, (2) short and long videos.…”

Section: Experimental Settingsmentioning

confidence: 99%

CTRN: Class-Temporal Relational Network for Action Detection

Dai¹,

Das²,

Brémond³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Action detection has received a lot of interest in recent years [10,12,18,26,44,46]. In this work, we focus on action detection in densely labelled videos [8,35,45]. The early attempts on modeling complex temporal relations are to use anchor-based methods [5,43], although dense action distribution requires large amount of anchors.…”

Section: Related Workmentioning

confidence: 99%

“…The input to the MS-TCT action detection network is an untrimmed video that can last for a very long duration [8]. Processing long videos in both spatial and temporal dimensions challenge the current computation resources.…”

Section: Visual Encodingmentioning

confidence: 99%

See 1 more Smart Citation

MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection

Dai,

Das,

Kahatapitiya

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Action detection is an essential and challenging task, especially for densely labelled datasets of untrimmed videos. The temporal relation is complex in those datasets, including challenges like composite action, and co-occurring action. For detecting actions in those complex videos, efficiently capturing both short-term and long-term temporal information in the video is critical. To this end, we propose a novel ConvTransformer network for action detection. This network comprises three main components: (1) Temporal Encoder module extensively explores global and local temporal relations at multiple temporal resolutions. (2) Temporal Scale Mixer module effectively fuses the multi-scale features to have a unified feature representation. (3) Classification module is used to learn the instance center-relative position and predict the frame-level classification scores. The extensive experiments on multiple datasets, including Charades, TSU and MultiTHUMOS, confirm the effectiveness of our proposed method. Our network outperforms the state-of-the-art methods on all the three datasets.

show abstract

Spotting Temporally Precise, Fine-Grained Events in Video

Hong¹,

Zhang²,

Gharbi³

et al. 2022

Preprint

View full text Add to dashboard Cite

We introduce the task of spotting temporally precise, finegrained events in video (detecting the precise moment in time events occur). Precise spotting requires models to reason globally about the full-time scale of actions and locally to identify subtle frame-to-frame appearance and motion differences that identify events during these actions. Surprisingly, we find that top performing solutions to prior video understanding tasks such as action detection and segmentation do not simultaneously meet both requirements. In response, we propose E2E-Spot, a compact, end-to-end model that performs well on the precise spotting task and can be trained quickly on a single GPU. We demonstrate that E2E-Spot significantly outperforms recent baselines adapted from the video action detection, segmentation, and spotting literature to the precise spotting task. Finally, we contribute new annotations and splits to several fine-grained sports action datasets to make these datasets suitable for future work on precise spotting.

show abstract

Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection

Cited by 3 publications

References 0 publications

CTRN: Class-Temporal Relational Network for Action Detection

CTRN: Class-Temporal Relational Network for Action Detection

MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection

Spotting Temporally Precise, Fine-Grained Events in Video

Contact Info

Product

Resources

About