Proposal-Free Temporal Action Detection via Global Segmentation Mask Learning

Nag, Sauradip; Zhu, Xiatian; Song, Yi-Zhe; Xiang, Tao

doi:10.1007/978-3-031-20062-5_37

Cited by 20 publications

(10 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…CSA [37] enriches the proposal temporal context via attention transfer. Unlike most previous models adopting a sequential localization and classification pipeline, TAGS [29] [36,38,41]. FSL is often realized by meta-learning which simulates new tasks with novel classes represented by only a handful of labeled samples.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Zero-Shot Temporal Action Detection via Vision-Language Prompting

Nag

Zhu

Song

et al. 2022

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Few-shot (FS) and zero-shot (ZS) learning are two different approaches for scaling temporal action detection (TAD) to new classes. The former adapts a pretrained vision model to a new task represented by as few as a single video per class, whilst the latter requires no training examples by exploiting a semantic description of the new class. In this work, we introduce a new multi-modality few-shot (MMFS) TAD problem, which can be considered as a marriage of FS-TAD and ZS-TAD by leveraging few-shot support videos and new class names jointly. To tackle this problem, we further introduce a novel MUlti-modality PromPt mETa-learning (MUPPET) method. This is enabled by efficiently bridging pretrained vision and language models whilst maximally reusing already learned capacity. Concretely, we construct multi-modal prompts by mapping support videos into the textual token space of a vision-language model using a meta-learned adapter-equipped visual semantics tokenizer. To tackle large intra-class variation, we further design a query feature regulation scheme. Extensive experiments on ActivityNetv1.3 and THUMOS14 demonstrate that our MUPPET outperforms state-of-the-art alternative methods, often by a large margin. We also show that our MUPPET can be easily extended to tackle the few-shot object detection problem and again achieves the state-ofthe-art performance on MS-COCO dataset. The code will be available in https://github.com/sauradip/ MUPPET

show abstract

Section: Related Workmentioning

confidence: 99%

“…The objective of temporal action detection (TAD) is to predict the temporal duration (i.e., start and end time) and the class label of each action instance in an untrimmed video [3,17]. Conventional TAD methods [2,28,29,42,46,47,56] annotations for training. This thus severely limits their ability to scale to many classes.…”

Section: Introductionmentioning

confidence: 99%

Zero-Shot Temporal Action Detection via Vision-Language Prompting

Nag

Zhu

Song

et al. 2022

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…Concretely, early anchor-based methods [5,30,35] typically employ the multi-scale anchors and attach a classification head and a boundary regression head to refine these pre-defined anchors. Anchor-free methods [17,18,22,33,40] Learning on Pseudo Labels is an important yet key technology in semisupervised learning. However, most approaches [6,13,16,23,41] are limited to learning directly from the target class, so it is inevitable that the model will be misled by noisy pseudo labels.…”

Section: Related Workmentioning

confidence: 99%

“…Evaluation Datasets. Following conventions [22,39], we evaluate our proposed method on two challenging TAL benchmarks, i.e., THUMOS14 [12] and Activi-tyNet v1.3 [3]. THUMOS14 [12] contains 200 validation videos and 213 testing videos, including 20 action categories.…”

Section: Datasets and Metricsmentioning

confidence: 99%

See 1 more Smart Citation

Progress and Perspective of Artificial Intelligence and Machine Learning of Prediction in Anesthesiology

Xia

Jiang

2021

J. Shanghai Jiaotong Univ. (Sci.)

View full text Add to dashboard Cite

Charts are common in literature across different scientific fields, conveying rich information easily accessible to readers. Current chart-related tasks focus on either chart perception which refers to extracting information from the visual charts, or performing reasoning given the extracted data, e.g. in a tabular form. In this paper, we aim to establish a unified and label-efficient learning paradigm for joint perception and reasoning tasks, which can be generally applicable to different downstream tasks, beyond the question-answering task as specifically studied in peer works. Specifically, StructChart first reformulates the chart information from the popular tubular form (specifically linearized CSV) to the proposed Structured Triplet Representations (STR), which is more friendly for reducing the task gap between chart perception and reasoning due to the employed structured information extraction for charts. We then propose a Structuring Chart-oriented Representation Metric (SCRM) to quantitatively evaluate the performance for the chart perception task. To enrich the dataset for training, we further explore the possibility of leveraging the Large Language Model (LLM), enhancing the chart diversity in terms of both chart visual style and its statistical information. Extensive experiments are conducted on various chart-related tasks, demonstrating the effectiveness and promising potential for a unified chart perception-reasoning paradigm to push the frontier of chart understanding. Our simulation chart dataset is available at: https://github.com/UniModal4Reasoning/SimChart9K, and the pre-training code will be released for reproduction.

show abstract