Diversity-Aware Multi-Video Summarization

Panda, Rameswar; Mithun, Niluthpol Chowdhury; Roy-Chowdhury, Amit K.

doi:10.1109/tip.2017.2708902

Cited by 58 publications

(35 citation statements)

References 68 publications

(143 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…(2) The video summarization datasets [16], [29], [49], [63] contain long videos arranging from different domains. The objective is to extract a set of informative frames in order to briefly summarize the video content.…”

Section: Datasets Related To Instructional Video Analysismentioning

confidence: 99%

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Tang

Lu²,

Zhou³

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Thanks to the substantial and explosively inscreased instructional videos on the Internet, novices are able to acquire knowledge for completing various tasks. Over the past decade, growing efforts have been devoted to investigating the problem on instructional video analysis. However, the most existing datasets in this area have limitations in diversity and scale, which makes them far from many real-world applications where more diverse activities occur. To address this, we present a large-scale dataset named as "COIN" for COmprehensive INstructional video analysis. Organized with a hierarchical structure, the COIN dataset contains 11,827 videos of 180 tasks in 12 domains (e.g., vehicles, gadgets, etc.) related to our daily life. With a new developed toolbox, all the videos are annotated efficiently with a series of step labels and the corresponding temporal boundaries. In order to provide a benchmark for instructional video analysis, we evaluate plenty of approaches on the COIN dataset under five different settings. Furthermore, we exploit two important characteristics (i.e., task-consistency and ordering-dependency) for localizing important steps in instructional videos. Accordingly, we propose two simple yet effective methods, which can be easily plugged into conventional proposal-based action detection models. We believe the introduction of the COIN dataset will promote the future in-depth research on instructional video analysis for the community. Our dataset, annotation toolbox and source code are available at http://coin-dataset.github.io. DomainTaskStep VehiclesHousehold Items Change the Car Tire {unscrew the screws, jack up the car, remove the tire, put on the tire, tighten the screws } {remove the door knob, remove bolt and pin board, install new pin board, install new bolt, install new door knob } Replace the Door Knob

show abstract

Section: Datasets Related To Instructional Video Analysismentioning

confidence: 99%

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Tang

Lu²,

Zhou³

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Each video has 5 user summaries in the form of set of key frames. Tour20 [24] consists of 140 videos with a total duration of 7 hours and is designed primarily for multi video summarization. It is a collection of videos of a tourist place.…”

Section: Related Work 21 Datasetsmentioning

confidence: 99%

Realistic Video Summarization through VISIOCITY

Kaushal

Kothawade²,

Iyer³

et al. 2020

Proceedings of the 2nd International Workshop on AI for Smart TV Content Production, Access and Delivery

View full text Add to dashboard Cite

Automatic video summarization is still an unsolved problem due to several challenges. We take steps towards making it more realistic by addressing the following challenges. Firstly, the currently available datasets either have very short videos or have few long videos of only a particular type. We introduce a new benchmarking dataset called VISIOCITY which comprises of longer videos across six different categories with dense concept annotations capable of supporting different flavors of video summarization and other vision problems. Secondly, for long videos, human reference summaries, necessary for supervised video summarization techniques, are difficult to obtain. We present a novel recipe based on pareto optimality to automatically generate multiple reference summaries from indirect ground truth present in VISIOCITY. We show that these summaries are at par with human summaries. Thirdly, we demonstrate that in the presence of multiple ground truth summaries (due to the highly subjective nature of the task), learning from a single combined ground truth summary using a single loss function is not a good idea. We propose a simple recipe VISIOCITY-SUM to enhance an existing model using a combination of losses and demonstrate that it beats the current state of the art techniques. We also present a study of different desired characteristics of a good summary and demonstrate that a single measure (say F1) to evaluate a summary, as is the current typical practice, falls short in some ways. We propose an evaluation framework for better quantitative assessment of summary quality which is closer to human judgment than a single measure. We report the performance of a few representative techniques of video summarization on VISIOCITY assessed using various measures and bring out the limitation of the techniques and/or the assessment mechanism in modeling human judgment and demonstrate the effectiveness of our evaluation framework in doing so.

show abstract

“…Vidal et al [11] propose a framework to detect and reject outliers from the dataset using the solution of the proposed optimization program. Panda et al [12] develop a diversity-aware sparse optimization method for multi-video summarization by exploring the complementarity within the videos. Panda and Roy-Chowdhury [13] propose an unsupervised framework for summarizing top related videos by exploring complementarity within videos, and a sparse optimization method is developed to extract a diverse summary that is both interesting and representative in describing the video collection.…”

Section: Sparse Optimizationsmentioning

confidence: 99%

“…The summaries' quality is evaluated by the accuracy rate and error rate. Panda et al [12] introduce Tour20 dataset, which contains 140 videos with multiple manually created summaries. Song et al [36] introduce TVSum50 dataset, which contains 50 videos with their shotlevel importance scores annotated via crowdsourcing.…”

Section: Quantitative Evaluation and Benchmarkmentioning

confidence: 99%

See 1 more Smart Citation

A bottom-up summarization algorithm for videos in the wild

Pan

Zheng

Zhang

et al. 2019

EURASIP J. Adv. Signal Process.

View full text Add to dashboard Cite

Video summarization aims to provide a compact video representation while preserving the essential activities of the original video. Most existing video summarization approaches relay on identifying important frames and optimizing target energy by a global optimum solution. But global optimum may fail to express continuous action or realistically validate how human beings perceive a story. In this paper, we present a bottom-up approach named clip growing for video summarization, which allows users to customize the quality of the video summaries. The proposed approach firstly uses clustering to oversegment video frames into video clips based on their similarity and proximity. Simultaneously, the importance of frames and clips is evaluated from their corresponding dissimilarity and representativeness. Then, video clips and frames are gradually selected according to their energy rank, until reaching the target length. Experimental results on SumMe dataset show that our algorithm can produce promising results compared to existing algorithms. Several video summarizations results are presented in supplementary material.

show abstract

Diversity-Aware Multi-Video Summarization

Cited by 58 publications

References 68 publications

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Realistic Video Summarization through VISIOCITY

A bottom-up summarization algorithm for videos in the wild

Contact Info

Product

Resources

About