Video-Story Composition via Plot Analysis

Choi, Jinsoo; Oh, Tae-Hyun; Kweon, In So

doi:10.1109/cvpr.2016.340

Cited by 16 publications

(50 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They proposed several approaches to solve this problem, and the best algorithm has been found to be the ensemble method. It uses LSTM [12] Table 1: Comparison between the proposed approach and existing solutions, Skip-RNN (SRNN) [1], Plot Analysis (PA) [8] and Two Stream RNN with Submodular Ranking (2RNNS) [35]. • means the function is originally not supported, but could be supported with minor adjustments.…”

Section: Visual Materials Sequencingmentioning

confidence: 99%

“…However, considering temporal order alone may not optimize the viewing experience. Choi et al proposed the problem of video-story composition, which is to sequence a set of video clips considering semantics, motions and dynamics [8]. They calculate dissimilarity and the plot penalty, which encourages the sequence of video clips to follow the proposed general plot of stories.…”

Section: Visual Materials Sequencingmentioning

confidence: 99%

“…The input, output and model structures are the same as RNN. • Plot Analysis (PA) [8]: which calculates the dynamics and dissimilarity to search for the best order of all visual materials. In the experiment, we only keep the first N max visual materials.…”

Section: Experiments Settingsmentioning

confidence: 99%

See 2 more Smart Citations

Generating Persuasive Visual Storylines for Promotional Videos

Liu

Dong

et al. 2019

Proceedings of the 28th ACM International Conference on Information and Knowledge Management

View full text Add to dashboard Cite

Video contents have become a critical tool for promoting products in E-commerce. However, the lack of automatic promotional video generation solutions makes large-scale video-based promotion campaigns infeasible. The first step of automatically producing promotional videos is to generate visual storylines, which is to select the building block footage and place them in an appropriate order. This task is related to the subjective viewing experience. It is hitherto performed by human experts and thus, hard to scale. To address this problem, we propose WundtBackpack, an algorithmic approach to generate storylines based on available visual materials, which can be video clips or images. It consists of two main parts, 1) the Learnable Wundt Curve to evaluate the perceived persuasiveness based on the stimulus intensity of a sequence of visual materials, which only requires a small volume of data to train; and 2) a clustering-based backpacking algorithm to generate persuasive sequences of visual materials while considering video length constraints. In this way, the proposed approach provides a dynamic structure to empower artificial intelligence (AI) to organize video footage in order to construct a sequence of visual stimuli with persuasive power. Extensive real-world experiments show that our approach achieves close to 10% higher perceived persuasiveness scores by human testers, and 12.5% higher expected revenue compared to the best performing state-of-the-art approach.

show abstract

Section: Visual Materials Sequencingmentioning

confidence: 99%

Section: Visual Materials Sequencingmentioning

confidence: 99%

See 1 more Smart Citation

Generating Persuasive Visual Storylines for Promotional Videos

Liu

Dong

et al. 2019

Proceedings of the 28th ACM International Conference on Information and Knowledge Management

View full text Add to dashboard Cite

show abstract

“…To organize these captured videos, they are usually edited and processed to be a concise format at a later time. Since the manual post-processing is time-consuming and labor-intensive, automatic algorithms are developed to process these unorganized videos, e.g., generation of a "short story" from a collection of videos [6]. In this work, we adopt this problem setting and aim to composite a smooth and meaningful video-story from video clips.…”

Section: Introductionmentioning

confidence: 99%

“…Specifically, we describe our task as: given a set of clips taken by a person during an activity or experience, we find out an order of the clips which composes a story containing smooth transitions in terms of semantics, motions, and activity dynamics that match the storyline structures [6] (see Figure 1). Note that, different from the video summarization task that aims to select keyframes out of a long video [32,34,21,17], the "story composition" problem described in this paper considers transitions between selected subshots and produces the consistent story in the temporal domain.…”

Section: Introductionmentioning

confidence: 99%

Learning Video-Story Composition via Recurrent Neural Network

Zhong

Tsai

Liu

et al. 2018

2018 IEEE Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

In this paper, we propose a learning-based method to compose a video-story from a group of video clips that describe an activity or experience. We learn the coherence between video clips from real videos via the Recurrent Neural Network (RNN) that jointly incorporates the spatial-temporal semantics and motion dynamics to generate smooth and relevant compositions. We further rearrange the results generated by the RNN to make the overall video-story compatible with the storyline structure via a submodular ranking optimization process. Experimental results on the video-story dataset show that the proposed algorithm outperforms the state-of-the-art approach.

show abstract

BERT-hLSTMs: BERT and hierarchical LSTMs for visual storytelling

Dai

Guérin

et al. 2021

Computer Speech & Language

View full text Add to dashboard Cite

Visual storytelling is a creative and challenging task, aiming to automatically generate a storylike description for a sequence of images. The descriptions generated by previous visual storytelling approaches lack coherence because they use word-level sequence generation methods and do not adequately consider sentence-level dependencies. To tackle this problem, we propose a novel hierarchical visual storytelling framework which separately models sentence-level and word-level semantics. We use the transformer-based BERT to obtain embeddings for sentences and words. We then employ a hierarchical LSTM network: the bottom LSTM receives as input the sentence vector representation from BERT, to learn the dependencies between the sentences corresponding to images, and the top LSTM is responsible for generating the corresponding word vector representations, taking input from the bottom LSTM. Experimental results demonstrate that our model outperforms most closely related baselines under automatic evaluation metrics BLEU and CIDEr, and also show the effectiveness of our method with human evaluation.

show abstract

Video-Story Composition via Plot Analysis

Cited by 16 publications

References 22 publications

Generating Persuasive Visual Storylines for Promotional Videos

Generating Persuasive Visual Storylines for Promotional Videos

Learning Video-Story Composition via Recurrent Neural Network

BERT-hLSTMs: BERT and hierarchical LSTMs for visual storytelling

Contact Info

Product

Resources

About