A Memory Network Approach for Story-Based Temporal Summarization of 360° Videos

Lee, Sang-Ho; Sung, Jinyoung; Yu, Yang; Kim, Gun-Hee

doi:10.1109/cvpr.2018.00153

Cited by 54 publications

(35 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Keys are used to address relevant memories whose corresponding values are returned. Recently, the memory networks have been applied to some vision problems such as personalized image captioning [25], visual tracking [41], movie understanding [23], and summarization [17].…”

Section: Memory Networkmentioning

confidence: 99%

Video Object Segmentation Using Space-Time Memory Networks

Lee

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

531

417

View full text Add to dashboard Cite

We propose a novel solution for semi-supervised video object segmentation. By the nature of the problem, available cues (e.g. video frame(s) with object masks) become richer with the intermediate predictions. However, the existing methods are unable to fully exploit this rich source of information. We resolve the issue by leveraging memory networks and learn to read relevant information from all available sources. In our framework, the past frames with object masks form an external memory, and the current frame as the query is segmented using the mask information in the memory. Specifically, the query and the memory are densely matched in the feature space, covering all the space-time pixel locations in a feed-forward fashion. Contrast to the previous approaches, the abundant use of the guidance information allows us to better handle the challenges such as appearance changes and occlussions. We validate our method on the latest benchmark sets and achieved the state-of-the-art performance (overall score of 79.4 on Youtube-VOS val set, J of 88.7 and 79.2 on DAVIS 2016/2017 val set respectively) while having a fast runtime (0.16 second/frame on DAVIS 2016 val set).

show abstract

Section: Memory Networkmentioning

confidence: 99%

Video Object Segmentation Using Space-Time Memory Networks

Lee

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

531

417

View full text Add to dashboard Cite

show abstract

“…It produces a spherical score map for 360 • video segments and uses a sliding window kernel to decide which view is suitable for highlighting. Lee et al [156] proposed a past-future memory network with two external memories. The memories are used for storing previously chosen subshots and future candidate subshots' embeddings for temporal summarization of 360 • videos.…”

Section: Vr Image and Video Editingmentioning

confidence: 99%

VR content creation and exploration with deep learning: A survey

Wang

Lyu

et al. 2020

Comp. Visual Media

View full text Add to dashboard Cite

Virtual reality (VR) offers an artificial, computer generated simulation of a real life environment. It originated in the 1960s and has evolved to provide increasing immersion, interactivity, imagination, and intelligence. Because deep learning systems are able to represent and compose information at various levels in a deep hierarchical fashion, they can build very powerful models which leverage large quantities of visual media data. Intelligence of VR methods and applications has been significantly boosted by the recent developments in deep learning techniques. VR content creation and exploration relates to image and video analysis, synthesis and editing, so deep learning methods such as fully convolutional networks and general adversarial networks are widely employed, designed specifically to handle panoramic images and video and virtual 3D scenes. This article surveys recent research that uses such deep learning methods for VR content creation and exploration. It considers the problems involved, and discusses possible future directions in this active and emerging research area. Keywords virtual reality; deep learning; neural networks; 360 • image and video virtual content

show abstract

“…So, the goal was to train the summarizer (that contains the generator) in order to maximally confuse the discriminator when trying to distinguish the original from the reconstructed video; a condition that indicates a highly representative keyframe summary. Based on an implemented variation of this model [5] (used to evaluate SUM-GAN when summarizing 360 • videos [15]), we scrutinized features of the architecture and the training process that could be fine-tuned to improve the model's performance. As depicted in the block-diagram of that: i) contains a linear compression layer which reduces the size of the input feature vectors and the number of learned parameters, ii) follows an incremental approach for training the model's components, and iii) applies a stepwise label-based learning strategy for the adversarial part of the architecture.…”

Section: Building On Adversarial Learningmentioning

confidence: 99%

Unsupervised Video Summarization via Attention-Driven Adversarial Learning

Apostolidis

Adamantidou

Metsai

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

This paper presents a new video summarization approach that integrates an attention mechanism to identify the significant parts of the video, and is trained unsupervisingly via generative adversarial learning. Starting from the SUM-GAN model, we first develop an improved version of it (called SUM-GAN-sl) that has a significantly reduced number of learned parameters, performs incremental training of the model's components, and applies a stepwise label-based strategy for updating the adversarial part. Subsequently, we introduce an attention mechanism to SUM-GAN-sl in two ways: i) by integrating an attention layer within the variational auto-encoder (VAE) of the architecture (SUM-GAN-VAAE), and ii) by replacing the VAE with a deterministic attention auto-encoder (SUM-GAN-AAE). Experimental evaluation on two datasets (SumMe and TVSum) documents the contribution of the attention auto-encoder to faster and more stable training of the model, resulting in a significant performance improvement with respect to the original model and demonstrating the competitiveness of the proposed SUM-GAN-AAE against the state of the art.

show abstract

A Memory Network Approach for Story-Based Temporal Summarization of 360° Videos

Cited by 54 publications

References 46 publications

Video Object Segmentation Using Space-Time Memory Networks

Video Object Segmentation Using Space-Time Memory Networks

VR content creation and exploration with deep learning: A survey

Unsupervised Video Summarization via Attention-Driven Adversarial Learning

Contact Info

Product

Resources

About