Personalized Cinemagraphs Using Semantic Understanding and Collaborative Learning

Oh, Tae-Hyun; Joo, Kyungdon; Joshi, Neel; Wang, Baoyuan; Kweon, In So; Kang, Sing Bing

doi:10.1109/iccv.2017.552

Cited by 8 publications

(3 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We can control the magnitude of dynamism in the cinemagraph by leveraging the controllability of magnification factors without retraining. We use the cinemagraph data generated by [21].…”

Section: A26 Applicationsmentioning

confidence: 99%

Learning-Based Video Motion Magnification

Oh¹,

Jaroensri²,

Kim³

et al. 2018

Lecture Notes in Computer Science

Self Cite

112

104

View full text Add to dashboard Cite

Video motion magnification techniques allow us to see small motions previously invisible to the naked eyes, such as those of vibrating airplane wings, or swaying buildings under the influence of the wind. Because the motion is small, the magnification results are prone to noise or excessive blurring. The state of the art relies on hand-designed filters to extract representations that may not be optimal. In this paper, we seek to learn the filters directly from examples using deep convolutional neural networks. To make training tractable, we carefully design a synthetic dataset that captures small motion well, and use two-frame input for training. We show that the learned filters achieve high-quality results on real videos, with less ringing artifacts and better noise characteristics than previous methods. While our model is not trained with temporal filters, we found that the temporal filters can be used with our extracted representations up to a moderate magnification, enabling a frequencybased motion selection. Finally, we analyze the learned filters and show that they behave similarly to the derivative filters used in previous works. Our code, trained model, and datasets will be available online.

show abstract

“…We can control the magnitude of dynamism in the cinemagraph by leveraging the controllability of magnification factors without retraining. We use the cinemagraph data generated by [21].…”

Section: A26 Applicationsmentioning

confidence: 99%

Learning-Based Video Motion Magnification

Oh¹,

Jaroensri²,

Kim³

et al. 2018

Lecture Notes in Computer Science

Self Cite

112

104

View full text Add to dashboard Cite

show abstract

“…Summaries generated by low-level features or model constraints rarely retain high level context information. More recent works focus on retaining semantic information for better human correspondence: important objects (Lee et al [17]), object tracks (Liu et al [20]), motion based summaries via semantic object context (Oh et al [26]), and object relationships (Lu et al [21]). While these methods deal with higher level context information, they only consider a single crafted criterion to produce summaries.…”

Section: Related Workmentioning

confidence: 99%

Contextually Customized Video Summaries Via Natural Language

Choi

Kweon

2018

2018 IEEE Winter Conference on Applications of Computer Vision (WACV)

Self Cite

View full text Add to dashboard Cite

The best summary of a long video differs among different people due to its highly subjective nature. Even for the same person, the best summary may change with time or mood. In this paper, we introduce the task of generating contextually customized video summaries through simple text. First, we train a deep architecture to effectively learn semantic embeddings of video frames by leveraging the abundance of image-caption data via a progressive manner, whereby our algorithm is able to select semantically relevant video segments for a contextually meaningful video summary, given a user-specific text description or even a single sentence. In order to evaluate our customized video summaries, we conduct experimental comparison with baseline methods that utilize ground-truth information. Despite the challenging baselines, our method still manages to show comparable or even exceeding performance. We also demonstrate that our method is able to automatically generate semantically diverse video summaries even without any text input.

show abstract

“…From a scenery image, humans can imagine how the clouds move and the sky color changes as time goes by. Reproducing such transitions in scenery images is a common subject of not only artistic contents called cinemagraph [Bai et al 2012;Liao et al 2013;Oh et al 2017] but also various techniques for image manipulation (e.g., scene completion [Hays and Efros 2007], time-lapse mining [Martin-Brualla et al 2015], attribute editing [Laffont et al 2014;Shih et al 2013], and sky replacement [Tsai et al 2016]). However, creating a natural animation from a scenery image remains a challenging task in the fields of computer graphics and computer vision.…”

Section: Introductionmentioning

confidence: 99%

Animating landscape

2019

View full text Add to dashboard Cite

Fig. 1. Given a single scenery image, our method predicts the motion (e.g., moving clouds) and appearance (e.g., time-varying colors) separately to generate a cyclic animation via self-supervised learning of time-lapse videos using our convolutional neural networks that infer backward flow fields (insets) and color transfer functions for converting the input image. The flow fields are visualized using the colormap shown in Figures 8 and 9. The output frame size is 1, 024 × 576. Please see the supplemental video for the resultant animations. Input photo: Pixabay/Pexel.com.Automatic generation of a high-quality video from a single image remains a challenging task despite the recent advances in deep generative models. This paper proposes a method that can create a high-resolution, long-term animation using convolutional neural networks (CNNs) from a single landscape image where we mainly focus on skies and waters. Our key observation is that the motion (e.g., moving clouds) and appearance (e.g., time-varying colors in the sky) in natural scenes have different time scales. We thus learn them separately and predict them with decoupled control while handling future uncertainty in both predictions by introducing latent codes. Unlike previous methods that infer output frames directly, our CNNs predict spatially-smooth intermediate data, i.e., for motion, flow fields for warping, and for appearance, color transfer maps, via self-supervised learning, i.e., without explicitly-provided ground truth. These intermediate data are applied not to each previous output frame, but to the input image only once for each output frame. This design is crucial to alleviate error accumulation in long-term predictions, which is the essential problem in previous recurrent approaches. The output frames can be looped like cinemagraph, and also be controlled directly by specifying latent codes or indirectly via visual annotations. We demonstrate the effectiveness of our method through comparisons with the state-of-the-arts on video prediction as well as appearance manipulation. Resultant videos, codes, and datasets will be available at

show abstract

Personalized Cinemagraphs Using Semantic Understanding and Collaborative Learning

Cited by 8 publications

References 34 publications

Learning-Based Video Motion Magnification

Learning-Based Video Motion Magnification

Contextually Customized Video Summaries Via Natural Language

Animating landscape

Contact Info

Product

Resources

About