Pano2Vid: Automatic Cinematography for Watching 360 $$^{\circ }$$  Videos

Su, Yu-Chuan; Jayaraman, Dinesh; Grauman, Kristen

doi:10.1007/978-3-319-54190-7_10

Cited by 81 publications

(169 citation statements)

References 40 publications

Supporting

Mentioning

167

Contrasting

Order By: Relevance

“…Several methods have been recently developed for navigating 360 • videos by finding NFoV of interest. These methods mainly leverage visual information such as saliency [47], [48], [49], [50], [51]. However, we tackle this problem from the perspective of audio sources.…”

Section: Video Applicationsmentioning

confidence: 99%

Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications

Senocak

Kim

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Visual events are usually accompanied by sounds in our daily lives. However, can the machines learn to correlate the visual scene and sound, as well as localize the sound source only by observing them like humans? To investigate its empirical learnability, in this work we first present a novel unsupervised algorithm to address the problem of localizing sound sources in visual scenes. In order to achieve this goal, a two-stream network structure which handles each modality with attention mechanism is developed for sound source localization. The network naturally reveals the localized response in the scene without human annotation. In addition, a new sound source dataset is developed for performance evaluation. Nevertheless, our empirical evaluation shows that the unsupervised method generates false conclusions in some cases. Thereby, we show that this false conclusion cannot be fixed without human prior knowledge due to the well-known correlation and causality mismatch misconception. To fix this issue, we extend our network to the supervised and semi-supervised network settings via a simple modification due to the general architecture of our two-stream network. We show that the false conclusions can be effectively corrected even with a small amount of supervision, i.e., semi-supervised setup. Furthermore, we present the versatility of the learned audio and visual embeddings on the cross-modal content alignment and we extend this proposed algorithm to a new application, sound saliency based automatic camera view panning in 360 • videos.

show abstract

Section: Video Applicationsmentioning

confidence: 99%

Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications

Senocak

Kim

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…After that, when the buffer is not empty, it is very important to predict the FoV accurately [41]- [43]. Existing FoV estimation methods can be roughly classified into three categories, i.e., data driven approaches [44], probability model based approaches [35] [45] and motion saliency detection based approaches [46]. Although data driven approaches and motion saliency detection based approaches achieve good performance, the viewport movement depends only on the subjective will of a user, and it can never be predicted accurately.…”

Section: B Fov Switching and Tile Priority Modelmentioning

confidence: 99%

Spatial and Temporal Consistency-Aware Dynamic Adaptive Streaming for 360-Degree Videos

Yuan

Zhao

Hou

et al. 2020

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

The 360-degree video allows users to enjoy the whole scene by interactively switching viewports. However, the huge data volume of the 360-degree video limits its remote applications via network. To provide high quality of experience (QoE) for remote web users, this paper presents a tile-based adaptive streaming method for 360-degree videos. First, we propose a simple yet effective rate adaptation algorithm to determine the requested bitrate for downloading the current video segment by considering the balance between the buffer length and video quality. Then, we propose to use a Gaussian model to predict the field of view at the beginning of each requested video segment. To deal with the circumstance that the view angle is switched during the display of a video segment, we propose to download all the tiles in the 360-degree video with different priorities based on a Zipf model. Finally, in order to allocate bitrates for all the tiles, a two-stage optimization algorithm is proposed to preserve the quality of tiles in FoV and guarantee the spatial and temporal smoothness. Experimental results demonstrate the effectiveness and advantage of the proposed method compared with the state-of-the-art methods. That is, our method preserves both the quality and the smoothness of tiles in FoV, thus providing the best QoE for users.Index Terms-360-degree video, field of view, rate adaptation, DASH, video compression, quality of experience.

show abstract

“…The Pano2Vid work by Su et al [2016] performs a related, but different task of automatically producing a narrow field-of-view video from a 360 • video. A recent follow-up work extended this to optimize for zoom as well [Su and Grauman 2017].…”

Section: Related Workmentioning

confidence: 99%

“…Our method can also be easily integrated with automatically generated constraints. Automatic saliency methods for 360 • video (e.g., [Su et al 2016]) determine the most visually salient regions, which can be directly interpreted as positive constraints. Alternately, as we are estimating 3D translation direction in Section 3.2, we can use this to keep the camera pointed roughly in the forward motion direction, which provides a comfortable "first-person" viewing experience.…”

Section: Source Of Constraintsmentioning

confidence: 99%

See 1 more Smart Citation

Joint Stabilization and Direction of 360° Videos

et al. 2019

View full text Add to dashboard Cite

360• video provides an immersive experience for viewers, allowing them to freely explore the world by turning their head. However, creating highquality 360 • video content can be challenging, as viewers may miss important events by looking in the wrong direction, or they may see things that ruin the immersion, such as stitching artifacts and the film crew. We take advantage of the fact that not all directions are equally likely to be observed; most viewers are more likely to see content located at "true north", i.e. in front of them, due to ergonomic constraints. We therefore propose 360 • video direction, where the video is jointly optimized to orient important events to the front of the viewer and visual clutter behind them, while producing smooth camera motion. Unlike traditional video, viewers can still explore the space as desired, but with the knowledge that the most important content is likely to be in front of them. Constraints can be user guided, either added directly on the equirectangular projection or by recording "guidance" viewing directions while watching the video in a VR headset, or automatically computed, such as via visual saliency or forward motion direction. To accomplish this, we propose a new motion estimation technique specifically designed for 360 • video which outperforms the commonly used 5-point algorithm on wide angle video. We additionally formulate the direction problem as an optimization where a novel parametrization of spherical warping allows us to correct for some degree of parallax effects. We compare our approach to recent methods that address stabilization-only and converting 360 • video to narrow field-of-view video. Our pipeline can also enable the viewing of wide angle non-360 • footage in a spherical 360 • space, giving an immersive "virtual cinema" experience for a wide range of existing content filmed with first-person cameras.

show abstract

Pano2Vid: Automatic Cinematography for Watching 360 $$^{\circ }$$ Videos

Cited by 81 publications

References 40 publications

Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications

Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications

Spatial and Temporal Consistency-Aware Dynamic Adaptive Streaming for 360-Degree Videos

Joint Stabilization and Direction of 360° Videos

Contact Info

Product

Resources

About