Towards Semantic Fast-Forward and Stabilized Egocentric Videos

Silva, Michel M.; Ramos, Washington; Ferreira, João Pedro Klock; Campos, Mário F. M.

doi:10.1007/978-3-319-46604-0_40

Cited by 23 publications

(51 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we describe the experimental results on the Semantic Dataset [27] and a new multimodal semantic egocentric dataset. After detailing the datasets, we present the results followed by the ablation study on the components and efficiency analysis.…”

Section: Methodsmentioning

confidence: 99%

“…), a video score profile is created by extracting the relevant information and assigning a semantic score for each frame of the video (Figure 2-a). The confidence of the classifier combined with the locality and size of the regions of interest score are used as the semantic score [23,27].…”

Section: Temporal Semantic Profile Segmentationmentioning

confidence: 99%

“…All selected frames of each segment are concatenated to compose the final video (Figure 2-f). In this last step, we also run the egocentric video stabilization proposed in the work of Silva et al [27], which is properly designed to fastforwarded egocentric videos. The stabilizer creates smooth transitions by applying weighted homographies.…”

Section: Video Compositingmentioning

confidence: 99%

See 2 more Smart Citations

A Weighted Sparse Sampling and Smoothing Frame Transition Approach for Semantic Fast-Forward First-Person Videos

Silva

Ramos

Ferreira

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

Thanks to the advances in the technology of low-cost digital cameras and the popularity of the self-recording culture, the amount of visual data on the Internet is going to the opposite side of the available time and patience of the users. Thus, most of the uploaded videos are doomed to be forgotten and unwatched in a computer folder or website. In this work, we address the problem of creating smooth fast-forward videos without losing the relevant content. We present a new adaptive frame selection formulated as a weighted minimum reconstruction problem, which combined with a smoothing frame transition method accelerates first-person videos emphasizing the relevant segments and avoids visual discontinuities. The experiments show that our method is able to fast-forward videos to retain as much relevant information and smoothness as the state-ofthe-art techniques in less time. We also present a new 80hour multimodal (RGB-D, IMU, and GPS) dataset of firstperson videos with annotations for recorder profile, frame scene, activities, interaction, and attention 1 .

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Temporal Semantic Profile Segmentationmentioning

confidence: 99%

Section: Video Compositingmentioning

confidence: 99%

See 1 more Smart Citation

A Weighted Sparse Sampling and Smoothing Frame Transition Approach for Semantic Fast-Forward First-Person Videos

Silva

Ramos

Ferreira

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

show abstract

“…Current automatic fast-forward approaches mostly focus on adapting the playback speed based on either the similar-ity of each candidate clip to the query clip [31] or the motion activity patterns present in a video [3,29,30]. Some recent works use mutual information between frames to describe the fast-forward policy [11,12], or use shortest path distance over the graph that is constructed with semantic information extracted from frames [34,38]. This family of methods is most relevant to our goal.…”

Section: Related Workmentioning

confidence: 99%

“…However, these techniques require the processing of entire video and often take a long time to generate the subset. There are also video fast-forwarding techniques where the playback speed of the video is adjusted to meet the needs of users [3,10,13,31,32,34,38], but they often do not present an accurate representation and may still require processing of the entire video. Both types of approaches are not suitable for the resource-limited and time-critical systems we discussed above.…”

Section: Introductionmentioning

confidence: 99%

FFNet: Video Fast-Forwarding via Reinforcement Learning

Lan¹,

Panda

Zhu³

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

For many applications with limited computation, communication, storage and energy resources, there is an imperative need of computer vision methods that could select an informative subset of the input video for efficient processing at or near real time. In the literature, there are two relevant groups of approaches: generating a "trailer" for a video or fast-forwarding while watching/processing the video. The first group is supported by video summarization techniques, which require processing of the entire video to select an important subset for showing to users. In the second group, current fast-forwarding methods depend on either manual control or automatic adaptation of playback speed, which often do not present an accurate representation and may still require processing of every frame. In this paper, we introduce FastForwardNet (FFNet), a reinforcement learning agent that gets inspiration from video summarization and does fast-forwarding differently. It is an online framework that automatically fast-forwards a video and presents a representative subset of frames to users on the fly. It does not require processing the entire video, but just the portion that is selected by the fast-forward agent, which makes the process very computationally efficient. The online nature of our proposed method also enables the users to begin fast-forwarding at any point of the video. Experiments on two real-world datasets demonstrate that our method can provide better representation of the input video (about 6%-20% improvement on coverage of important frames) with much less processing requirement (more than 80% reduction in the number of frames processed).

show abstract

Making a long story short: A multi-importance fast-forwarding egocentric videos with the emphasis on relevant objects

Silva¹,

Ramos²,

Cadar³

et al. 2018

Journal of Visual Communication and Image Representation

View full text Add to dashboard Cite

The emergence of low-cost high-quality personal wearable cameras combined with the increasing storage capacity of video-sharing websites have evoked a growing interest in first-person videos, since most videos are composed of longrunning unedited streams which are usually tedious and unpleasant to watch. State-of-the-art semantic fast-forward methods currently face the challenge of providing an adequate balance between smoothness in visual flow and the emphasis on the relevant parts. In this work, we present the Multi-Importance Fast-Forward (MIFF), a fully automatic methodology to fast-forward egocentric videos facing these challenges. The dilemma of defining what is the semantic information of a video is addressed by a learning process based on the preferences of the user. Results show that the proposed method keeps over 3 times more semantic content than the state-of-the-art fast-forward. Finally, we discuss the need of a particular video stabilization technique for fast-forward egocentric videos 1 .

show abstract

Towards Semantic Fast-Forward and Stabilized Egocentric Videos

Cited by 23 publications

References 15 publications

A Weighted Sparse Sampling and Smoothing Frame Transition Approach for Semantic Fast-Forward First-Person Videos

A Weighted Sparse Sampling and Smoothing Frame Transition Approach for Semantic Fast-Forward First-Person Videos

FFNet: Video Fast-Forwarding via Reinforcement Learning

Making a long story short: A multi-importance fast-forwarding egocentric videos with the emphasis on relevant objects

Contact Info

Product

Resources

About