Multimodal extraction of events and of information about the recording activity in user generated videos

Abstract

In this work we propose methods that exploit context sensor data modalities for the task of detecting interesting events and extracting high-level contextual information about the recording activity in user generated videos. Indeed, most camera-enabled electronic devices contain various auxiliary sensors such as accelerometers, compasses, GPS receivers, etc. Data captured by these sensors during the media acquisition have already been used to limit camera degradations such as shake and also to provide some basic tagging information such as the location. However, exploiting the sensor-recordings modality for subsequent higher-level information extraction such as interesting events has been a subject of rather limited research, further constrained to specialized acquisition setups. In this work, we show how these sensor modalities allow inferring information (camera movements, content degradations) about each individual video recording. In addition, we consider a multi-camera scenario, where multiple user generated recordings of a common scene (e.g., music concerts) are available. For this kind of scenarios we jointly analyze these multiple video recordings and their associated sensor modalities in order to extract higher-level semantics of the recorded media: based on the orientation of cameras we identify the region of interest of the recorded scene, by exploiting correlation in the motion of different cameras we detect generic interesting events and estimate their relative position. Furthermore, by Multimed Tools Appl (2014) analyzing also the audio content captured by multiple users we detect more specific interesting events. We show that the proposed multimodal analysis methods perform well on various recordings obtained in real live music performances.

Monitoring of User Generated Video Broadcasting Services

“…This information is used to annotate the generated video stream with the motion, environmental information such as the position of the device 4 . Based on this data, the detection of user behavior that leads to degraded video quality, such as camera shaking or wrong orientation of the phone, can also be observed directly on the users' device without analyzing (or transmitting) the video data itself [5]. In previous work [10], we found that degradations such as camera shakes have a huge impact on the QoE.…”

“…Also, software-based sensors indicate the current video recording properties or energy level of the device. Here, patterns can be used to identify recording degradations based on sensor readingsas shown by CriCri [5].…”

“…CriCri [5] shows a first step towards replacing video analysis with mechanisms that leverage different sensors in the recording devices to compose the video. We propose a monitoring and metric framework for the increasing need of mobile broadcasting systems that rely on video collection of multiple streams and composition of one superior video.…”

Video composition by the crowd

“…MoviMash is an advanced approach existing for live video composition. CriCri [3] shows a first step towards replacing video analysis with mechanisms, that leverage different sensors in the recording devices to compose the video. Without respecting visual features the resulting quality of such approaches degrades [4] but the speed increases.…”

“…Common artifacts in compressing video streams such as blockiness are mitigated by the agreed video parameters (framerate, resolution and target bitrate). Thus, CrowdRecord investigates degradations commonly occurring during recording with retail smartphones, to (1) ensure the correct camera orientation, (2) identify and measure camera shaking, (3) identify the field of interest of a recording, (4) find under-/overexposed frame regions, and (5) measure frame sharpness. Streams that fail in one of these categories are not considered for composition.…”