Unsupervised event segmentation of news content with multimodal cues

Broilo, Mattia; Zavesky, Eric; Basso, Andrea; Natale, Francesco G. B. De

doi:10.1145/1877850.1877862

Cited by 9 publications

(8 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the following we firstly discuss some of the existing content-based multimodal event detection methods (i.e., those combining the analysis of video, audio and eventually text) for the purpose of multimedia retrieval. In [4], events are extracted from news videos in a multimodal unsupervised fashion. This method combines information from audio, visual appearance, faces, and mid-level semantic concepts by applying coherence rules.…”

Section: Prior Artmentioning

confidence: 99%

“…As in [26], in one of our event detection methods we also exploit correlations between different modalities, namely between motion sensor data and audio-content data. However, the works described in [4] and [26] analyze visual data (apart from other types of data) which is usually computationally expensive. The analysis methods that we propose are instead light-weight, as they mainly consider auxiliary sensor data sampled at low rates, as we will describe in the next sections of this paper.…”

Section: Prior Artmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal extraction of events and of information about the recording activity in user generated videos

Cricri

Dabov

Curcio

et al. 2012

Multimed Tools Appl

View full text Add to dashboard Cite

In this work we propose methods that exploit context sensor data modalities for the task of detecting interesting events and extracting high-level contextual information about the recording activity in user generated videos. Indeed, most camera-enabled electronic devices contain various auxiliary sensors such as accelerometers, compasses, GPS receivers, etc. Data captured by these sensors during the media acquisition have already been used to limit camera degradations such as shake and also to provide some basic tagging information such as the location. However, exploiting the sensor-recordings modality for subsequent higher-level information extraction such as interesting events has been a subject of rather limited research, further constrained to specialized acquisition setups. In this work, we show how these sensor modalities allow inferring information (camera movements, content degradations) about each individual video recording. In addition, we consider a multi-camera scenario, where multiple user generated recordings of a common scene (e.g., music concerts) are available. For this kind of scenarios we jointly analyze these multiple video recordings and their associated sensor modalities in order to extract higher-level semantics of the recorded media: based on the orientation of cameras we identify the region of interest of the recorded scene, by exploiting correlation in the motion of different cameras we detect generic interesting events and estimate their relative position. Furthermore, by Multimed Tools Appl (2014) analyzing also the audio content captured by multiple users we detect more specific interesting events. We show that the proposed multimodal analysis methods perform well on various recordings obtained in real live music performances.

show abstract

Section: Prior Artmentioning

confidence: 99%

Section: Prior Artmentioning

confidence: 99%

Multimodal extraction of events and of information about the recording activity in user generated videos

Cricri

Dabov

Curcio

et al. 2012

Multimed Tools Appl

View full text Add to dashboard Cite

show abstract

“…Several approaches have focused on the discovery of near-duplicate repetitions [6,7,8] but cannot deal with the crucial issue of variability across repetitions. Since structurally relevant events are often characterized by their strong visual consistency, the problem of mining repeating structural elements has been addressed using clustering techniques (see, e.g., [9,10,11,12,13,14]). But several problems arise from clustering, such as deciding the optimal number of clusters or dealing with outliers.…”

Section: Introductionmentioning

confidence: 99%

Unsupervised mining of audiovisually consistent segments in videos with application to structure analysis

Ben

Gravier

2011

2011 IEEE International Conference on Multimedia and Expo

View full text Add to dashboard Cite

In this paper, a multimodal event mining technique is proposed to discover repeating video segments exhibiting audio and visual consistency in a totally unsupervised manner. The mining strategy first exploits independent audio and visual cluster analysis to provide segments which are consistent in both their visual and audio modalities, thus likely corresponding to a unique underlying event. A subsequent modeling stage using discriminative models enables accurate detection of the underlying event throughout the video. Event mining is applied to unsupervised video structure analysis, using simple heuristics on occurrence patterns of the events discovered to select those relevant to the video structure. Results on TV programs ranging from news to talk shows and games, show that structurally relevant events are discovered with precisions ranging from 87 % to 98 % and recalls from 59 % to 94 %.

show abstract

“…Starting from this independent analysis of the individual features of the content, we add a compositional mining step on the similarities found across video shots to detect patterns containing the same speaking person in the same environment. This algorithm builds on our early work on event detection [14] using the same clustering process but greatly extend the relevant potential introducing the differentiation of the role played by the people in the video better exploiting the time component of the detected anchorpersons. This time analysis expressed by appearance duration in video, by interval time between appearances and by number of appearance times of the anchorpersons results an innovative approach for unsupervised differentiation avoiding any training phase.…”

Section: Introductionmentioning

confidence: 99%

“…The Viola-Jones face detection algorithm is used to find faces locations inside the key-frames images [9]. Audio segments, key-frames and detected faces are then analyzed separately using different tuned clustering algorithms based on the method presented in [14]. With each independent analysis phase, it is possible to add a data mining step on the similarities found in the video to obtain patterns containing the same speaking person in the same environment.…”

Section: Introductionmentioning

confidence: 99%

Unsupervised anchorpersons differentiation in news video

Broilo

Basso

Natale

2011

2011 9th International Workshop on Content-Based Multimedia Indexing (CBMI)

Self Cite

View full text Add to dashboard Cite

The automatic extraction of video structure from content is of key importance to enable a variety of multimedia services that span from search and retrieval to content manipulation. An unsupervised independent unimodal clustering method for anchorpersons detection and differentiation in newscasts is presented in this paper. The algorithm exploits audio, frame and face information to identify major cast in the content. These three components are first processed independently during the cluster analysis and then jointly in a compositional mining phase. A differentiation of the role played by the people in the video has been implemented exploiting the temporal characteristics of the detected anchorpersons.Experiments show significant precision/recall results thus opening further research directions in video analysis, particularly when the content is highly structured as in TV newscasts.

show abstract

Unsupervised event segmentation of news content with multimodal cues

Cited by 9 publications

References 10 publications

Multimodal extraction of events and of information about the recording activity in user generated videos

Multimodal extraction of events and of information about the recording activity in user generated videos

Unsupervised mining of audiovisually consistent segments in videos with application to structure analysis

Unsupervised anchorpersons differentiation in news video

Contact Info

Product

Resources

About