A Content-Adaptive Analysis and Representation Framework for Audio Event Discovery from "Unscripted" Multimedia

Abstract

We propose a content-adaptive analysis and representation framework to discover events using audio features from "unscripted" multimedia such as sports and surveillance for summarization. The proposed analysis framework performs an inlier/outlier-based temporal segmentation of the content. It is motivated by the observation that "interesting" events in unscripted multimedia occur sparsely in a background of usual or "uninteresting" events. We treat the sequence of low/mid-level features extracted from the audio as a time series and identify subsequences that are outliers. The outlier detection is based on eigenvector analysis of the affinity matrix constructed from statistical models estimated from the subsequences of the time series. We define the confidence measure on each of the detected outliers as the probability that it is an outlier. Then, we establish a relationship between the parameters of the proposed framework and the confidence measure. Furthermore, we use the confidence measure to rank the detected outliers in terms of their departures from the background process. Our experimental results with sequences of low-and mid-level audio features extracted from sports video show that "highlight" events can be extracted effectively as outliers from a background process using the proposed framework. We proceed to show the effectiveness of the proposed framework in bringing out suspicious events from surveillance videos without any a priori knowledge. We show that such temporal segmentation into background and outliers, along with the ranking based on the departure from the background, can be used to generate content summaries of any desired length. Finally, we also show that the proposed framework can be used to systematically select "key audio classes" that are indicative of events of interest in the chosen domain.

Semi-supervised context adaptation: case study of audience excitement recognition

“…On the other hand, systems that do not rely on recognition of pre-defined context-specific cues usually require computationally expensive re-training, as e.g. in [28], where adaptation to different types of sport events (soccer, golf, etc.) is performed via audio segmentation and outlier detection in the data of each context.…”

Graph-based multilevel temporal video segmentation

“…Moreover, a bipartite graph-based audiovisual alignment algorithm was introduced. In [18], a content-adaptive analysis and representation framework using graph-based approaches was proposed for audio event discovery. In that contribution, foreground cut criterion [16] was adapted to detect the foreground when the background is structured.…”

Generative Process Tracking for Audio Analysis

“…Towards that end, in our previous work, we had detected audio "backgrounds" and "foregrounds" from a time series of cepstral features extracted from stored content [5]. Recall our definition of "background" and "foreground" in time series.…”

“…We have shown that one can detect highlight segments in sports audio and suspicious events in surveillance audio by detecting such "backgrounds" and "foregrounds". However, the computational complexity and the latency of the proposed approach in [5] is high.…”

“…In this paper, we present an application where we arrive at the "backgrounds" and the "foregrounds" using only the causal time series and using fewer computational resources thereby making the approach in [5] more practical and realizable. The proposed approach adaptively tracks the background process to detect outliers that cannot be explained by the background model estimate at that time.…”