Clustering and synchronizing multi-camera video via landmark cross-correlation

Abstract

We propose a method to both identify and synchronize multicamera video recordings within a large collection of video and/or audio files. Landmark-based audio fingerprinting is used to match multiple recordings of the same event together and time-synchronize each file within the groups. Compared to prior work, we offer improvements towards event identification and a new synchronization refinement method that resolves inconsistent estimates and allows non-overlapping content to be synchronized within larger groups of recordings. Furthermore, the audio fingerprinting-based synchronization is shown to be equivalent to an efficient and scalable time-difference-of-arrival method using cross-correlation performed on a non-linearly transformed signal.

Fine landmark-based synchronization of ad-hoc microphone arrays

“…Traditional microphone-array techniques, such as beamforming and sound source localization, which rely on the knowledge of microphone positions and assume samplesynchronized audio channels, cannot be applied directly [2,3]. The synchronization problem between multiple audio channels has been addressed using generalized cross-correlation [2,4,5] and audio fingerprinting [5][6][7][8].…”

“…able for audio channels that are already coarsely synchronized [4,5]. Another synchronization approach is based on audio fingerprinting, which has been originally applied to music information retrieval [6], and clustering and synchronizing multi-camera videos [7,8]. By matching the audio fingerprints extracted from the sound track, the audio channels can be synchronized.…”

“…PRH extracts more detailed features including all the energy changes in consecutive bands and frames, and is more robust than onset to ambient noise and audio compression distortion [12]. Audio landmark operates in the time-frequency domain, and the extracted features consist of a set of timefrequency pairs [6][7][8]. With low computational cost and robustness against noise, audio fingerprinting techniques have been widely used in movie and video synchronization [13].…”

Temporal synchronization of multiple audio signals

“…For example, Bryan et al use audio-fingerprinting [13] to match strongly correlating signal-pairs [14] that are iteratively merged into larger clusters until a global solution is found. Similarly, Shrestha et al [15] and Cremer et al [16] generate multiple audiofingerprints for small segments in each audio track which are then individually matched against each other.…”

“…1). Our global approach complements techniques that prune bad or non-overlapping pairwise matches (e.g., by thresholding or fingerprint consistency [14]), and provides additional robustness to any remaining outlier pairwise matches.…”