Learning Multimodal Temporal Representation for Dubbing Detection in Broadcast Media

Abstract

Person discovery in the absence of prior identity knowledge requires accurate association of visual and auditory cues. In broadcast data, multimodal analysis faces additional challenges due to narrated voices over muted scenes or dubbing in different languages. To address these challenges, we define and analyze the problem of dubbing detection in broadcast data, which has not been explored before. We propose a method to represent the temporal relationship between the auditory and visual streams. This method consists of canonical correlation analysis to learn a joint multimodal space, and long short term memory (LSTM) networks to model cross-modality temporal dependencies. Our contributions also include the introduction of a newly acquired dataset of face-speech segments from TV data, which we have made publicly available. The proposed method achieves promising performance on this real world dataset as compared to several baselines.

Speaker Inconsistency Detection in Tampered Video

“…For each database, the Train set was used to train PCA matrix, which was then applied to all combined features for all samples in the database. Canonical-correlation analysis (CCA) is also sometimes [2], [3] used to harmonize features of two modalities prior to the dimensionality reduction, but, in our experiments, we found this technique to have little effect on the results (about 1% reduction in error) and, therefore, do not report it in this paper.…”

“…As per the latest related work [3], [4], [5], [6], we also use 13 MFCC features with their delta, double-delta derivatives [11], and energy (40 coefficients in total) to characterize speech in audio. MFCCs are computed from a power spectrum (power of magnitude of 512-sized FFT) on 20ms-long windows with 10ms overlap.…”

“…Therefore, we concatenate visual features and MFCC features into one joint vector. Since visual features are extracted at 25 per seconds (as per the video frame rate) and audio features at 100 per seconds (MFFCs are computed on windows shifted by 10ms), we tried the following ways to combine features: (i) upsample video frame rate to match 100 fps of the audio, (ii) downsample audio features to match video 25 fps [5], and (iii) for a fixed temporal window, combine all features into one vector [3]. Third option allows preserving local temporal context in each resulted feature vector that can be learnt by the classifier.…”

“…Typically, most of the latest approaches [2], [3], [4], [5], [6], [7] for lip-syncing or dubbing detection focus on extracting separate feature sets for audio and video. For audio, melscale frequency cepstral coefficients (MFCC) are usually used, while different visual features, varying from optical flow [3] to features learned with deep neural networks (DNNs) [6], are extracted from the mouth region of a face. The features then undergo some processing before they are fed into a classifier, best performing examples including long short-term memory (LSTM) [4] or convolutional neural networks [7].…”

“…In this paper, building upon related work, we selected a range of approaches suitable for audio-visual inconsistency detection and performed a preliminary study of different feature processing techniques, classifiers, and their parameters on different databases with tampering attacks. For our tampering detection systems, we used MFFCs as audio features [3] and distances between mouth landmarks as visual features (inspired by [8]). We explored different ways to post-process the features, including ways to combine two types of features, reduce the dimensionality of blocks of features with principal component analysis (PCA), and project both modalities into a common space with canonical correspondence analysis (CCA).…”