Multimodal Memorability: Modeling Effects of Semantics and Decay on Video Memorability

The Influence of Audio on Video Memorability with an Audio Gestalt Regulated Video Memorability System

“…Our audio gestalt based system was the best performing approach, achieving a Spearman score of 0.6181. To evaluate the effectiveness of our approach, we compare against the Memento10k benchmark scores [32]. From Table IV we can see that our audio gestalt based approach outperforms all other approaches except SemanticMemNet [32]-the model introduced alongside the Memento10k dataset.…”

“…However, one of our audio-based submissions [31] did achieve the best-in-class results from among all participants for longterm memorability predictions. In this paper, we evaluate the utility of including the audio modality in short-term video "recognition memorability" prediction, and assess our gestalt based video memorability prediction system by benchmarking it on the Memento10k dataset [32], comparing it to state-of-the-art solutions. Our contributions are two-fold: A) we assess the influence of the audio modality on video memorability B) we propose a multimodal deep learning-based late fusion system that uses audio gestalt to estimate the influence of the audio modality on overall video memorability, and selectively leverage audio features accordingly.…”

“…For auditory features, we train a network to predict a video's recognition memorability from audio spectrogramsour Spectrogram model. We extracted Mel-frequency cepstral coefficients (n fft:2048, hop length:256, n mels:128) from the the 6,890 Memento10k [32] training videos with audio, and stacked them with their delta coefficients in order to create three channel spectrogram images. These spectrogram images are then used to train an ImageNet-pretrained xRes-Net34 model for 15 epochs; with a max learning rate of 1e-2; and weight decay of 1e-3 to predict audio recognition memorability.…”

“…We evaluate the extent to which static visual features contribute to video recognition memorability by training a network to predict a video's recognition memorability from a single frame-our Frame model. We train an ImageNet-pretrained xResNet50 to predict image recognition memorability by first training on the LaMem dataset [3] for 50 epochs; with a maximum learning rate of 3e-2; and weight decay of 1e-2, and then fine-tuning on the 6,890 Memento10k [32] training videos which have audio with the same hyperparameters. At test time, a video's recognition memorability score is calculated by averaging predictions of the first, middle, and last frame.…”

“…A Wiki-103-pretrained language model is fine-tuned on the first 300,000 captions from Google's Conceptual Captions dataset [42] for a total of 10 epochs with a dropout multiplier of 0.5 and max learning rate of 2e-3, resulting in a final language model accuracy of 37%. The encoder from that model is re-used in another model of the same architecture, but trained on captions from the 6,890 Memento10k [32] training videos with audio, for a total of 15 epochs with a dropout multiplier of 0.8 and a max learning rate of 1e-3, to predict recognition memorability scores, rather than the next word in a sentence. An additional network is trained the same way, but fine-tuned on captions that are augmented with audio tags extracted using the PANNs [38] network-our Augmented Caption model.…”