Audio Captioning with Composition of Acoustic and Semantic Information

Abstract

Generating audio captions is a new research area that combines audio and natural language processing to create meaningful textual descriptions for audio clips. To address this problem, previous studies mostly use the encoder–decoder-based models without considering semantic information. To fill this gap, we present a novel encoder–decoder architecture using bi-directional Gated Recurrent Units (BiGRU) with audio and semantic embeddings. We extract semantic embedding by obtaining subjects and verbs from the audio clip captions and combine these embedding with audio embedding to feed the BiGRU-based encoder–decoder model. To enable semantic embeddings for the test audios, we introduce a Multilayer Perceptron classifier to predict the semantic embeddings of those clips. We also present exhaustive experiments to show the efficiency of different features and datasets for our proposed model the audio captioning task. To extract audio features, we use the log Mel energy features, VGGish embeddings, and a pretrained audio neural network (PANN) embeddings. Extensive experiments on two audio captioning datasets Clotho and AudioCaps show that our proposed model outperforms state-of-the-art audio captioning models across different evaluation metrics and using the semantic information improves the captioning performance.

Automated Audio Captioning With Topic Modeling

“…The growing presence of publicly available datasets has led to increasing research in the AAC task. Several studies have addressed audio captioning on the Clotho [17]- [19] and AudioCaps [18], [20] datasets.…”

“…A transformer model with keyword estimation is proposed in [4]. [18] improves audio captioning performance by extracting subject-verb keywords from the captions.…”

“…Previous studies have shown the performance of the pretrained acoustic embeddings such as VGGish [33] and pretrained Audio Neural Networks (PANNs) [34] than other representations such as spectrograms, log Mel energies [18], [22]. Thus, we use PANNs as feature extractor.…”

“…For keyword extraction, we use our previous keyword extraction method in [18]. We extract subjects and verbs from the dataset captions.…”

Audio captioning and retrieval with improved cross-modal objectives

“…For instance, [62] used a basic sequence-to-sequence Bi-LSTM [63] model for encoding hidden states of the audio and text. Similarly, GRUs have also been used to encode audio data with less success [11,64] used to apply non-linear transformations to the features extracted by the selfattention mechanism. GPT-2 was trained on a large corpus of text data [21], and its objective was to predict the next word in a sequence given the preceding context.…”