Automated Audio Captioning Using Transfer Learning and Reconstruction Latent Space Similarity Regularization

Koh, Andrew; Xue, Fuzhao; Chng, Eng Siong

doi:10.1109/icassp43922.2022.9747676

Cited by 15 publications

(18 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is a plethora of pretrained models available publicly. These pretrained models are often used for transfer learning to another related domain [13], hence there is a need for finetuning. In our case, we find that it is sufficient to simply use these pretrained models as it is without finetuning.…”

Section: Pretrained Embeddingsmentioning

confidence: 99%

Language-Based Audio Retrieval with Converging Tied Layers and Contrastive Loss

Koh¹,

Chng²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we tackle the new Language-Based Audio Retrieval task proposed in DCASE 2022 1 . Firstly, we introduce a simple, scalable architecture which ties both the audio and text encoder together. Secondly, we show that using this architecture along with contrastive loss allows the model to significantly beat the performance of the baseline model. Finally, in addition to having an extremely low training memory requirement, we are able to use pretrained models as it is without needing to finetune them. We test our methods and show that using a combination of our methods beats the baseline scores significantly.

show abstract

Section: Pretrained Embeddingsmentioning

confidence: 99%

Language-Based Audio Retrieval with Converging Tied Layers and Contrastive Loss

Koh¹,

Chng²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For instance, the AT-CNN [8], and the Audio Captioning Transformer [9] pretrains their model encoder on audio tagging as an pretraining task before finetuning their model directly on Automated Audio Captioning. Other authors also use self supervised supplementary objectives [3] or caption retrieval [10] to guide the model training. Self-Critical Sequence Training [11], a reinforcement learning tactic used in Image Captioning, has also been used to optimize the model for Audio Captioning.…”

Section: Model Architectures and Objectivesmentioning

confidence: 99%

“…In this work, we build on the work of [3] and [4] by using their model architectures that has been proven to achieve competitive results. Henceforth, we refer [4] and [3] as System 1 and System 2 respectively. Figure 1 delineates their system architectures.…”

Section: Model Architectures and Objectivesmentioning

confidence: 99%

“…We test Epochal Difficult Captions using two different publicly available systems [3,4] from previous work. Both systems are of the typical encoder-decoder structure.…”

Section: Introductionmentioning

confidence: 99%

“…Both systems are of the typical encoder-decoder structure. The encoder consists of either a single [4] pretrained CNN or both [3] a pretrained CNN and transformer encoder. The decoder used is a 2 layer transformer decoder.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Automated Audio Captioning with Epochal Difficult Captions for Curriculum Learning

Koh¹,

Tiwari²,

Chng³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we propose an algorithm, Epochal Difficult Captions, to supplement the training of any model for the Automated Audio Captioning task. Epochal Difficult Captions is an elegant evolution to the keyword estimation task that previous work have used to train the encoder of the AAC model. Epochal Difficult Captions modifies the target captions based on a curriculum and a difficulty level determined as a function of current epoch. Epochal Difficult Captions can be used with any model architecture and is a lightweight function that does not increase training time. We test our results on three systems and show that using Epochal Difficult Captions consistently improves performance.

show abstract

Audio Captioning Using Semantic Alignment Enhancer

Park,

Chang

2023

2023 8th IEEE International Conference on Network Intelligence and Digital Content (IC-NIDC)

View full text Add to dashboard Cite

Automated Audio Captioning Using Transfer Learning and Reconstruction Latent Space Similarity Regularization

Cited by 15 publications

References 9 publications

Language-Based Audio Retrieval with Converging Tied Layers and Contrastive Loss

Language-Based Audio Retrieval with Converging Tied Layers and Contrastive Loss

Automated Audio Captioning with Epochal Difficult Captions for Curriculum Learning

Audio Captioning Using Semantic Alignment Enhancer

Contact Info

Product

Resources

About