Weakly Supervised Representation Learning for Audio-Visual Scene Analysis

Parekh, Sanjeel; Essid, Slim; Ozerov, Alexey; Duong, Ngoc Q. K.; Pérez, Patrick; Richard, Gaël

doi:10.1109/taslp.2019.2957889

Cited by 27 publications

(26 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further post-challenge research was carried by some other authors. For example in [44], the authors used an audio-visual approach to match co-occurrences of images and sounds to locate the target sound from the weak labels. The authors in [45] used a multi-level attention model to focus on the target sound indicated by the weakly labels.…”

Section: Systemmentioning

confidence: 99%

Sound Event Detection in the DCASE 2017 Challenge

Mesaros

Diment

Elizalde

et al. 2019

IEEE/ACM Trans. Audio Speech Lang. Process.

114

View full text Add to dashboard Cite

Each edition of the challenge on Detection and Classification of Acoustic Scenes and Events (DCASE) contained several tasks involving sound event detection in different setups. DCASE 2017 presented participants with three such tasks, each having specific datasets and detection requirements: Task 2, in which target sound events were very rare in both training and testing data, Task 3 having overlapping events annotated in real-life audio, and Task 4, in which only weakly-labeled data was available for training. In this paper, we present the three tasks, including the datasets and baseline systems, and analyze the challenge entries for each task. We observe the popularity of methods using deep neural networks, and the still widely used mel frequency based representations, with only few approaches standing out as radically different. Analysis of the systems behavior reveals that task-specific optimization has a big role in producing good performance; however, often this optimization closely follows the ranking metric, and its maximization/minimization does not result in universally good performance. We also introduce the calculation of confidence intervals based on a jackknife resampling procedure, to perform statistical analysis of the challenge results. The analysis indicates that while the 95% confidence intervals for many systems overlap, there are significant difference in performance between the top systems and the baseline for all tasks.

show abstract

Section: Systemmentioning

confidence: 99%

Sound Event Detection in the DCASE 2017 Challenge

Mesaros

Diment

Elizalde

et al. 2019

IEEE/ACM Trans. Audio Speech Lang. Process.

114

View full text Add to dashboard Cite

show abstract

“…• A (TSP): temporal segment proposals, • A (NCP): NMF component proposals, • A (TSP, NCP): all TSPs and NCPs are put together into the same bag and fed to the audio network. While systems using only TSP give state-of-the-art results [1], they serve as a strong baseline for establishing the usefulness of NCPs in classification. For source enhancement we compare with the following NMF related methods:…”

Section: Setupmentioning

confidence: 99%

Identify, Locate and Separate: Audio-Visual Object Extraction in Large Video Collections Using Weak Supervision

Parekh

Ozerov²,

Essid

et al. 2019

2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)

Self Cite

View full text Add to dashboard Cite

We tackle the problem of audio-visual scene analysis for weaklylabeled data. To this end, we build upon our previous audio-visual representation learning framework to perform object classification in noisy acoustic environments and integrate audio source enhancement capability. This is made possible by a novel use of non-negative matrix factorization for the audio modality. Our approach is founded on the multiple instance learning paradigm. Its effectiveness is established through experiments over a challenging dataset of music instrument performance videos. We also show encouraging visual object localization results.

show abstract

“…Unfortunately, due to availability issues, videos could not be obtained for all of the audio files in the original data set. Ultimately, we ended up creating a slightly reduced collection of audio and video files consisting of 48883 training, 465 validation and 1050 test samples: about 5% of samples of the original data was discarded in the process [14].…”

Section: Data Setmentioning

confidence: 99%

“…The micro-averaged F1 score achieved by this model is reported in Table 3. Table 3: F1 score of prior audiovisual classification model model F1 score two-stream audiovisual neural network [14] 64.2% The multimodal audiovisual transformers strongly improve upon the two-stream audiovisual neural network. As both of these types of models use nearly the same data and employ similar externally pretrained embeddings, this truly illustrates the power of the transformer architecture for the task at hand.…”

Section: 14mentioning

confidence: 99%

Audiovisual Transformer Architectures for Large-Scale Classification and Synchronization of Weakly Labeled Audio Events

Boes

hamme

2019

Proceedings of the 27th ACM International Conference on Multimedia

View full text Add to dashboard Cite

We tackle the task of environmental event classification by drawing inspiration from the transformer neural network architecture used in machine translation. We modify this attention-based feedforward structure in such a way that allows the resulting model to use audio as well as video to compute sound event predictions.We perform extensive experiments with these adapted transformers on an audiovisual data set, obtained by appending relevant visual information to an existing large-scale weakly labeled audio collection. The employed multi-label data contains clip-level annotation indicating the presence or absence of 17 classes of environmental sounds, and does not include temporal information.We show that the proposed modified transformers strongly improve upon previously introduced models and in fact achieve stateof-the-art results. We also make a compelling case for devoting more attention to research in multimodal audiovisual classification by proving the usefulness of visual information for the task at hand, namely audio event recognition.In addition, we visualize internal attention patterns of the audiovisual transformers and in doing so demonstrate their potential for performing multimodal synchronization. CCS CONCEPTS• Computing methodologies → Neural networks.

show abstract

Weakly Supervised Representation Learning for Audio-Visual Scene Analysis

Cited by 27 publications

References 60 publications

Sound Event Detection in the DCASE 2017 Challenge

Sound Event Detection in the DCASE 2017 Challenge

Identify, Locate and Separate: Audio-Visual Object Extraction in Large Video Collections Using Weak Supervision

Audiovisual Transformer Architectures for Large-Scale Classification and Synchronization of Weakly Labeled Audio Events

Contact Info

Product

Resources

About