Relation-guided acoustic scene classification aided with event embeddings

Abstract

In real life, acoustic scenes and audio events are naturally correlated. Humans instinctively rely on fine-grained audio events as well as the overall sound characteristics to distinguish diverse acoustic scenes. Yet, most previous approaches treat acoustic scene classification (ASC) and audio event classification (AEC) as two independent tasks. A few studies on scene and event joint classification either use synthetic audio datasets that hardly match the real world, or simply use the multi-task framework to perform two tasks at the same time. Neither of these two ways makes full use of the implicit and inherent relation between fine-grained events and coarse-grained scenes. To this end, this paper proposes a relation-guided ASC (RGASC) model to further exploit and coordinate the scene-event relation for the mutual benefit of scene and event recognition. The TUT Urban Acoustic Scenes 2018 dataset (TUT2018) is annotated with pseudo labels of events by a simple and efficient audiorelated pre-trained model PANN, which is one of the state-ofthe-art AEC models. Then, a prior scene-event relation matrix is defined as the average probability of the presence of each event type in each scene class. Finally, the two-tower RGASC model is jointly trained on the real-life dataset TUT2018 for both scene and event classification. The following results are achieved. 1) RGASC effectively coordinates the true information of coarsegrained scenes and the pseudo information of fine-grained events.2) The event embeddings learned from pseudo labels under the guidance of prior scene-event relations help reduce the confusion between similar acoustic scenes. 3) Compared with other (nonensemble) methods, RGASC improves the scene classification accuracy on the real-life dataset.

Artificial intelligence-based collaborative acoustic scene and event classification to support urban soundscape analysis and classification

“…However, real-life acoustic scenes and audio events naturally have implicit relationships with each other, and these relationships between scenes and events are not fully explored and used in Framework2. To this end, we recently proposed a new Relation-Guided ASC (RGASC) model to further exploit and coordinate the scene-event relation for the mutual benefit of scene and event recognition [19].…”

“…Inspired by the idea of RGASC [19], to jointly classify the auditory scene and label sound events, the collaborative scene-event classification (CSEC) framework is introduced. It uniquely extends current practice models by introducing a learnable coupling matrix between a scene classification branch that solely relies on basic acoustic features and an event identification branch that solely relies on acoustic features, to assist the acoustic scene classification.…”

Audio-visual scene classification via contrastive event-object alignment and semantic-based fusion

“…The encoder layers are followed by a linear embedding layer with ReLU activation that maps the high-level representations of audio events to labels for classification. As the audio branch performs multilabel classification, binary cross-entropy (BCE) loss is used [9]. Denote the output of audio branch as ŷe ∈ R Ce , and the corresponding label as y e ∈ R Ce , the loss can be defined as:…”

“…Then, the event and object embeddings are concatenated together to form audio-visual semantic embeddings, and the fusion layer with ReLU activation maps the audio-visual embeddings into scene classes. As scene classification performs single-label multiclass classification, cross-entropy loss [9] is used between the output ŷs ∈ R Cs and the scene label y s ∈ R Cs ,…”

“…For example, in a park scene, birds singing and dogs barking are more likely to occur than keyboard sounds, where the later are often found in the office scenes. To exploit the inherent relationships between the coarse-grained scenes and corresponding fine-grained events, relation-guided ASC [9] coordinates scene-event relationships for the mutual benefit of scene and event recognition. For ISC models [10] [11], the input is usually an image or image sequence [12], and then the scene is recognized based on the rich object information, spatial layout information, as well as the relationship between the objects and layouts.…”