Dynamic Emotion Modeling With Learnable Graphs and Graph Inception Network

Shirian, Amir; Tripathi, Subarna; Guha, Tanaya

doi:10.1109/tmm.2021.3059169

Cited by 34 publications

(15 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, all these pooling functions treat adjacent nodes with equal importance, which may not be optimal. To this end and following [29], we propose to learn a pooling function Ψ that combines the node embeddings from the K-th layer to produce an embedding for the entire graph. The pooling layer for each modality is thus defined as follows:…”

Section: Audio Graphmentioning

confidence: 99%

Visually-aware Acoustic Event Detection using Heterogeneous Graphs

SHIRIAN¹,

Somandepalli²,

Sánchez³

et al. 2022

Interspeech 2022

Self Cite

View full text Add to dashboard Cite

Perception of auditory events is inherently multimodal relying on both audio and visual cues. A large number of existing multimodal approaches process each modality using modality-specific models and then fuse the embeddings to encode the joint information. In contrast, we employ heterogeneous graphs to explicitly capture the spatial and temporal relationships between the modalities and represent detailed information about the underlying signal. Using heterogeneous graph approaches to address the task of visually-aware acoustic event classification, which serves as a compact, efficient and scalable way to represent data in the form of graphs. Through heterogeneous graphs, we show efficiently modelling of intra-and inter-modality relationships both at spatial and temporal scales. Our model can easily be adapted to different scales of events through relevant hyperparameters. Experiments on AudioSet, a large benchmark, shows that our model achieves state-of-the-art performance. Our code is available at github.com/AmirSh15/VAED HeterGraph

show abstract

Section: Audio Graphmentioning

confidence: 99%

Visually-aware Acoustic Event Detection using Heterogeneous Graphs

SHIRIAN¹,

Somandepalli²,

Sánchez³

et al. 2022

Interspeech 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…This work used a simple cycle and line graph to describe a given audio data sample. A follow-up work generalised such graph representation of audio to a learnable graph structure [19]. A graphbased neural network was utilised to capture the relationships within various speech segments of speakers in a conversation for speech emotion classification [20].…”

Section: B Graph Neural Network In Audiomentioning

confidence: 99%

Self-supervised Graphs for Audio Representation Learning with Limited Labeled Data

Shirian¹,

Somandepalli²,

Guha³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Large scale databases with high-quality manual annotations are scarce in audio domain. We thus explore a self-supervised graph approach to learning audio representations from highly limited labeled data. Considering each audio sample as a graph node, we propose a subgraph-based framework with novel self-supervision tasks that can learn effective audio representations. During training, subgraphs are constructed by sampling the entire pool of available training data to exploit the relationship between the labeled and unlabeled audio samples. During inference, we use random edges to alleviate the overhead of graph construction. We evaluate our model on three benchmark audio databases, and two tasks: acoustic event detection and speech emotion recognition. Our semi-supervised model performs better or on par with fully supervised models and outperforms several competitive existing models. Our model is compact (240k parameters), and can produce generalized audio representations that are robust to different types of signal noise.

show abstract

“…This work used a simple cycle and line graph to describe a given audio data sample. A follow-up work generalised such graph representation of audio to a learnable graph structure [22]. A graphbased neural network was utilised to capture the relationships within various speech segments of speakers in a conversation for speech emotion classification [23].…”

Section: B Graph Neural Network In Audiomentioning

confidence: 99%