Heterogeneous Graph Learning for Acoustic Event Classification

Shirian, Amir; Ahmadian, Mona; Somandepalli, Krishna; Guha, Tanaya

doi:10.48550/arxiv.2303.02665

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It wasn't until recently that a small amount of research [67,69] introduce the idea of graph construction for modeling audio-visual data, which is exciting. Specifically, by slicing audio and video into segments, each segment can be consider as a node, and the correspondence between them can be consider as edges or relations.…”

Section: Introductionmentioning

confidence: 99%

TMac: Temporal Multi-Modal Graph Learning for Acoustic Event Classification

Liu,

Liang,

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

Audiovisual data is everywhere in this digital age, which raises higher requirements for the deep learning models developed on them. To well handle the information of the multi-modal data is the key to a better audiovisual modal. We observe that these audiovisual data naturally have temporal attributes, such as the time information for each frame in the video. More concretely, such data is inherently multi-modal according to both audio and visual cues, which proceed in a strict chronological order. It indicates that temporal information is important in multi-modal acoustic event modeling for both intra-and inter-modal. However, existing methods deal with each modal feature independently and simply fuse them together, which neglects the mining of temporal relation and thus leads to sub-optimal performance. With this motivation, we propose a Temporal Multi-modal graph learning method for Acoustic event Classification, called TMac, by modeling such temporal information via graph learning techniques. In particular, we construct a temporal graph for each acoustic event, dividing its audio data and video data into multiple segments. Each segment can be considered as a node, and the temporal relationships between nodes can be considered as timestamps on their edges. In this case, we can smoothly capture the dynamic information in intra-modal and inter-modal. Several experiments are conducted to demonstrate

show abstract

Section: Introductionmentioning

confidence: 99%