Time-Continuous Audiovisual Fusion with Recurrence vs Attention for In-The-Wild Affect Recognition

Karas, Vincent; Tellamekala, Mani Kumar; Mallol-Ragolta, Adria; Valstar, Michel; Schuller, Björn W.

doi:10.1109/cvprw56347.2022.00266

Cited by 13 publications

(6 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Attention models for ER: Recently, multimodal transformers with CA showed significant improvement for ER [6,7,8]. Parthasarathy et al [9] explored multimodal transformers, where the CA module is integrated with the self-attention module to obtain the A-V cross-modal feature representa-tions.…”

Section: Related Workmentioning

confidence: 99%

“…Zhang et al [10] proposed a leader-follower attention mechanism by considering the visual modality as the primary channel, while the audio modality is used as a supplementary channel to boost visual performance. Karas et al [6] and Meng et al [8] showed improvement in fusion performance by exploring a set of fusion models based on LSTMs and transformers. Zhou et al [7] explored temporal convolutional networks (TCNs) for individual modalities, whereas Zhang et al [11] exploited masked auto-encoders for visual modality.…”

Section: Related Workmentioning

confidence: 99%

“…Zhou et al [7] explored temporal convolutional networks (TCNs) for individual modalities, whereas Zhang et al [11] exploited masked auto-encoders for visual modality. However, most of these methods [7,11,6,8] rely on a naive fusion approach or ensemble-based fusion using transformers and LSTMs. Unlike these approaches, Praveen et al [2] proposed a CA model to effectively leverage complementary relationships by allowing the modalities to interact with each other.…”

Section: Related Workmentioning

confidence: 99%

“…Parthasarathy et al [16] further extended their idea by employing the cross-modal attention in a bidirectional fashion, where audio modality is used to attend to visual modality and vice-versa. Karas et al [40] provided a comprehensive evaluation of the fusion models based on self-attention, cross-attention and LSTMs for DER. Zhang et al [41] proposed leader-follower attention, where audio, visual and text modalities are combined to obtain the attended feature using the modality-wise attention scores, which is further concatenated to visual features for final predictions.…”

Section: Attention Models For Multimodal Emotion Recognitionmentioning

confidence: 99%

See 3 more Smart Citations

A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition

Rajasekar¹,

Melo²,

Ullah³

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

In video-based emotion recognition, audio and visual modalities are often expected to have a complementary relationship, which is widely explored using cross-attention. However, they may also exhibit weak complementary relationships, resulting in poor representations of audio-visual features, thus degrading the performance of the system. To address this issue, we propose Dynamic Cross-Attention (DCA) that can dynamically select cross-attended or unattended features on the fly based on their strong or weak complementary relationship with each other, respectively. Specifically, a simple yet efficient gating layer is designed to evaluate the contribution of the cross-attention mechanism and choose crossattended features only when they exhibit a strong complementary relationship, otherwise unattended features. We evaluate the performance of the proposed approach on the challenging RECOLA and Aff-Wild2 datasets. We also compare the proposed approach with other variants of cross-attention and show that the proposed model consistently improves the performance on both datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Attention Models For Multimodal Emotion Recognitionmentioning

confidence: 99%

See 2 more Smart Citations

A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition

Rajasekar¹,

Melo²,

Ullah³

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

show abstract

“…Media audiovisual mengandung dua unsur pembangun, yaitu gambar dan suara yang menghasilkan objek terlihat hidup dan menarik saat ditayangkan. Dengan bantuan media audiovisual (video), siswa dapat lebih mengapresiasi pembelajaran secara konkret (Rahma, dkk, 2015:3) (Fitri et al, 2020;Karas et al, 2022).…”

unclassified

Pembelajaran Bahasa dan Sastra indonesia yang Kreatif Berbasis Media Audiovisual

Asril¹

2022

Preprint

View full text Add to dashboard Cite

Zaman sekarang sudah didominasi dengan teknologi, mengikuti perkembangan zaman tersebut, masyarakat sekarang banyak menggunakan media sosial untuk membagikan berbagai kegiatan mereka.Maka dari itu, Sistem pembelajaran juga harus dikembangkan, seperti menggunakan media audiovisual untuk pembelajaran. Media audiovisual adalah media yang memiliki unsur suara dan juga unsur gambar.Penelitian ini menggunakan metode survei. Instrumen yang digunakan adalah lembar angket atau kuesioner yang dikembangkan melalui formulir Google dan disebarkan secara online melalui grup media sosial WhatsApp. Karakteristik mahasiswa yang diteliti berkisar antara usia 18-23 tahun. Data angket dianalisis dengan mendeskripsikan temuan-temuan yang diperoleh yang dideskripsikan melalui statistik deskriptif (Ramadhan, Atmazaki, et al., 2022). Hasil persentase data survei diperoleh dari 51 responden mahasiwa.

show abstract

ABAW: Learning from Synthetic Data & Multi-task Learning Challenges

Kollias

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Time-Continuous Audiovisual Fusion with Recurrence vs Attention for In-The-Wild Affect Recognition

Cited by 13 publications

References 33 publications

A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition

A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition

Pembelajaran Bahasa dan Sastra indonesia yang Kreatif Berbasis Media Audiovisual

ABAW: Learning from Synthetic Data & Multi-task Learning Challenges

Contact Info

Product

Resources

About