Two-Level Bimodal Association for Audio-Visual Speech Recognition

Lee, Jong Seok; Ebrahimi, Touradj

doi:10.1007/978-3-642-04697-1_13

“…One of the driving factors beyond the adoption of multi-input AI approaches is data availability. As one can easily imagine, video data lead the list as they consist in tuples of images and audio signals 42,43 . However, the list goes beyond video data, and several multi-modal datasets can be found publicly [44][45][46][47] .…”

Section: Multi-input Classificationmentioning

confidence: 99%

MiFL: Multi-Input Neural Networks in Federated Learning

Casella¹,

Riviera²,

Aldinucci³

et al. 2023

Preprint

1

7

0

View full text Add to dashboard Cite

<p>Driven by the Deep Learning (DL) revolution, Artificial intelligence (AI) has become a fundamental tool for many Bio-Medical tasks, including AI-assisted diagnosis. These include analysing and classifying images (2D and 3D), where, for some tasks, DL exhibits superhuman performance. Diagnostic imaging, however, is not the only diagnostic tool. Tabular data, such as personal data, vital signs, and genomic/blood tests, are commonly collected for every patient entering a clinical institution. However, it is rarely considered in DL pipelines, although it carries diagnostic information. The training of DL models requires large datasets, so large that every institution might need more data that should be pooled from different sites. Data pooling generates newfound concerns about data access and movement across other institutions spawning multiple dimensions, such as performance, energy efficiency, privacy, criticality, and security. Federated Learning (FL) is a cooperative learning paradigm aiming at addressing these concerns by moving models instead of data across different institutions. This paper proposes a Federated multi-input model that leverages images and tabular data, providing a proof of concept of the feasibility of multi-input FL architectures. The proposed model was evaluated on two showcases: the prognosis of CoViD-19 disease and the patients’ stratification for Alzheimer’s disease. Results show that enabling multi-input architectures in the FL framework allows for improving the performance regarding both accuracy and generalizability with respect to non-federated models while ensuring security and data protection peculiar to FL.</p>

show abstract

“…If the multimodal data are acquired from the same target, there usually exist cross-modal correlations between the data obtained from different sensors. For example, utilizing correlations between acoustic features and visual cues in speech recognition is beneficial to improve the overall performance [20].…”

Section: Considering Correlations Between Modalitiesmentioning

confidence: 99%

“…Multimodal data have been widely employed in recent decades [18]. One of the most prominent multimodal data is videos, which consist of image frames and audio signals [19,20,21]. In addition, human activity recognition systems usually employ data obtained from multiple sensors, including motion capture system, depth cameras, accelerometers, and microphones [12,13,15].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

EmbraceNet: A robust deep learning architecture for multimodal classification

Choi

¹

,

Lee

²

2019

Self Cite

View full text Add to dashboard Cite

Classification using multimodal data arises in many machine learning applications. It is crucial not only to model cross-modal relationship effectively but also to ensure robustness against loss of part of data or modalities. In this paper, we propose a novel deep learning-based multimodal fusion architecture for classification tasks, which guarantees compatibility with any kind of learning models, deals with cross-modal information carefully, and prevents performance degradation due to partial absence of data. We employ two datasets for multimodal classification tasks, build models based on our architecture and other state-of-the-art models, and analyze their performance on various situations. The results show that our architecture outperforms the other multimodal fusion architectures when some parts of data are not available.

show abstract

“…One of the driving factors beyond the adoption of multi-input AI approaches is data availability. As one can easily imagine, video data lead the list as they consist in tuples of images and audio signals 42,43 . However, the list goes beyond video data, and several multi-modal datasets can be found publicly [44][45][46][47] .…”

Section: Multi-input Classificationmentioning

confidence: 99%

MiFL: Multi-Input Neural Networks in Federated Learning

Casella¹,

Riviera²,

Aldinucci³

et al. 2023

Preprint

0

View full text Add to dashboard Cite

<p>Driven by the Deep Learning (DL) revolution, Artificial intelligence (AI) has become a fundamental tool for many Bio-Medical tasks, including AI-assisted diagnosis. These include analysing and classifying images (2D and 3D), where, for some tasks, DL exhibits superhuman performance. Diagnostic imaging, however, is not the only diagnostic tool. Tabular data, such as personal data, vital signs, and genomic/blood tests, are commonly collected for every patient entering a clinical institution. However, it is rarely considered in DL pipelines, although it carries diagnostic information. The training of DL models requires large datasets, so large that every institution might need more data that should be pooled from different sites. Data pooling generates newfound concerns about data access and movement across other institutions spawning multiple dimensions, such as performance, energy efficiency, privacy, criticality, and security. Federated Learning (FL) is a cooperative learning paradigm aiming at addressing these concerns by moving models instead of data across different institutions. This paper proposes a Federated multi-input model that leverages images and tabular data, providing a proof of concept of the feasibility of multi-input FL architectures. The proposed model was evaluated on two showcases: the prognosis of CoViD-19 disease and the patients’ stratification for Alzheimer’s disease. Results show that enabling multi-input architectures in the FL framework allows for improving the performance regarding both accuracy and generalizability with respect to non-federated models while ensuring security and data protection peculiar to FL.</p>

show abstract

Two-Level Bimodal Association for Audio-Visual Speech Recognition

Cited by 3 publications

References 20 publications

MiFL: Multi-Input Neural Networks in Federated Learning

MiFL: Multi-Input Neural Networks in Federated Learning

EmbraceNet: A robust deep learning architecture for multimodal classification

MiFL: Multi-Input Neural Networks in Federated Learning

Contact Info

Product

Resources

About