Multimodal Co-learning: Challenges, applications with datasets, recent advances and future directions

Rahate, Anil; Walambe, Rahee; Ramanna, Sheela; Kotecha, Ketan

doi:10.1016/j.inffus.2021.12.003

Cited by 91 publications

(57 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the past few years, some works have investigated multi-modal data, demonstrating the effectiveness of deep learning for multi-modal data fusion. Several surveys reviewed the progress of multi-view deep learning Baltrušaitis et al 2018;Chen et al 2020c;Summaira et al 2021;Rahate et al 2021;Ramachandram and Taylor 2017;Zhao et al 2017a). discussed some recent researches on deep multi-modal models from two aspects of clustering and classification, focusing on the application of generative adversarial network (GAN) in clustering and cross-modal learning.…”

Section: Introductionmentioning

confidence: 99%

“…Summaira et al (2021) discussed the latest advancements and trends in multi-modal deep learning, and adopted a new fine-grained taxonomy to classify existing multi-modal networks. Rahate et al (2021) reviewed the relevant literature in multi-modal deep learning and categorized multi-modal co-learning from multiple perspectives. These aforementioned surveys are instructive for our MvSD investigation, at the same time, some spatio temporal data (STD) analysis works provide us with application fields and current progress for reference.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep learning on multi-view sequential data: a survey

et al. 2022

View full text Add to dashboard Cite

With the progress of human daily interaction activities and the development of industrial society, a large amount of media data and sensor data become accessible. Humans collect these multi-source data in chronological order, called multi-view sequential data (MvSD). MvSD has numerous potential application domains, including intelligent transportation, climate science, health care, public safety and multimedia, etc. However, as the volume and scale of MvSD increases, the traditional machine learning methods become difficult to withstand such large-scale data, and it is no longer appropriate to use hand-craft features to represent these complex data. In addition, there is no general framework in the process of mining multi-view relationships and integrating multi-view information. In this paper, We first introduce four common data types that constitute MvSD, including point data, sequence data, graph data, and raster data. Then, we summarize the technical challenges of MvSD. Subsequently, we review the recent progress in deep learning technology applied to MvSD. Meanwhile, we discuss how the network represents and learns features of MvSD. Finally, we summarize the applications of MvSD in different domains and give potential research directions.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Deep learning on multi-view sequential data: a survey

et al. 2022

View full text Add to dashboard Cite

show abstract

“…For our system GaitFi, the multimodal machine learning method is feature-level fusion. Co-learning is another popular research topic in the multimodal machine learning domain, which can model resource-poor modalities by leveraging knowledge from other resource-rich modalities [55]. It achieves this capability by using transfer learning and domain adaptation methods [56].…”

Section: Multimodal Machine Learningmentioning

confidence: 99%

GaitFi: Robust Device-Free Human Identification via WiFi and Vision Multimodal Learning

Deng

Yang

Yuan

et al. 2023

IEEE Internet Things J.

View full text Add to dashboard Cite

As an important biomarker for human identification, human gait can be collected at a distance by passive sensors without subject cooperation, which plays an essential role in crime prevention, security detection and other human identification applications. At present, most research works are based on cameras and computer vision techniques to perform gait recognition. However, vision-based methods are not reliable when confronting poor illuminations, leading to degrading performances. In this paper, we propose a novel multimodal gait recognition method, namely GaitFi, which leverages WiFi signals and videos for human identification. In GaitFi, Channel State Information (CSI) that reflects the multi-path propagation of WiFi is collected to capture human gaits, while videos are captured by cameras. To learn robust gait information, we propose a Lightweight Residual Convolution Network (LRCN) as the backbone network, and further propose the two-stream GaitFi by integrating WiFi and vision features for the gait retrieval task. The GaitFi is trained by the triplet loss and classification loss on different levels of features. Extensive experiments are conducted in the real world, which demonstrates that the GaitFi outperforms state-of-the-art gait recognition methods based on single WiFi or camera, achieving 94.2% for human identification tasks of 12 subjects.

show abstract

“…A major advantage of applications with multiple modalities is that they perform better than applications with one modality. This is due to the advancement of deep learning techniques, increased computing infrastructure, and large datasets [21]. The study in [22] represents that a model deployed with multiple modalities perform better over single modality.…”

Section: Introductionmentioning

confidence: 99%

Rice Transformer: A Novel Integrated Management System for Controlling Rice Diseases

Patil

Kumar

2022

IEEE Access

View full text Add to dashboard Cite

Rice disease classification is vital during the cultivation of rice crops. However, rice diseases were initially detected by visual examination from agricultural experts. Later the detection process progressed to automation, which involved images. The images captured lead to a lack of supporting information. The traditional approaches are less accurate when used with real time images. To address this limitation, a novel Rice Transformer is proposed in the paper that merges inputs from agricultural sensors and image data captured from the fields simultaneously. The proposed system consists of two branches: the sensor and image branches. Specifically, the attention approach is employed to extract the features from both modalities. Later, the extracted features are sent to the cross-attention module as input in a crisscross fashion, enhancing the ability to identify the features specific to rice diseases. The extracted features are further pooled, merged, and later passed through the Softmax classifier to classify the rice disease precisely. The dataset collected is a customized dataset with 4200 samples collected on a real-time basis from rice farms. The experiments conducted on the dataset represent that the proposed approach outperforms all the other fusion and attention models considered for comparison in this paper. The ablation analysis and performance metrics are measured to determine the effectiveness of the proposed system. The results achieved are quite promising as the proposed Rice transformer model achieves an accuracy of 97.38% for controlling rice disease.

show abstract

Multimodal Co-learning: Challenges, applications with datasets, recent advances and future directions

Cited by 91 publications

References 54 publications

Deep learning on multi-view sequential data: a survey

Deep learning on multi-view sequential data: a survey

GaitFi: Robust Device-Free Human Identification via WiFi and Vision Multimodal Learning

Rice Transformer: A Novel Integrated Management System for Controlling Rice Diseases

Contact Info

Product

Resources

About