Robust Semantic Segmentation With Multi-Teacher Knowledge Distillation

Abstract

Recent studies have recently exploited knowledge distillation (KD) technique to address timeconsuming annotation task in semantic segmentation, through which one teacher trained on a single dataset could be leveraged for annotating unlabeled data. However, in this context, knowledge capacity is restricted, and knowledge variety is rare in different conditions, such as cross-model KD, in which the single teacher KD prohibits the student model from distilling information using cross-domain context. To fix this concern, we have looked into learning a lightweight student from a group of teachers. To be more specific, we train five distinct lightweight convolutional neural networks (CNNs) for semantic segmentation on several datasets. Several state-of-the-art augmentation transformations have also been utilized in our training phase. The impacts of such training scenarios are then assessed in terms of student robustness and accuracy. As the main contribution of this paper, our proposed multi-teacher KD paradigm endows the student with the ability to amalgamate and capture a variety of knowledge illustrations from different sources. Results demonstrated that our method outperforms the existing studies on both clean and corrupted data in the semantic segmentation task while benefiting from our proposed score weight system. Experiments validate that our multi-teacher framework results in an improvement of 9% up to 32.18% compared to the singleteacher paradigm. Moreover, it is demonstrated that our paradigm surpasses previous supervised real-time studies in the semantic segmentation challenge.

A Novel Fusion Model of Hand-Crafted Features With Deep Convolutional Neural Networks for Classification of Several Chest Diseases Using X-Ray Images

“…According to several studies [5][6][7][8][9][10][11][12][13][14][15][16], the signs and symptoms of COVID-19 are comparable to those of many different conditions, including normal, LC, ATE, COL, TB, PNET, EDE, PNEU, and PLT. When it came to COVID-19 and other diseases, it was difficult for doctors and healthcare professionals to correctly diagnose and identify them using CXR.…”

“…Using a variety of medical imaging modalities, such as sonography, CXR, MRI, and CT scans, one of the most significant duties in DL [14] is the classification of respiratory system disorders. It has been suggested in a few studies that CXR pictures could be used to find COVID-19, which would save time and effort for those working in the medical field [16][17][18][19][20][21][22][23][24].…”

Multi-Level Residual Feature Fusion Network for Thoracic Disease Classification in Chest X-Ray Images

“…Thus, the lightweight model is one of the areas of focus for researchers. Amirkhani et al [38] trained a lightweight student model using multi-teacher distillation to improve the segmentation performance and robustness of the student model. Mahbub et al [39] designed an easy-to-train and lightweight CNN model that achieved high accuracy in identifying COVID-19.…”

Multi‐domain autonomous driving dataset: Towards enhancing the generalization of the convolutional neural networks in new environments

“…Similar to the main idea of KITTI, the BDD100K dataset has been presented with the goal of multitask learning so that the researchers could respond to the existing challenges in the field of AVs by relying on different image processing approaches such as semantic/instance segmentation, detection of a vehicle's movement line and the identification of the obstacles around a moving vehicle. Since the deep learning models need an immense quantity of data for training purposes, many of the AV-related image processing techniques have leaned towards semi-supervised [15] or self-supervised [16] learning in recent years. One of the new datasets that can be used in line with these AV-related machine vision applications is the SODA10M [12] dataset.…”