Unsupervised Multi-Task Learning for 3D Subtomogram Image Alignment, Clustering and Segmentation

Abstract

3D subtomogram image alignment, clustering, and segmentation are vital to macromolecular structure recognition in cryo-electron tomography (cryo-ET). However, acquiring ground-truth labels to train a unified deep learning model that can simultaneously deal with these tasks is unaffordable. To this end, we propose an end-to-end unified multi-task learning framework to simultaneously complete the three tasks, where models are trained in an unsupervised manner without using any labels. In particular, we have three parallel branches. In the alignment branch, we adopt a two-stage training scheme, i.e., self-supervised pretraining and constrained unsupervised training using our proposed skip correlation attention layer and constrained loss. Synchronously, in the clustering branch, the learned deep cluster features are utilized to iteratively cluster subtomograms into groups using pseudo-labels from an image-wise Gaussian Mixture Model (GMM). Meanwhile, in the segmentation branch, we use rough pseudo-labels generated from a voxel-wise GMM as supervision signals, and prior knowledge from humans is utilized to jointly learn how to correct these labels as well as predict reliable segmentation results. Benefiting from the end-to-end unified network architecture, our method achieves overall state-of-the-art performance on both simulated and real subtomogram processing benchmarks.

Towards full-stack deep learning-empowered data processing pipeline for synchrotron tomography experiments

“…In the field of ET, segmentation of the target particle within the imaging area is a significant challenge due to complex cellular structures and the impact of incorporated ice. For DL architectures, common models, like CNN 106 and encoder-decoder, 107 , 108 , 109 , 110 , 111 , 112 , 113 are often used. To segment a single particle, some researchers 106 , 108 combined prior knowledge with these DL models to get more easily trained networks.…”

“…For DL architectures, common models, like CNN 106 and encoder-decoder, 107 , 108 , 109 , 110 , 111 , 112 , 113 are often used. To segment a single particle, some researchers 106 , 108 combined prior knowledge with these DL models to get more easily trained networks. In addition, research on image segmentation using a diffusion model (DM) is being carried out in the medical field.…”