Mixture-of-Experts Variational Autoencoder for clustering and generating from similarity-based representations on single cell data

Abstract

Clustering high-dimensional data, such as images or biological measurements, is a long-standing problem and has been studied extensively. Recently, Deep Clustering has gained popularity due to its flexibility in fitting the specific peculiarities of complex data. Here we introduce the Mixture-of-Experts Similarity Variational Autoencoder (MoE-Sim-VAE), a novel generative clustering model. The model can learn multi-modal distributions of high-dimensional data and use these to generate realistic data with high efficacy and efficiency. MoE-Sim-VAE is based on a Variational Autoencoder (VAE), where the decoder consists of a Mixture-of-Experts (MoE) architecture. This specific architecture allows for various modes of the data to be automatically learned by means of the experts. Additionally, we encourage the lower dimensional latent representation of our model to follow a Gaussian mixture distribution and to accurately represent the similarities between the data points. We assess the performance of our model on the MNIST benchmark data set and challenging real-world tasks of clustering mouse organs from single-cell RNA-sequencing measurements and defining cell subpopulations from mass cytometry (CyTOF) measurements on hundreds of different datasets. MoE-Sim-VAE exhibits superior clustering performance on all these tasks in comparison to the baselines as well as competitor methods.

Generalized Probabilistic U-Net for medical image segementation

“…Any arbitrary (continuous) distribution can be modelled by using a mixture of a sufficient number of Gaussians, with appropriate mixture weights [3]. Mixtures of Gaussians have been used to model posterior distributions in variational autoencoders for semi-supervised classification [19] and clustering [8], [16].…”

Deformability Cytometry Clustering with Variational Autoencoders

“…VAEs reduces the dimensionality of input data to arbitrary dimensions 16 and has been previously used for clustering cell data. 8,9…”

“…These issues motivated us to look into deep learning models for clustering, namely a variational autoencoder (VAE), which has previously shown success in clustering cell data. 8,9…”

“…These issues motivated us to look into deep learning models for clustering, namely a variational autoencoder (VAE), which has previously shown success in clustering cell data. 8,9 Here, a novel unsupervised clustering algorithm using a VAE for deformability cytometry was developed, which is referred to as DeepDeform. A VAE alone was tested first as a benchmark of clustering performance.…”

A Transformer-based Framework for Multivariate Time Series Representation Learning

“…Unsupervised learning for multivariate time series: Recent work on unsupervised learning for multivariate time series has predominantly employed autoencoders, trained with an input reconstruction objective and implemented either as Multi-Layer Perceptrons (MLP) or RNN (most commonly, LSTM) networks. As interesting variations of the former, Kopf et al (2019) and additionally incorporated Variational Autoencoding into this approach, but focused on clustering and the visualization of shifting sample topology with time. As an example of the lat-ter, Malhotra et al (2017) presented a multi-layered RNN sequence-to-sequence autoencoder, while Lyu et al (2018) developed a multi-layered LSTM with an attention mechanism and evaluated both an input reconstruction (autoencoding) as well as a forecasting loss for unsupervised representation learning of Electronic Healthcare Record multivariate time series.…”