Parametric nonlinear dimensionality reduction using kernel t-SNE

Abstract

Novel non-parametric dimensionality reduction techniques such as t-distributed stochastic neighbor embedding (t-SNE) lead to a powerful and flexible visualization of high-dimensional data. One drawback of non-parametric techniques is their lack of an explicit out-of-sample extension. In this contribution, we propose an efficient extension of t-SNE to a parametric framework, kernel t-SNE, which preserves the flexibility of basic t-SNE, but enables explicit out-of-sample extensions. We test the ability of kernel t-SNE in comparison to standard t-SNE for benchmark data sets, in particular addressing the generalization ability of the mapping for novel data. In the context of large data sets, this procedure enables us to train a mapping for a fixed size subset only, mapping all data afterwards in linear time. We demonstrate that this technique yields satisfactory results also for large data sets provided missing information due to the small size of the subset is accounted for by auxiliary information such as class labels, which can be integrated into kernel t-SNE based on the Fisher information.

Parametric t-Distributed Stochastic Exemplar-Centered Embedding

“…The MNIST dataset 5 consists of 60,000 training and 10,000 test gray-level 784-dimensional images. The Fashion dataset 6 has the same number of classes, training and test data points as that of MNIST, but is designed to classify 10 fashion products, such as boot, coat, and bag, where each contains a set of pictures taken by professional photographers from different aspects of the product, such as looks from front, back, with model, and in an outfit.…”

“…This small modification has significant benefits. Because z << n, compared to the quadratic computational complexity with respect to n of Equation 6, the objective function in Equation 12 has a significantly reduced computational cost, considering that the number of representative exemplars is often much much smaller than n for real-world large datasets in practice.…”

“…On the MNIST dataset, we show the best training and test error rates of kNN with k ≥ 1 using the two-dimensional embedding generated by different methods in Talbe 4, which consistently shows that dt-see and hot-see significantly outperforms pt-SNE and supports our claims above. Another evaluation metric based on Quality Score was used by a recent method called kernel t-SNE (kt-SNE) [6]. The Quality Score metric computes the k (neighborhood size) nearest neighbors of each data point, respectively, in the high-dimensional space and in the low-dimensional space, and the metric calculates the preserved percentage of the high-dimensional neighborhood in the low-dimensional neighborhood averaged over all test data points as the Quality Score, with respect to different neighborhood size k. In Table 5, we compute the quality scores of different methods on the MNIST test data for preserving their neighborhood on the training data with neighborhood size ranging from 1 to 100.…”

Autonomous Learning of Representations

“…On the contrary, supervised approaches guarantee that the information which is used to represent the data is relevant for the task, suppressing irrelevant aspects and noise. This is particularly relevant if only few data points are available since the problem would be ill-defined without shaping it according to such auxiliary knowledge [19].…”

Γ-stochastic neighbour embedding for feed-forward data visualization

“…It was found that mismatching the neighbourhood distributions allowed for better local clustering, with examples given in the supplementary material of [29]. Further research into t-SNE has allowed for feed-forward mappings using both kernel methods [5,13] and deep belief nets [27]. In addition to this, the impact of different measures of 'closeness' of the observed and visualized neighbourhood distributions is discussed in [6,9].…”