Quantitative Evaluation of Time‐Dependent Multidimensional Projection Techniques

Abstract

Dimensionality reduction methods are an essential tool for multidimensional data analysis, and many interesting processes can be studied as time‐dependent multivariate datasets. There are, however, few studies and proposals that leverage on the concise power of expression of projections in the context of dynamic/temporal data. In this paper, we aim at providing an approach to assess projection techniques for dynamic data and understand the relationship between visual quality and stability. Our approach relies on an experimental setup that consists of existing techniques designed for time‐dependent data and new variations of static methods. To support the evaluation of these techniques, we provide a collection of datasets that has a wide variety of traits that encode dynamic patterns, as well as a set of spatial and temporal stability metrics that assess the quality of the layouts. We present an evaluation of 9 methods, 10 datasets, and 12 quality metrics, and elect the best‐suited methods for projecting time‐dependent multivariate data, exploring the design choices and characteristics of each method. Additional results can be found in the online benchmark repository. We designed our evaluation pipeline and benchmark specifically to be a live resource, open to all researchers who can further add their favorite datasets and techniques at any point in the future.

Visual Analytics for Machine Learning: Computing and Leveraging Decision Boundary Maps

“…The addedvalue of this is obvious when computing projections, which should be stable with respect to small changes in the input data, to maintain the user's mental map. A detailed discussion of projections stability has recently been proposed by Vernier et al [157]. Van der Maaten has recognized this earlier, and proposed a modi cation of the t-SNE method to behave parametrically, using deep learning [81].…”

Learning Multidimensional Projections with Neural Networks

“…Three notable recent benchmarks are Espadoto et al [56] (18 datasets, 44 projection techniques, and 7 metrics. See Chapter 3); Vernier et al [209] (focus in dynamic DR -10 datasets, 11 techniques, 12 metrics); and SmallVis [139] (focus on t-SNE, UMAP, and LargeVis [198]).…”

“…These implementations, and their main functional and non-functional aspects, are listed in Table 7. Additional functional aspects of these techniques, such as complexity and quality are provided in recent surveys [56,209].…”

“…For this, benchmarks are needed. To our knowledge, only three such benchmarks exist in DR landscape [56,139,209]. Yet, such benchmarks are never complete, so they need to be extended with additional DR implementations, datasets, and metrics.…”

“…Importantly, by a benchmark, we do not understand here just a 'passive' collection of high-dimensional datasets to battle-test DR techniques, but rather a runnable software system that lets one select datasets, DR technique implementations (and their parameter values), metrics, execute them, and visually inspect the results in an easy and highly automated way. The only two benchmarks that approach this de nition are [56] (for static DR techniques) and [209] (for dynamic DR techniques). However, these benchmarks have their own limitations.…”