ELKI: A Software System for Evaluation of Subspace Clustering Algorithms

Abstract. We propose an original outlier detection schema that detects outliers in varying subspaces of a high dimensional feature space. In particular, for each object in the data set, we explore the axis-parallel subspace spanned by its neighbors and determine how much the object deviates from the neighbors in this subspace. In our experiments, we show that our novel subspace outlier detection is superior to existing fulldimensional approaches and scales well to high dimensional databases.

show abstract

“…All competitors are implemented within the ELKI-framework [6]. We first applied the competing outlier models to several synthetic data sets.…”

Section: Methodsmentioning

confidence: 99%

Outlier Detection in Axis-Parallel Subspaces of High Dimensional Data

Kriegel

Kröger

Schubert

et al. 2009

Lecture Notes in Computer Science

Self Cite

238

167

View full text Add to dashboard Cite

show abstract

“…As the source code was not available to us we used our own implementation of SURFING following the algorithm presented in [9]. For CLIQUE we used the ELKI 4 platform [2]. We used several synthetic datasets and two astronomical datasets for this purpose.…”

Section: Experiments and Resultsmentioning

confidence: 99%

Finding and visualizing relevant subspaces for clustering high-dimensional astronomical data using connected morphological operators

Ferdosi¹,

Buddelmeijer²,

Trager³

et al. 2010

2010 IEEE Symposium on Visual Analytics Science and Technology

View full text Add to dashboard Cite

Data sets in astronomy are growing to enormous sizes. Modern astronomical surveys provide not only image data but also catalogues of millions of objects (stars, galaxies), each object with hundreds of associated parameters. Exploration of this very high-dimensional data space poses a huge challenge. Subspace clustering is one among several approaches which have been proposed for this purpose in recent years. However, many clustering algorithms require the user to set a large number of parameters without any guidelines. Some methods also do not provide a concise summary of the datasets, or, if they do, they lack additional important information such as the number of clusters present or the significance of the clusters. In this paper, we propose a method for ranking subspaces for clustering which overcomes many of the above limitations. First we carry out a transformation from parametric space to discrete image space where the data are represented by a grid-based density field. Then we apply so-called connected morphological operators on this density field of astronomical objects that provides visual support for the analysis of the important subspaces. Clusters in subspaces correspond to high-intensity regions in the density image. The importance of a cluster is measured by a new quality criterion based on the dynamics of local maxima of the density. Connected operators are able to extract such regions with an indication of the number of clusters present. The subspaces are visualized during computation of the quality measure, so that the user can interact with the system to improve the results. In the result stage, we use three visualization toolkits linked within a graphical user interface so that the user can perform an in-depth exploration of the ranked subspaces. Evaluation based on synthetic as well as real astronomical datasets demonstrates the power of the new method. We recover various known astronomical relations directly from the data with little or no a priori assumptions. Hence, our method holds good prospects for discovering new relations as well.

show abstract

“…We automated the analysis of each final frame using the ELKI [35] implementation of the OP-TICS algorithm. Note that in contrast to our simulations, where we knew the number of clusters and could therefore leverage k-means clustering, we assume here that this number is unknown and, in the case of very complex datasets, unknowable.…”

Section: In Action: a Recombination Event In Staphylococcus Aureusmentioning

confidence: 99%

Clusterflock: A Flocking Algorithm for Isolating Congruent Phylogenomic Datasets

Narechania

Baker

DeSalle

et al. 2016

Preprint

View full text Add to dashboard Cite

Background: Collective animal behavior, such as the flocking of birds or the shoaling of fish, has inspired a class of algorithms designed to optimize distance-based clusters in various applications, including document analysis and DNA microarrays. In a flocking model, individual agents respond only to their immediate environment and move according to a few simple rules. After several iterations the agents self-organize, and clusters emerge without the need for partitional seeds. In addition to its unsupervised nature, flocking offers several computational advantages, including the potential to reduce the number of required comparisons. Findings: In the tool presented here, Clusterflock, we have implemented a flocking algorithm designed to locate groups (flocks) of orthologous gene families (OGFs) that share an evolutionary history. Pairwise distances that measure phylogenetic incongruence between OGFs guide flock formation. We tested this approach on several simulated datasets by varying the number of underlying topologies, the proportion of missing data, and evolutionary rates, and show that in datasets containing high levels of missing data and rate heterogeneity, Clusterflock outperforms other well-established clustering techniques. We also verified its utility on a known, large-scale recombination event in Staphylococcus aureus. By isolating sets of OGFs with divergent phylogenetic signals, we were able to pinpoint the recombined region without forcing a pre-determined number of groupings or defining a pre-determined incongruence threshold. Conclusions:Clusterflock is an open-source tool that can be used to discover horizontally transferred genes, recombined areas of chromosomes, and the phylogenetic 'core' of a genome. Although we used it here in an evolutionary context, it is generalizable to any clustering problem. Users can write extensions to calculate any distance metric on the unit interval, and can use these distances to 'flock' any type of data.

show abstract

ELKI: A Software System for Evaluation of Subspace Clustering Algorithms

Cited by 68 publications

References 25 publications

Outlier Detection in Axis-Parallel Subspaces of High Dimensional Data

Outlier Detection in Axis-Parallel Subspaces of High Dimensional Data

Finding and visualizing relevant subspaces for clustering high-dimensional astronomical data using connected morphological operators

Clusterflock: A Flocking Algorithm for Isolating Congruent Phylogenomic Datasets

Contact Info

Product

Resources

About