High Dimensional Data Clustering by means of Distributed Dirichlet Process Mixture Models

Abstract

Clustering is a data mining technique intensively used for data analytics, with applications to marketing, security, text/document analysis, or sciences like biology, astronomy, and many more. Dirichlet Process Mixture (DPM) is a model used for multivariate clustering with the advantage of discovering the number of clusters automatically and offering favorable characteristics. However, in the case of high dimensional data, it becomes an important challenge with numerical and theoretical pitfalls. The advantages of DPM come at the price of prohibitive running times, which impair its adoption and makes centralized DPM approaches inefficient, especially with high dimensional data. We propose HD4C (High Dimensional Data Distributed Dirichlet Clustering), a parallel clustering solution that addresses the curse of dimensionality by two means. First it gracefully scales to massive datasets by distributed computing, while remaining DPM-compliant. Second, it performs clustering of high dimensional data such as time series (as a function of time), hyperspectral data (as a function of wavelength) etc. Our experiments, on both synthetic and real world data, illustrate the high performance of our approach.

Massively Distributed Clustering via Dirichlet Process Mixture

“…This is one of the keys in distributed data science systems supporting algorithms often originally designed for centralized environments. Our solutions rely on sufficient statistics, by means of synchronization between machines in terms of proposed clusters at each step [1,2]. We provide an interactive demonstration of these previous results.…”

“…HD4C [2] presents a parallel clustering approach adapted for high dimensional data. Actually, DC-DPM is a solution proposed to this issue when data is multivariate.…”

“…This inner product is mandatory to compute the likelihood and the posterior. To do that, HD4C [2] uses the properties of the Reproducible Kernel Hilbert Spaces (RKHS). We assume that the random variable takes its values in a space of infinite dimension.…”

Clustering of Companies Based on Sustainability Performance using ESG Materiality Approach: Evidence from Indonesia

“…In this research, the clustering of ESG scores uses K-means clustering algorithm analysis with R Studio tools. K-means has been widely used in previous research because it has several advantages, including: (1) it considers a collection of observations (x1, x2,…, xn); and (2) simplicity, fast convergence, and good scalability (Kwedlo and Czochanski, 2019;Meguelati et al, 2019).…”

Creating and Characterising Electricity Load Profiles of Residential Buildings

“…Clustering of time-series data has been an active area of research over the last few decades and many good techniques have been developed (c.f. [2,11,17] for surveys and [13] for some recent work). The challenge in clustering the smart meter data stems from:…”