Method evaluation, parameterization, and result validation in unsupervised data mining: A critical survey

Abstract

Machine Learning (ML) and Data Mining (DM) build tools intended to help users solve data‐related problems that are infeasible for “unaugmented” humans. Tools need manuals, however, and in the case of ML/DM methods, this means guidance with respect to which technique to choose, how to parameterize it, and how to interpret derived results to arrive at knowledge about the phenomena underlying the data. While such information is available in the literature, it has not yet been collected in one place. We survey three types of work for clustering and pattern mining: (1) comparisons of existing techniques, (2) evaluations of different parameterization options and studies providing guidance for setting parameter values, and (3) work comparing mining results with the ground truth. We find that although interesting results exist, as a whole the body of work on these questions is too limited. In addition, we survey recent studies in the field of community detection, as a contrasting example. We argue that an objective obstacle for performing needed studies is a lack of data and survey the state of available data, pointing out certain limitations. As a solution, we propose to augment existing data by artificially generated data, review the state‐of‐the‐art in data generation in unsupervised mining, and identify shortcomings. In more general terms, we call for the development of a true “Data Science” that—based on work in other domains, results in ML, and existing tools—develops needed data generators and builds up the knowledge needed to effectively employ unsupervised mining techniques. This article is categorized under: Fundamental Concepts of Data and Knowledge > Key Design Issues in Data Mining Ensemble Methods > Structure Discovery Internet > Society and Culture Fundamental Concepts of Data and Knowledge > Motivation and Emergence of Data Mining

Validation of cluster analysis results on validation data: A systematic framework

“…There is some work on the so-called benchmarking of clustering methods (Van Mechelen et al, 2018;Zimmermann, 2020). This is different from our approach.…”

Validation of cluster analysis results on validation data: A systematic framework

“…However, when conducting cluster analysis, researchers are confronted with an overwhelming number of existing methods. They must preprocess the data, choose a clustering algorithm, and set parameters, such as the number of clusters (Van Mechelen et al, 2018; Zimmermann, 2020). It is often unclear a priori which choice should be made for the analysis, and even once a choice is made, it may remain unclear how good the quality of the resulting clustering is.…”

“…The phrase “cluster validation” also appears in the literature about benchmarking of clustering methods (Boulesteix & Hatz, 2017; Van Mechelen et al, 2018; Zimmermann, 2020). A benchmarking study is a systematic comparison of different clustering methods on a class of data distributions or datasets.…”

“…But when validation techniques are used as selection tool, it is still an open issue whether the results generalize to new data, and this is addressed by Step 2. The phrase "cluster validation" also appears in the literature about benchmarking of clustering methods (Boulesteix & Hatz, 2017;Van Mechelen et al, 2018;Zimmermann, 2020). A benchmarking study is a systematic comparison of different clustering methods on a class of data distributions or datasets.…”

Over-optimism in benchmark studies and the multiplicity of design and analysis options when interpreting their results

“…Regarding the appropriate design and analysis of benchmark studies, the available literature ranges from general guidelines (Weber et al, 2019;Boulesteix, 2015) and statistical frameworks (Demšar, 2006;Hothorn et al, 2005;Eugster et al, 2012;Boulesteix et al, 2015, all with focus on supervised learning), to recommendations for context-specific benchmarks (e.g. Mangul et al, 2019;Bokulich et al, 2020;Zimmermann, 2020;Kreutz, 2019). However, for many issues relevant in practice (e.g.…”