An Analysis of Machine- and Human-Analytics in Classification

Abstract

In this work, we present a study that traces the technical and cognitive processes in two visual analytics applications to a common theoretic model of soft knowledge that may be added into a visual analytics process for constructing a decision-tree model. Both case studies involved the development of classification models based on the "bag of features" approach. Both compared a visual analytics approach using parallel coordinates with a machine-learning approach using information theory. Both found that the visual analytics approach had some advantages over the machine learning approach, especially when sparse datasets were used as the ground truth. We examine various possible factors that may have contributed to such advantages, and collect empirical evidence for supporting the observation and reasoning of these factors. We propose an information-theoretic model as a common theoretic basis to explain the phenomena exhibited in these two case studies. Together we provide interconnected empirical and theoretical evidence to support the usefulness of visual analytics.

Empirically Measuring Soft Knowledge in Visualization

“…The analytical evidence in [TKC17] shows that humans possess a tremendous amount of knowledge that could be deployed in a visualization process. In order to measure such knowledge in a statistically meaningful way, an empirical study has to focus on some particular pieces of knowledge.…”

“…In order to measure such knowledge in a statistically meaningful way, an empirical study has to focus on some particular pieces of knowledge. In this study, we measure human knowledge in the form of three soft models [TKC17]. The first model, M T , is a function that takes the input of a time series alphabet 𝔸 and a clue C T about the contextual type of the application, and divides the alphabet into two groups, one matches C T , and one does not.…”

“…As stated in [Coh95], hypotheses established in exploratory studies are normally tested in observation or manipulation experiments. The recent study by Tam et al [TKC17] is an observation experiment that made naturalistic observations of two analysts’ contributions to the construction of different classification models. They narrowed down the soft knowledge to a number of variables referred to as “observations”, and used information theory to estimate the quantities of such knowledge.…”

“…In [CG16], there was no suggestion as to how to measure the capacity of different soft knowledge. In [TKC17], an attempt was made to estimate soft knowledge analytically. Because the estimated quantities were derived based on the observations of two expert analysts who performed highly complex model development tasks, it would be difficult to confirm the estimated quantities using a laboratory‐based empirical study.…”

The State‐of‐the‐Art in Predictive Visual Analytics

“…The four steps of the predictive analytics pipeline (data preprocessing, feature engineering, model building, and model selection and validation) serve as a basis for defining the PVA pipeline (Figure ). Our definition of the PVA pipeline is further informed by the knowledge discovery process of Pirolli and Card [PC05] and a variety of recent surveys on topics ranging from visual analytics pipelines and frameworks [CT05, KMS*08, WZM*16] to human‐centered machine learning [BL09, SSZ*16, TKC17] to knowledge discovery.…”