Selection Bias Tracking and Detailed Subset Comparison for High-Dimensional Data

Abstract

Fig. 1. Exploratory cohort selection in high-dimensional datasets can lead to selection bias-unintended side-effects in variable distributions-that may go unnoticed by the user. Our selection bias tracking system and detailed cohort comparison visualizations, deployed in a medical temporal event sequence visual analytics tool, include (a) a cohort provenance tree to keep track of created cohorts and indicate when selection bias may have occurred, (b-d) a suite of high-dimensional cohort comparison visualizations that employ hierarchical aggregation to display the differences between two cohorts in detail, and (e) data-type dependent comparisons of individual variable distributions.Abstract-The collection of large, complex datasets has become common across a wide variety of domains. Visual analytics tools increasingly play a key role in exploring and answering complex questions about these large datasets. However, many visualizations are not designed to concurrently visualize the large number of dimensions present in complex datasets (e.g. tens of thousands of distinct codes in an electronic health record system). This fact, combined with the ability of many visual analytics systems to enable rapid, ad-hoc specification of groups, or cohorts, of individuals based on a small subset of visualized dimensions, leads to the possibility of introducing selection bias-when the user creates a cohort based on a specified set of dimensions, differences across many other unseen dimensions may also be introduced. These unintended side effects may result in the cohort no longer being representative of the larger population intended to be studied, which can negatively affect the validity of subsequent analyses. We present techniques for selection bias tracking and visualization that can be incorporated into high-dimensional exploratory visual analytics systems, with a focus on medical data with existing data hierarchies. These techniques include: (1) tree-based cohort provenance and visualization, including a user-specified baseline cohort that all other cohorts are compared against, and visual encoding of cohort "drift", which indicates where selection bias may have occurred, and (2) a set of visualizations, including a novel icicle-plot based visualization, to compare in detail the per-dimension differences between the baseline and a user-specified focus cohort. These techniques are integrated into a medical temporal event sequence visual analytics tool. We present example use cases and report findings from domain expert user interviews.

The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations

“…All these interactions enable the selection of particular sentences and then further exploration of the content with suggestions stemming from the system side. Abstract/elaborate is another interaction technique found in, e.g., Borland et al [BWZ*20] and can be interpreted as different granularities that the visualization allows users to explore the data. Sevastjanova et al [SBE*18] argue that a combination of visualization and verbalization methods is advantageous for generating wide and versatile insights into the structure and decision‐making processes of ML models.…”

Exploring Multi‐dimensional Data via Subset Embedding

“…Gratzl et al [GGL ∗ 14] propose Domino that supports the flexible exploration of subsets and their relationships. Borland et al [BWZ ∗ 19] propose a visual analytics method to unbiasedly selecting a representative subset for a large dataset. Gotz et al [GZW ∗ 19] propose a method for interactively determining the most informative event subset in a specific analysis context.…”

Explaining artificial intelligence with visual analytics in healthcare

“…We classified Similarity measures like cosine, Jaccard and Hellinger distance together with clustering, because they are often used to group data. For example, Borland et al (2020;Figure 5d) hierarchically aggregate similar events in an icicle plot to track selection bias in patient cohorts; Barlowe et al (2013;Figure 2b) rank protein flexibility plots by similarity to a target.…”

“…( 2) Collapsing components removes visual clutter, for example lines in line graphs (Afzal et al, 2011) or parallel coordinates (Huang et al, 2019), identical rows in matrices (Dang et al, 2015), or similar points in scatter plots (Kwon et al, 2018). Conversely, expanding components shows extra information like individual lines instead of ribbons in parallel sets (Kwon et al, 2018), subsequences of sequential health records (Malik et al, 2015), or groups in icicle plots (Borland et al, 2020). ( 3) Zooming enlarges a visualization (e.g., Males et al, 2020;Kumar et al, 2015, Figure 3a).…”