DendroMap: Visual Exploration of Large-Scale Image Datasets for Machine Learning with Treemaps

Bertucci, Donald; Hamid, Md Montaser; Anand, Yashwanthi; Ruangrotsakun, Anita; Tabatabai, Delyar; Perez, Melissa; Kahng, Minsuk

doi:10.48550/arxiv.2205.06935

Cited by 1 publication

(1 citation statement)

References 49 publications

(79 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We envision several alternative ways to present each slice beyond the static grid used in our slice overview. For example, one could design novel interactive workflows that allow users to explore the model's performance on multiple slices simultaneously (Bertucci et al 2022;Cabrera et al 2023), or compare the images within to images outside of each slice. Designing an appropriate visualization that accounts for the cognitive load and biases of the user (Gajos and Chauncey 2017;Cabrera et al 2022) is an important direction for future work.…”

Section: Design Opportunitiesmentioning

confidence: 99%

Where Does My Model Underperform? A Human Evaluation of Slice Discovery Algorithms

Johnson,

Cabrera,

Plumb

et al. 2023

HCOMP

View full text Add to dashboard Cite

Machine learning (ML) models that achieve high average accuracy can still underperform on semantically coherent subsets ("slices") of data. This behavior can have significant societal consequences for the safety or bias of the model in deployment, but identifying these underperforming slices can be difficult in practice, especially in domains where practitioners lack access to group annotations to define coherent subsets of their data. Motivated by these challenges, ML researchers have developed new slice discovery algorithms that aim to group together coherent and high-error subsets of data. However, there has been little evaluation focused on whether these tools help humans form correct hypotheses about where (for which groups) their model underperforms. We conduct a controlled user study (N = 15) where we show 40 slices output by two state-of-the-art slice discovery algorithms to users, and ask them to form hypotheses about an object detection model. Our results provide positive evidence that these tools provide some benefit over a naive baseline, and also shed light on challenges faced by users during the hypothesis formation step. We conclude by discussing design opportunities for ML and HCI researchers. Our findings point to the importance of centering users when creating and evaluating new tools for slice discovery.

show abstract

Section: Design Opportunitiesmentioning

confidence: 99%