Using Simulation to Aid the Design and Optimization of Intelligent User Interfaces for Quality Assurance Processes in Machine Learning

Abstract

Many mission-critical applications of machine learning (ML) in the real-world require a quality assurance (QA) process before the decisions or predictions of an ML model can be deployed. Because QA4ML users have to view a non-trivial amount of data and perform many input actions to correct errors made by the ML model, an optimally-designed user interface (UI) can reduce the cost of interactions significantly. A UI's effectiveness can be affected by many factors, such as the number of data objects processed concurrently, the types of commands for correcting errors, and the availability of algorithms for assisting users. We propose using simulation to aid the design and optimization of intelligent user interfaces for QA4ML processes. In particular, we focus on simulating the combined effects of human intelligence in selecting appropriate commands and algorithms, and machine intelligence in providing a collection of general-purpose algorithms for reordering data objects to be quality-assured.CCS Concepts: • Human-centered computing → HCI design and evaluation methods; • Information systems → Multimedia information systems.

OneLabeler: A Flexible System for Building Data Labeling Tools

“…Annotators may also exploratorily review labels and search for potentially mislabeled data objects to correct [4,29,75]. In missioncritical applications, quality assurance may require going through all the labels one by one [86]. • Stoppage Analysis (4/188) decides whether to keep assigning tasks to annotators or stop (Fig.…”