QUESTO: Interactive Construction of Objective Functions for Classification Tasks

Abstract

Building effective classifiers requires providing the modeling algorithms with information about the training data and modeling goals in order to create a model that makes proper tradeoffs. Machine learning algorithms allow for flexible specification of such meta‐information through the design of the objective functions that they solve. However, such objective functions are hard for users to specify as they are a specific mathematical formulation of their intents. In this paper, we present an approach that allows users to generate objective functions for classification problems through an interactive visual interface. Our approach adopts a semantic interaction design in that user interactions over data elements in the visualization are translated into objective function terms. The generated objective functions are solved by a machine learning solver that provides candidate models, which can be inspected by the user, and used to suggest refinements to the specifications. We demonstrate a visual analytics system QUESTO for users to manipulate objective functions to define domain‐specific constraints. Through a user study we show that QUESTO helps users create various objective functions that satisfy their goals.

The Role of Interactive Visualization in Fostering Trust in AI

“…Additionally, when domain-specific inaccuracies or discrepancies are found in the outputs of these models, user feedback is often used to incrementally adapt the models to be better-suited to specific domains or decisions. Recently, Das et al 8 presented the concept of interactive objective functions as the method for this user feedback to be incorporated. Their approach allows users to specify particular aspects of the model and data that are important for the task or domain, which are then translated into constraints of the objective function used to generate and select models through the use of AutoML techniques.…”

CACTUS: Detecting and Resolving Conflicts in Objective Functions

“…Others have analysed variety in pareto-optimal solutions in multiobjective optimization functions [69], [70]. While none of these approaches allowed users to express specifications in objective functions interactively, Das et al prototyped QUESTO, allowing users to interactively create objective functions to select optimal classifiers trained on tabular data [7]. Unlike QUESTO, in this paper we seek to extend research on interactive objective functions by showing users conflicts in objectives as opposed to allowing interactive creation of objective functions.…”

“…These VA systems use models that are driven by an objective function as designed by an expert ML practitioner or a data scientist to achieve a desired data analytic goal, such as correctly predicting class labels of unseen data, or predict a quantitative value etc. Recently, Das et al have demonstrated a VA system, QUESTO [7] that facilitated interactive creation of objective functions to solve a classification task utilising an Auto-ML system. While their approach helped users to interactively explore and express a wide array of objectives to an objective function, the authors discussed potential conflicts that may occur in interactive specification of objectives as a limitation to QUESTO's workflow.…”

“…While their approach helped users to interactively explore and express a wide array of objectives to an objective function, the authors discussed potential conflicts that may occur in interactive specification of objectives as a limitation to QUESTO's workflow. For example, a user may emphasize to predict a set of relevant/critical data instances correctly, while mistakenly expressing that a subset of these data instances are outliers or noise in the data [7].…”

“…(1) Visualization of a multi-objective objective function that is defined using a python code (e.g, defined in a Jupyter notebook), or a visual interface such as QUESTO [7], (2) Highlighting conflicts between interactively specified objectives, (3) Helping users to resolve these conflicts to improve their objective functions, and (4) Train multiple classifiers incrementally (over multiple iterations) to perform trade-off analysis of objectives in multi-objective objective function. As objective functions drive all ML algorithms, we consider visualizing objective functions may help users (e.g., novice ML users, ML experts who may use GUI to debug models, etc.)…”