Classification Under Human Assistance

De, Abir; Okati, Nastaran; Zarezade, Ali; Gomez-Rodriguez, Manuel

doi:10.1609/aaai.v35i7.16738

Cited by 11 publications

(12 citation statements)

References 27 publications

(31 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To tackle this problem, we reformulate it into a cardinality-constrained set function optimization task. Subsequently, we introduce novel set function properties, (m, m)-partial monotonicity and (γ, γ)-weak submodularity, extending recent notions of partial monotonicity (Mualem and Feldman 2022) and approximate submodularity (Elenberg et al 2018;Harshaw et al 2019;De et al 2021). These properties allow us to design a greedy algorithm GENEX, to compute near-optimal feature subsets, with new approximation guarantee.…”

Section: Discrete Continuous Training Frameworkmentioning

confidence: 99%

Generator Assisted Mixture of Experts for Feature Acquisition in Batch

Asgaonkar,

Jain,

2024

AAAI

View full text Add to dashboard Cite

Given a set of observations, feature acquisition is about finding the subset of unobserved features which would enhance accuracy. Such problems has been explored in a sequential setting in prior work. Here, the model receives feedback from every new feature acquireed and chooses to explore more features or to predict. However, sequential acquisition is not feasible in some settings where time is of essence. We consider the problem of feature acquisition in batch, where the subset of features to be queried in batch is chosen based on the currently observed features, and then acquired as a batch, followed by prediction. We solve this problem using several technical innovations. First, we use a feature generator to draw a subset of the synthetic features for some examples, which reduces the cost of oracle queries. Second, to make the feature acquisition problem tractable for the large heterogeneous observed features, we partition the data into buckets, by borrowing tools from locality sensitive hashing and then train a mixture of experts model. Third, we design a tractable lower bound of the original objective. We use a greedy algorithm combined with model training to solve the underlying problem. Experiments with four datasets shows that our approach outperforms these methods in terms of trade off between accuracy and feature acquisition cost.

show abstract

Section: Discrete Continuous Training Frameworkmentioning

confidence: 99%

Generator Assisted Mixture of Experts for Feature Acquisition in Batch

Asgaonkar,

Jain,

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…In this context, several approaches jointly train the classifier together with a deferral system (Madras, Pitassi, and Zemel 2018;Okati, De, and Rodriguez 2021;Wilder, Horvitz, and Kamar 2020). Further work utilizes objective functions with theoretical guarantees for regression (De et al 2020) and classification tasks (De et al 2021). Moreover, Mozannar and Sontag (2020) propose a consistent surrogate loss function inspired by cost-sensitive learning.…”

Section: Related Workmentioning

confidence: 99%

Learning to Defer with Limited Expert Predictions

Hemmer

Thede

Vössing

et al. 2023

AAAI

View full text Add to dashboard Cite

Recent research suggests that combining AI models with a human expert can exceed the performance of either alone. The combination of their capabilities is often realized by learning to defer algorithms that enable the AI to learn to decide whether to make a prediction for a particular instance or defer it to the human expert. However, to accurately learn which instances should be deferred to the human expert, a large number of expert predictions that accurately reflect the expert's capabilities are required—in addition to the ground truth labels needed to train the AI. This requirement shared by many learning to defer algorithms hinders their adoption in scenarios where the responsible expert regularly changes or where acquiring a sufficient number of expert predictions is costly. In this paper, we propose a three-step approach to reduce the number of expert predictions required to train learning to defer algorithms. It encompasses (1) the training of an embedding model with ground truth labels to generate feature representations that serve as a basis for (2) the training of an expertise predictor model to approximate the expert's capabilities. (3) The expertise predictor generates artificial expert predictions for instances not yet labeled by the expert, which are required by the learning to defer algorithms. We evaluate our approach on two public datasets. One with "synthetically" generated human experts and another from the medical domain containing real-world radiologists' predictions. Our experiments show that the approach allows the training of various learning to defer algorithms with a minimal number of human expert predictions. Furthermore, we demonstrate that even a small number of expert predictions per class is sufficient for these algorithms to exceed the performance the AI and the human expert can achieve individually.

show abstract

“…Cognitive scientists have developed and tested different theories about the cognitive process underpinning responsibility judgments (Alicke, 2000;Chockler & Halpern, 2004;Gerstenberg & Lagnado, 2010;Shaver, 2012). However, the increasing development of AI systems that assist and collaborate with humans, rather than replacing them (Balazadeh Meresht et al, 2022;De et al, 2020De et al, , 2021Mozannar et al, 2022;Okati et al, 2021;Raghu et al, 2019;Straitouri et al, 2021;Wilder et al, 2021), calls for more empirical and theoretical research to shed light on the way humans make responsibility judgments in situations involving human-AI teams (Cañas, 2022). Recent work in that area has identified several factors that influence responsibility judgments (Awad et al, 2020;Lima et al, 2021;Longin et al, 2023).…”

Section: Introductionmentioning

confidence: 99%

Towards a computational model of responsibility judgments in sequential human-AI collaboration

Tsirtsis,

Rodriguez,

Gerstenberg

2024

Preprint

View full text Add to dashboard Cite

When a human and an AI agent collaborate to complete a task and something goes wrong, who is responsible? Prior work has developed theories to describe how people assign responsibility to individuals in teams. However, there has been little work studying the cognitive processes that underlie responsibility judgments in human-AI collaborations, especially for tasks comprising a sequence of interdependent actions. In this work, we take a step towards filling this gap. Using semi-autonomous driving as a paradigm, we develop an environment that simulates stylized cases of human-AI collaboration using a generative model of agent behavior. We propose a model of responsibility that considers how unexpected an agent’s action was, and what would have happened had they acted differently. We test the model’s predictions empirically and find that in addition to action expectations and counterfactual considerations, participants’ responsibility judgments are also affected by how much each agent actually contributed to the outcome.

show abstract

Classification Under Human Assistance

Cited by 11 publications

References 27 publications

Generator Assisted Mixture of Experts for Feature Acquisition in Batch

Generator Assisted Mixture of Experts for Feature Acquisition in Batch

Learning to Defer with Limited Expert Predictions

Towards a computational model of responsibility judgments in sequential human-AI collaboration

Contact Info

Product

Resources

About