Trust in AI: Interpretability is not necessary or sufficient, while black-box interaction is necessary and sufficient

Abstract

The problem of human trust in artificial intelligence is one of the most fundamental problems in applied machine learning. Our processes for evaluating AI trustworthiness have substantial ramifications for ML's impact on science, health, and humanity, yet confusion surrounds foundational concepts. What does it mean to trust an AI, and how do humans assess AI trustworthiness? What are the mechanisms for building trustworthy AI? And what is the role of interpretable ML in trust?Here, we draw from statistical learning theory and sociological lenses on human-automation trust to motivate an AI-as-tool framework, which distinguishes human-AI trust from human-AI-human trust. Evaluating an AI's contractual trustworthiness involves predicting future model behavior using behavior certificates (BCs) that aggregate behavioral evidence from diverse sources including empirical out-of-distribution and out-of-task evaluation and theoretical proofs linking model architecture to behavior.We clarify the role of interpretability in trust with a ladder of model access. Interpretability (level 3) is not necessary or even sufficient for trust, while the ability to run a black-box model at-will (level 2) is necessary and sufficient. While interpretability can offer benefits for trust, it can also incur costs. We clarify ways interpretability can contribute to trust, while questioning the perceived centrality of interpretability to trust in popular discourse.How can we empower people with tools to evaluate trust? Instead of trying to understand how a model works, we argue for understanding how a model behaves. Instead of opening up black boxes, we should create more behavior certificates that are more correct, relevant, and understandable. We discuss how to build trusted and trustworthy AI responsibly with contract-aware model design, robustness testing, and favoring trust calibration over maximization. CCS Concepts: • Computing methodologies → Artificial intelligence; • Social and professional topics → Computing / technology policy; • Human-centered computing → Interaction design theory, concepts and paradigms.

Resilience-aware MLOps for AI-based medical diagnostic system

“…Pseudocode of model-agnostic meta-learning with evolution strategies for AI-system resilience optimization. (39). The level of perturbation is limited by the L °-norm or L 0 0 L -norm.…”

“…In addition to Uncertainty Monitoring, the Explainable AI mechanism can be used to assist decision-making by the human to whom control is delegated in case of uncertainty. The article ( 39 ) questions the necessity and adequacy of existing methods of explaining decisions, so the explanation mechanism will be excluded from further consideration, but for generality, the diagram shows this MLOps stage.…”

“…For differentiable models, FGSM attacks or PGD attacks can be used ( 36 , 41 ). It is proposed to use adversarial attacks based on the search algorithm of the covariance matrix adaptation evolution strategy for non-differentiable models ( 39 ). The level of perturbation is limited by the -norm or -norm.…”

Generating Neural Archetypes to Instruct Fast and Interpretable Decisions

“…Our work solves this issue by introducing the first interpretable concept-based model that learns logic rules from concept embeddings. Our approach draws from t-norm fuzzy logic learning paradigms (Diligenti et al, 2021;Badreddine et al, 2022;van Krieken et al, 2022)…”

“…Concept-based models (Kim et al, 2018;Chen et al, 2020) aim to increase human trust in deep learning models by using human-understandable concepts to train interpretable models-such as logistic regression or decision trees (Rudin, 2019;Koh et al, 2020;Kazhdan et al, 2020) (Figure 1). This approach significantly increases human trust in the AI predictor (Rudin, 2019;Shen, 2022) as it allows users to clearly understand a model's decision process. However, state-of-the-art concept-based models, which rely on concept embeddings (Yeh et al, 2020;Kazhdan et al, 2020;Mahinpei et al, 2021;Espinosa Zarlenga et al, 2022) to attain high performance, are not completely interpretable.…”