The Role of Domain Expertise in User Trust and the Impact of First Impressions with Intelligent Systems

Abstract

Domain-specific intelligent systems are meant to help system users in their decision-making process. Many systems aim to simultaneously support different users with varying levels of domain expertise, but prior domain knowledge can affect user trust and confidence in detecting system errors. While it is also known that user trust can be influenced by first impressions with intelligent systems, our research explores the relationship between ordering bias and domain expertise when encountering errors in intelligent systems. In this paper, we present a controlled user study to explore the role of domain knowledge in establishing trust and susceptibility to the influence of first impressions on user trust. Participants reviewed an explainable image classifier with a constant accuracy and two different orders of observing system errors (observing errors in the beginning of usage vs. in the end). Our findings indicate that encountering errors early-on can cause negative first impressions for domain experts, negatively impacting their trust over the course of interactions. However, encountering correct outputs early helps more knowledgable users to dynamically adjust their trust based on their observations of system performance. In contrast, novice users suffer from over-reliance due to their lack of proper knowledge to detect errors. IntroductionSystem designers and practitioners incorporate machine learning and artificial intelligence (ML/AI) models to help end-users achieve their goals and make decisions. Intelligent systems are used across a wide variety of domains, such as medical diagnosis assistance (Goyal et al. 2018;Bussone, Stumpf, and O'Sullivan 2015), cybersecurity monitoring (Goyal and Sharma 2019), and criminal justice (Rudin and Ustun 2018;Berk and Hyatt 2015). The intended end users of such systems often possess different levels of background domain knowledge. For instance, medical decision support systems incorporate AI/ML approaches to help with automated diagnoses for diseases. While doctors and medical practitioners can use these systems to make a diagnosis or verify it, patients may use similar systems to input their symptoms for an early diagnosis.

Explaining Recommendations in E-Learning: Effects on Adolescents' Trust

“…It also entails different themes like competence, benevolence, and reliance [7,12,13,25,34,42]. On the other hand, it has been recognized that trust is not static but evolves [31,44,47]. Thus, measuring trust in automated systems is challenging and can be done either explicitly or implicitly.…”

“…Explicit measurements apply questionnaires or interviews to ask people about their trust perceptions. One-dimensional approaches measure trust with a single Likert-type question [31,40,44]. Although this method is quick and easy, it is susceptible to people interpreting 'trust' differently.…”

“…Future work can validate our questionnaire. Sixth, our short-term study could only assess initial trust, whereas trust is an entity that evolves [31,44]. In contrast, long-term studies could open up the possibility of measuring trust implicitly through loyalty [38,51].…”

Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative

“…Due to the importance of calibrating trust in decision aids, several recent studies have analyzed how the presentation of model decisions modulates trust. They've investigated factors including reported model confidence, model explanations, overall model accuracy, and initial model accuracy [34,37,53,54]. In these studies, users choose to accept or reject the model's output on discrete binary tasks and self-report their level of trust in the model.…”

Leveraging Large Language Models to Enhance Domain Expert Inclusion in Data Science Workflows

“…For instance, the domain expert might provide knowledge about specific data patterns useful for missing value imputations; based on current model performance, they may suggest creating new features such as calculating blood sugar levels over a 3-month period. Domain experts play a critical role in ensuring ML solutions are tailored to the broader domain goals, including fairness, transparency, and accountability [33,53].…”