Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience

Abstract

The rise of powerful large language models (LLMs) brings about tremendous opportunities for innovation but also looming risks for individuals and society at large. We have reached a pivotal moment for ensuring that LLMs and LLM-infused applications are developed and deployed responsibly. However, a central pillar of responsible AI-transparency-is largely missing from the current discourse around LLMs. It is paramount to pursue new approaches to provide transparency for LLMs, and years of research at the intersection of AI and human-computer interaction (HCI) highlight that we must do so with a human-centered perspective: Transparency is fundamentally about supporting appropriate human understanding, and this understanding is sought by different stakeholders with different goals in different contexts. In this new era of LLMs, we must develop and design approaches to transparency by considering the needs of stakeholders in the emerging LLM ecosystem, the novel types of LLM-infused applications being built, and the new usage patterns and challenges around LLMs, all while building on lessons learned about how people process, interact with, and make use of information. We reflect on the unique challenges that arise in providing transparency for LLMs, along with lessons learned from HCI and responsible AI research that has taken a human-centered perspective on AI transparency. We then lay out four common approaches that the community has taken to achieve transparency-model reporting, publishing evaluation results, providing explanations, and communicating uncertainty-and call out open questions around how these approaches may or may not be applied to LLMs. We hope this provides a starting point for discussion and a useful roadmap for future research.

Sketching AI Concepts with Capabilities and Examples: AI Innovation in the Intensive Care Unit

“…Prior research notes that stakeholders with little to no background in data science or AI are rarely involved in problem selection and formulation, if involved at all [29,39,55,67]. There is a knowledge gap between data science and domain expertise [74,132,140]: Domain experts and designers struggle to understand what AI can do, they often envision AI services that cannot be built [35,78,135,142]. Data scientists find it challenging to elicit needs from domain experts, and without this input, they tend to envision AI services that users and impacted stakeholders do not want [74,81,88,96].…”

Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology

“…Prior work noted that stakeholders with little to no background in data science or AI (e.g., domain experts, UX designers, policymakers, etc) might be involved in the design of an AI system's user interface, but rarely in conversations around the objective of the underlying model or the overall problem formulation [39,41,109,133,144]. Recently, a growing body of work in HCI and AI has called for human-centered approaches for broadening participation in AI design to meaningfully engage domain stakeholders to brainstorm and reflect on whether an envisioned future technology is in fact addressing the right problem in the first place [10,34,35,40,70,78,151].…”

Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking

“…For example, some experimental work is being explored by integrating LLMs directly into feature-rich applications, such as Copilot in Microsoft 365 [43] and Firefly in Adobe [1]. This process has shown that developers can face new challenges in ensuring accurate and effective use of this new avenue of conversational UX experience [29,52]. As many of these interfaces are still at a nascent stage, it is unclear how this current practice (i.e., integrating the context of these feature-rich applications) can help novice end-users in seeking accurate and relevant assistance from LLMs.…”

“…Unlike traditional help-seeking mediums that rely on keyword matching, prompt-based interactions within LLMs offer humanlike language capabilities [29], which is unique, but can also be unreliable. This unreliability comes from the biases (e.g., hallucinating and non-deterministic output) inherent within prompt-based interactions of LLMs.…”

“…However, the effectiveness of LLMbased assistance and how end-users employ LLMs to seek help for feature-rich applications remain important open questions. Furthermore, there have been calls [29] for a better understanding of the dynamics of user interaction across diverse AI usage scenarios, aiming to avoid false assumptions.…”