Explainability for Large Language Models: A Survey

Abstract

Large language models (LLMs) have demonstrated impressive capabilities in natural language processing. However, their internal mechanisms are still unclear and this lack of transparency poses unwanted risks for downstream applications. Therefore, understanding and explaining these models is crucial for elucidating their behaviors, limitations, and social impacts. In this paper, we introduce a taxonomy of explainability techniques and provide a structured overview of methods for explaining Transformer-based language models. We categorize techniques based on the training paradigms of LLMs: traditional fine-tuning-based paradigm and prompting-based paradigm. For each paradigm, we summarize the goals and dominant approaches for generating local explanations of individual predictions and global explanations of overall model knowledge. We also discuss metrics for evaluating generated explanations, and discuss how explanations can be leveraged to debug models and improve performance. Lastly, we examine key challenges and emerging opportunities for explanation techniques in the era of LLMs in comparison to conventional deep learning models.

Leveraging Large Language Models for Sensor Data Retrieval

“…Moreover LLMs are often seen as "black boxes" due to their complex and opaque nature, making it difficult to understand how they process data and arrive at specific outputs [63]. This lack of transparency can hinder the identification and rectification of privacy and security issues within the model.…”

Future Shock: Generative AI and the International AI Policy and Governance Crisis

“…Ensuring sufficient interpretability can help AI research scientists and developers to debug the models they are building and to uncover otherwise hidden or unforeseeable failure modes, thereby improving downstream model functioning and performance (Bastings et al, 2022;Luo & Specia, 2024;. It can also help detect and mitigate discriminatory biases that may be buried within model architectures (Alikhademi et al, 2021;Zhao, Chen, et al, 2024;Zhou et al, 2020). Furnishing understandable and accessible explanations of the rationale behind system outputs can likewise help to establish the lawfulness of AI systems (e.g., their compliance with data protection law and equality law) (Chuang et al, 2024; ICO/Turing, 2020) as well as to ensure responsible and trustworthy implementation by system deployers, who are better equipped to grasp system capabilities, limitations, and flaws and to integrate system outputs into their own reasoning, judgment, and experience (ICO/Turing, 2020; Leslie, Rincón, et al, 2024).…”

“…While the field of explainable AI (often referred to simply as XAI) has made notable progress over the past several years in advancing knowledge about the behaviors and potential flaws of opaque AI systems (Angelov et al, 2021;Räuker et al, 2023;Zhao, Chen, et al, 2024), myriad critical voices have emphasized that applications of contemporary AI explainability methods to black-box AI systems are rife with shortcomings that continue to hamper their real-world utility. These critics have cautioned against 'false hopes' that current explainability techniques provide justified reassurance about the safety, accuracy, reliability, and fairness of black-box models, stressing that contemporary approaches often generate misleading or unfaithful explanations (Ghassemi et al, 2021, p. e746).…”

“…The new order and scale of complexity of FMs has only compounded these difficulties faced by conventional XAI techniques in attempting to reliably and accurately explain deep learning models. Though some interpretability researchers, working on FMs/LLMs, have endeavored to build on local, feature attributionbased methods (Kokalj et al, 2021;Sanyal & Ren, 2021;Sikdar et al, 2021;Singh et al, 2024), the scaling of model depth and complexity has largely rendered traditional XAI methods unfit, incapable, or even obsolete (Chuang et al, 2024;Wu et al, 2024;Zhao, Chen, et al, 2024;Zou et al, 2023).…”

“…In response to the unprecedented demand for approaches to interpretability that can access the mechanics and higher level cognitive functions of complex model architectures, researchers have started to develop novel techniques that strive to open the FM/LLM black box. The most prominent of these can be organized into three categories: top-down methods of representation engineering, bottom-up methods of mechanistic interpretability, and outside-in methods of prompt-based self-explanation and prediction decomposition (Singh et al, 2024;Zhao, Chen, et al, 2024). While each of these emerging techniques heralds prospects of increasing model transparency, they are all also fraught with significant and yet-to-be-addressed problems.…”