On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective

Abstract

ChatGPT is a recent chatbot service released by OpenAI and is receiving increasing attention over the past few months. While evaluations of various aspects of Chat-GPT have been done, its robustness, i.e., the performance to unexpected inputs, is still unclear to the public. Robustness is of particular concern in responsible AI, especially for safety-critical applications. In this paper, we conduct a thorough evaluation of the robustness of ChatGPT from the adversarial and out-of-distribution (OOD) perspective. To do so, we employ the AdvGLUE and ANLI benchmarks to assess adversarial robustness and the Flipkart review and DDXPlus medical diagnosis datasets for OOD evaluation. We select several popular foundation models as baselines. Results show that ChatGPT shows consistent advantages on most adversarial and OOD classification and translation tasks. However, the absolute performance is far from perfection, which suggests that adversarial and OOD robustness remains a significant threat to foundation models. Moreover, ChatGPT shows astounding performance in understanding dialogue-related texts and we find that it tends to provide informal suggestions for medical tasks instead of definitive answers. Finally, we present in-depth discussions of possible research directions.

A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4

“…These instances fall outside the range of the model's training data and present challenges to the model's performance and generalization ability. Some of the recent research works focused on evaluating the robustness of GLLMs to out-of-distribution instances [456], [461], adversarial prompts [458]- [460] and adversarial inputs [425], [455], [457], [462] in one or more natural language processing tasks. Table 21 presents a summary of research works assessing GLLMs robustness to out-of-distribution instances, adversarial prompts and adversarial inputs.…”

“…Moreover, ChatGPT demonstrates better robustness to adversarial inputs than SOTA models in text-to-SQL generation. Some of the research works evaluated the GLLM robustness in multiple natural language understanding and generation tasks [455], [456], [458], [460]. Chen et al [455] assessed the robustness of GPT-3 and GPT-3.5 models on 21 datasets covering nine natural language understanding tasks.…”

“…The authors observed that the models are robust in tasks like machine reading comprehension and exhibit performance degradation of more than 35% in tasks like sentiment analysis and natural language inference. Wang et al [456] evaluated the robustness of GPT-3.5 and ChatGPT models on adversarial and out-of-distribution (OOD) samples on nine datasets covering four NLU tasks and machine translation. The authors observed that ChatGPT exhibits good performances on adversarial and OOD samples, but still, there is much room for improvement.…”

“…In some of the tasks like data labelling [373]- [375], [383], text classification [144], relation extraction [156], question answering [132], [179], keyphrase generation [217], etc., these models achieved even SOTA results. However, some of the recent research works exposed the brittleness of these models towards out-of-distribution inputs [456], [461], adversarial prompts [458]- [460] and inputs [425], [455], [457], [462] . For example, Liu et al [461] reported that ChatGPT and GPT-4 perform well in multiple choice question answering but struggle to answer out-of-distribution questions.…”

ChatGPT: A Brief Narrative Review

“…It has developed the ability to detect typical grammatical constructions and idioms after being trained on a vast corpus of text data, which includes books, papers, and websites [33]. This implies that even when the data it gets is not properly constructed or includes faults, it may nevertheless provide replies that are grammatically accurate and semantically relevant [34].…”