Can AI-Generated Text be Reliably Detected?

Abstract

The rapid progress of Large Language Models (LLMs) has made them capable of performing astonishingly well on various tasks including document completion and question answering. The unregulated use of these models, however, can potentially lead to malicious consequences such as plagiarism, generating fake news, spamming, etc. Therefore, reliable detection of AI-generated text can be critical to ensure the responsible use of LLMs. Recent works attempt to tackle this problem either using certain model signatures present in the generated text outputs or by applying watermarking techniques that imprint specific patterns onto them. In this paper, both empirically and theoretically, we show that these detectors are not reliable in practical scenarios. Empirically, we show that paraphrasing attacks, where a light paraphraser is applied on top of the generative text model, can break a whole range of detectors, including the ones using the watermarking schemes as well as neural network-based detectors and zero-shot classifiers. We then provide a theoretical impossibility result indicating that for a sufficiently good language model, even the best-possible detector can only perform marginally better than a random classifier. Finally, we show that even LLMs protected by watermarking schemes can be vulnerable against spoofing attacks where adversarial humans can infer hidden watermarking signatures and add them to their generated text to be detected as text generated by the LLMs, potentially causing reputational damages to their developers. We believe these results can open an honest conversation in the community regarding the ethical and reliable use of AI-generated text.Preprint. Under review.

How generative artificial intelligence has blurred notions of authorial identity and academic norms in higher education, necessitating clear university usage policies

“…Academics may view it in a more negative light and a tool which is perhaps more likely to jeopardise authorial integrity. At the time of writing, new GenAI detection software is being developed and tested to identify AI-generated text (Sadasivan et al, 2023;OpenAI, 2023a, b, c). It may therefore be the case that, in time, such software may nullify the overarching issue of the use of GenAI in academia.…”

“…The emergence of these results, and similar studies in this area (Gao et al, 2022;Malinka et al, 2022;Fowler, 2023;Biswas, 2023), have raised concerns across HE regarding the potential misuse of ChatGPT to write summative assessments (Dwivedi et al, 2023). Sadasivan et al (2023) tested 10,000 text samples, testing the efficacy of ten distinct detection modalities. The findings highlighted that, although certain detection methods showcased commendable accuracy, none proved to be entirely infallible.…”

Have AI-Generated Texts from LLM Infiltrated the Realm of Scientific Writing? A Large-Scale Analysis of Preprint Platforms

“…Regardless of personal opinions, AI tools such as ChatGPT have become embedded in daily human communication. However, unlike scenarios where students employ AI to complete assignments, scientific writing has traditionally been viewed as a means of disseminating ideas and new knowledge to humanity, raising concerns within the community about ethical issues more than just plagiarism (Sadasivan et al, 2023).…”

AI and Democracy's Digital Identity Crisis

“…AI PUTS DEMOCRATIC SYSTEMS AT RISK Modern AI models are increasingly capable of mimicking targeted individuals [1], [2], [3], [4], [5]. Users of these models can influence policymakers by impersonating constituents and can influence populations via compelling disinformation.…”