Authorship Attribution for Neural Text Generation

Abstract

In recent years, the task of generating realistic short and long texts have made tremendous advancements. In particular, several recently proposed neural network-based language models have demonstrated their astonishing capabilities to generate texts that are challenging to distinguish from human-written texts with the naked eye. Despite many benefits and utilities of such neural methods, in some applications, being able to tell the "author" of a text in question becomes critically important. In this work, in the context of this Turing Test, we investigate the so-called authorship attribution problem in three versions: (1) given two texts T 1 and T 2 , are both generated by the same method or not? (2) is the given text T written by a human or machine? (3) given a text T and k candidate neural methods, can we single out the method (among k alternatives) that generated T ? Against one humanwritten and eight machine-generated texts (i.e., CTRL, GPT, GPT2, GROVER, XLM, XL-NET, PPLM, FAIR), we empirically experiment with the performance of various models in three problems. By and large, we find that most generators still generate texts significantly different from human-written ones, thereby making three problems easier to solve. However, the qualities of texts generated by GPT2, GROVER, and FAIR are better, often confusing machine classifiers in solving three problems. All codes and datasets of our experiments are

Automatic Detection of Machine Generated Text: A Critical Survey

“…They show that word order does not matter much as a bag-of-words detector performs very similar to detectors based on complex encoder (e.g., transformer). This result is consistent with the recent work done by Uchendu et al, (2020), which shows that simple models (traditional ML models trained on psychological features and simple neural network architectures) perform well in three settings: (i) classify if two given articles are generated by the same TGM; (ii) classify if a given article is written by a human or a TGM (the original detection problem); (iii) identify the TGM that generated a given article (similar to Tay et al, (2020)). For the original detection problem, the authors find that the text generated by the GPT-2 model to be hard to detect among several TGMs (see Appendix for the list of studied TGMs).…”

“…TGMs can also be used to generate text that approximately matches the style of human language, which benefits applications such as story generation (Fan et al, 2018), conversational response generation (Zhang et al, 2020), code auto-completion (TabNine, 2020), and radiology report generation (Liu et al, 2019a). Malicious usage: TGMs can have unfortunate uses by (even low-skilled) adversaries for malicious purposes, such as fake news generation (Zellers et al, 2019;Brown et al, 2020;Uchendu et al, 2020), fake product reviews generation (Adelani et al, 2020), and spamming/phishing (Weiss, 2019). Humans can spot fake news articles (Brown et al, 2020), fake product reviews (Adelani et al, 2020), and fake comments (Weiss, 2019) generated by TGM only at chance level.…”

“…The RoBERTa detector also outperforms existing detectors in spotting news articles generated by several TGMs (Uchendu et al, 2020) and product reviews generated by the GPT-2 model fine-tuned on Amazon product reviews (Adelani et al, 2020).…”

“…TGMs are useful in a wide variety of applications, including story generation (Fan et al, 2018), conversational response generation (Zhang et al, 2020), code auto-completion (Solaiman et al, 2019), and radiology report generation (Liu et al, 2019a). However, TGMs can also be misused for fake news generation (Zellers et al, 2019;Brown et al, 2020;Uchendu et al, 2020), fake product reviews generation (Adelani et al, 2020), and spamming/phishing. (Weiss, 2019).…”

“…The classifier, henceforth called detector, can be used to automatically remove machine generated text from online platforms such as social media, e-commerce, email clients, and government forums, when the intention of the TGM generated text is abuse. An ideal detector should be: (i) accurate, that is, good accuracy with a good trade-off for false positives and false negatives depending on the online platform (email client, social media) on which TGM is applied (Solaiman et al, 2019); (ii) data-efficient, that is, needs as few examples as possible from the TGM used by the attacker (Zellers et al, 2019); (iii) generalizable, that is, detects text generated by different modeling choices of the TGM used by the attacker such as model architecture, TGM training data, TGM conditioning prompt length, model size, and text decoding method (Solaiman et al, 2019;Bakhtin et al, 2020;Uchendu et al, 2020); and (iv) interpretable, that is, detector decisions need to be understandable to humans (Gehrmann et al, 2019); and (v) robust, that is, detector can handle adversarial examples (Wolff, 2020). Given the importance of this problem, there has been a flurry of research recently from both NLP and ML communities on building useful detectors.…”