Identifying computer-generated text using statistical analysis

Abstract

Computer-based automatically generated text are used in various applications (e.g. text summarization, machine translation) and such the machine-generated text significantly helps our social life. However, machine-generated text may produce confusing information sometimes due to errors or inappropriate use of wordings caused by language processing, which could be a critical issue in president elections or in product advertisements. Previous methods for detecting such machinegenerated text typically estimates the text fluency, but, this may not be useful in near future because recently proposed neuralnetwork based natural language generation results in improved wording close to human-crafted one. However, we hypothesize that the habit of human on writing is still more consistent. For instance, the Zipf's law states that the most frequent word in the text written by human approximates twice the second most frequent word, nearly three times the third most frequent word, and so forth. We found that this is not true in the case of machine-generated text. We hence propose a method to identify the machine-generated text based on such the statistics-First, word distributed frequencies are compared with the Zipfian distribution to extract frequency features. Second, complex phrase features are extracted to show that humangenerated text contains more sophisticated phrases than machinegenerated one. Finally, the higher consistency of the humangenerated text is quantified at both the sentence level using phrasal verbs and at the paragraph level based on coreference resolution relationships, which are integrated into consistency features. The combination of the frequency, the complex phrase, and the consistency features is evaluated on a hundred of original English books and a hundred of translated ones from Finnish. The result shows that our method achieves the better performance (accuracy = 98.0% and equal error rate = 2.9%) comparing with a state-of-the-art method using parsing tree feature extraction. An advantage of this method is that this method can be used for large collections of text such as books efficiently. Other evaluation results in two other languages including French and Dutch showed similar results. They demonstrated that the proposed method works consistently in various languages.

Detecting Machine-Translated Paragraphs by Matching Similar Words

“…The limitation of this model is that meaningful features are only given from few nearby words, common in three. Other work [5,9] analyzes the histogram of word distribution from a massive amount of words, particularly suitable for document level. A recent method [10], the closest one to this paper, estimates text coherence by mutually matching words across pairwise sentences using word similarity.…”

“…This metric is perfectly used for classifying artificial and real papers with accuracy up to 100%, but Nguyen-Son et at. [9] indicated that the inter-textual metric is just suitable for paper detection and they developed another method for translation detection also based on word distribution. This method pointed out that a word distribution of human text is closer with a Zipfian distribution than that of machine one.…”

“…The coherence-based method using mean, variance, and their combination in both Cosine and Euclidean distance is compared with four previous methods on the same task. While three methods based on word distribution with coreference resolution (coreref) [9], N -gram model [1], and word similarity [10] can directly extract features from a paragraph, the other [6] based on parsing tree only obtains such features from an individual sentence. Thus, we adopt this method for a paragraph by calculating average on the features.…”

“…Through all previous methods, the highest performances (in bold) are identical or competitive to best classifiers indicated in italic with the maximum deviation only 0.6%. In these classifiers, the method based on word distribution and coreference resolution (coreref) [9] attained the lowest performance. The main reason is that the distribution is affected by a limited number of words within a paragraph.…”

“…The translators can upgrade the text coherence by integrating this mark into the translation. Intertextual metric [5] Word distribution and coreref [9] Parsing tree [6] N-gram and functional words [1] Word similarity [10] Combination (Euclidean)…”