Stay On-Topic: Generating Context-Specific Fake Restaurant Reviews

Abstract

Automatically generated fake restaurant reviews are a threat to online review systems. Recent research has shown that users have difficulties in detecting machine-generated fake reviews hiding among real restaurant reviews. The method used in this work (char-LSTM ) has one drawback: it has difficulties staying in context, i.e. when it generates a review for specific target entity, the resulting review may contain phrases that are unrelated to the target, thus increasing its detectability. In this work, we present and evaluate a more sophisticated technique based on neural machine translation (NMT) with which we can generate reviews that stay on-topic. We test multiple variants of our technique using native English speakers on Amazon Mechanical Turk. We demonstrate that reviews generated by the best variant have almost optimal undetectability (class-averaged F-score 47%). We conduct a user study with experienced users and show that our method evades detection more frequently compared to the state-of-the-art (average evasion 3.2/4 vs 1.5/4) with statistical significance, at level α = 1% (Section 4.3). We develop very effective detection tools and reach average F-score of 97% in classifying these. Although fake reviews are very effective in fooling people, effective automatic detection is still feasible.

Detecting Machine-Translated Text using Back Translation

“…Therefore, the quality is downgraded and this text is more easily distinguishable. The most changing comes from Juuti et al (2018)'s work in which the AD-ABOOST reaches the best accuracy contrasting with the result from the previous task in Table 1. Moreover, the best F -score places in another classifier, i.e., SVM(SMO).…”

“…Therefore, they suggested an inter-textual distance to estimate the similarity between two word distributions and used the distance to recognize the machinegenerated text. In fake review detection, Juuti et al (2018) extracted features from thirteen readability metrics. Moreover, they used N -gram models for various text components including words, simple POS, detailed POS and syntactic dependency.…”

“…On the other hand, the method based on the parsing tree (Li et al, 2015) can slightly improve the outcome but the structures of human and machine pairs seem to be similar in this balanced dataset. The most reasonable methods (Aharoni et al, 2014;Juuti et al, 2018) are to use N -gram model for measuring the text fluency. However, the performances are unstable among classifiers especially in Juuti et al (2018)'s work.…”

“…The most reasonable methods (Aharoni et al, 2014;Juuti et al, 2018) are to use N -gram model for measuring the text fluency. However, the performances are unstable among classifiers especially in Juuti et al (2018)'s work. It indicates that the current neural translator has already improved, so the only use of internal text information is insufficient to recognize machine-translated texts.…”

“…Other researchers prove that the coherence of the humanwritten text is better than the machine-translated one (Nguyen-Son et al, 2018, 2019. Beyond detecting machine-translated text, artificial fake reviews and papers are also recognized by readability (Juuti et al, 2018) and duplicate patterns (Labbé and Labbé, 2013), respectively. The limitation in all existing methods above is that they only analyze the intrinsic contents of machinegenerated texts but ignore the original processes which are used to produce the texts.…”