Defending Against Neural Fake News

Zellers, Rowan; Holtzman, Ari; Rashkin, Hannah; Bisk, Yonatan; Farhadi, Ali; Roesner, Franziska; Choi, Yejin

doi:10.48550/arxiv.1905.12616

Cited by 46 publications

(72 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Among the models trained on CommonCrawl include GPT-3 (Brown et al, 2020) with the addition of book datasets, GROVER (Zellers et al, 2019) on a restricted subset filtered to news domains called RealNews, and T5 (Raffel et al, 2020) on a cleaned version of common crawl called C4. Other models are trained on more curated Internet sources-for example Guo et al (2020) used high quality processed Wikipedia text from 40 different languages to train monolingual 141.4M parameter language models.…”

Section: Related Workmentioning

confidence: 99%

“…RealNews is a subset of the Common Crawl consisting of articles from news domains (Zellers et al, 2019). It contains 31M documents with average length 793 BPE tokens.…”

Section: Language Modeling Datasetsmentioning

confidence: 99%

“…Train-test leakage does not just impact models trained on C4. In Table 5, we show that whether or not an evaluation example has a near-duplicate in the train set has a significant impact on model perplexity for two standard models: Transformer-XL (Dai et al, 2019), which was trained on LM1B, and GROVER (Zellers et al, 2019), which was trained on RealNews. For Transformer XL, the perplexity halves on examples identified as nearduplicates.…”

Section: Impact On Existing Modelsmentioning

confidence: 99%

See 2 more Smart Citations

Deduplicating Training Data Makes Language Models Better

Lee¹,

Ippolito²,

Nystrom³

et al. 2021

Preprint

View full text Add to dashboard Cite

We find that existing language modeling datasets contain many near-duplicate examples and long repetitive substrings.As a result, over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data. We develop two tools that allow us to deduplicate training datasets-for example removing from C4 a single 61 word English sentence that is repeated over 60,000 times. Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer train steps to achieve the same or better accuracy. We can also reduce train-test overlap, which affects over 4% of the validation set of standard datasets, thus allowing for more accurate evaluation. We release code for reproducing our work and performing dataset deduplication at https://github.com/goog e-research/ dedup icate-text-datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

“…RealNews is a subset of the Common Crawl consisting of articles from news domains (Zellers et al, 2019). It contains 31M documents with average length 793 BPE tokens.…”

Section: Language Modeling Datasetsmentioning

confidence: 99%

Section: Impact On Existing Modelsmentioning

confidence: 99%

See 1 more Smart Citation

Deduplicating Training Data Makes Language Models Better

Lee¹,

Ippolito²,

Nystrom³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…[154], [155], defense against neural fake news: e.g. [156] and [157], [40], and word/sentence embedding-based defense: e.g. [158], [159].…”

Section: Misinformationmentioning

confidence: 99%

Adversarial Attacks and Defenses for Social Network Text Processing Applications: Techniques, Challenges and Future Research Directions

Alsmadi¹,

Ahmad²,

Nazzal³

et al. 2021

Preprint

View full text Add to dashboard Cite

The growing use of social media has led to the development of several Machine Learning (ML) and Natural Language Processing (NLP) tools to process the unprecedented amount of social media content to make actionable decisions. However, these ML and NLP algorithms have been widely shown to be vulnerable to adversarial attacks. These vulnerabilities allow adversaries to launch a diversified set of adversarial attacks on these algorithms in different applications of social media text processing. In this paper, we provide a comprehensive review of the main approaches for adversarial attacks and defenses in the context of social media applications with a particular focus on key challenges and future research directions. In detail, we cover literature on six key applications, namely (i) rumors detection, (ii) satires detection, (iii) clickbaits & spams identification, (iv) hate speech detection, (v) misinformation detection, and (vi) sentiment analysis. We then highlight the concurrent and anticipated future research questions and provide recommendations and directions for future work.

show abstract

“…Anyway, the latter does not become a stopper for NNs since we are living the Big Data, Big Compute, Big Models revolution [4,5,17]. In this way, following the exponential evolution of the availability of data and computational power, the tendency of solving complex tasks with complex models is increasing [4,15,39,48]. Arguably, the latter does not consider resource consumption regulations, and does not promote less dataintensive and lighter models, which sets the bases of greener AI.…”

mentioning

confidence: 99%