The Myth of Double-Blind Review Revisited: ACL vs. EMNLP

Caragea, Cornelia; Uban, Ana Sabina; Dinu, Liviu P.

doi:10.18653/v1/d19-1236

“…To illustrate the latter point: a quick search in the ACL anthology revealed only four conference papers on peer review from a meta-research perspective: a paper-reviewer matching tool (Anjum et al, 2019), a corpus of reviews (Kang et al, 2018), and two experimental studies using NLP to explain the observed reviews (Caragea et al, 2019;Gao et al, 2019). We could not find any ACL-published 2019) offers an actionable insight: ACL reviewers appear to be victims of conformity bias, converging to the mean of reviews.…”

Section: So What Can We Do?mentioning

confidence: 94%

What Can We Do to Improve Peer Review in NLP?

Rogers¹,

Augenstein²

2020

Preprint

View full text Add to dashboard Cite

Peer review is our best tool for judging the quality of conference submissions, but it is becoming increasingly spurious. We argue that a part of the problem is that the reviewers and area chairs face a poorly defined task forcing apples-to-oranges comparisons. There are several potential ways forward, but the key difficulty is creating the incentives and mechanisms for their consistent implementation in the NLP community.

show abstract

“…To illustrate the latter point: a quick search in the ACL anthology revealed only four conference papers on peer review from a meta-research perspective: a paper-reviewer matching tool (Anjum et al, 2019), a corpus of reviews (Kang et al, 2018), and two experimental studies using NLP to explain the observed reviews (Caragea et al, 2019;Gao et al, 2019). We could not find any ACL-published 9 E.g.…”

Section: So What Can We Do?mentioning

confidence: 98%

What Can We Do to Improve Peer Review in NLP?

Rogers¹,

Augenstein²

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Peer review is our best tool for judging the quality of conference submissions, but it is becoming increasingly spurious. We argue that a part of the problem is that the reviewers and area chairs face a poorly defined task forcing apples-to-oranges comparisons. There are several potential ways forward, but the key difficulty is creating the incentives and mechanisms for their consistent implementation in the NLP community.

show abstract

“…Therefore, debiasing and attribute removal are techniques to mitigate the undesired discriminatory effects arising from unfair models. Examples of use case tasks covering bias and fairness encompass learning gender-neutral word embeddings (Zhao et al 2018;Bolukbasi et al 2016), analysis and reduction of gender bias in multi-lingual word embeddings Font and Costa-jussà 2019), text rewriting (Xu et al 2019), analysis of biases in contextualized word representations (Tan and Celis 2019; Gonen and Goldberg 2019;, detection, reduction and evaluation of biases for demographic attributes in word embeddings (Papakyriakopoulos et al 2020;Najafian 2020, 2019;Kaneko and Bollegala 2019), analogy detection (Nissim et al 2020), cyberbullying text detection (Gencoglu 2020), fair representation learning , protected attributes removal (Elazar and Goldberg 2018;, analysis of racial disparity in NLP (Blodgett and O'Connor 2017), and prediction of scientific papers authorship during double-blind review (Caragea et al 2019).…”

Section: Privacy Terminologymentioning

confidence: 99%

How to keep text private? A systematic review of deep learning methods for privacy-preserving natural language processing

Sousa¹,

Kern²

2022

Preprint

0

View full text Add to dashboard Cite

Deep learning (DL) models for natural language processing (NLP) tasks often handle private data, demanding protection against breaches and disclosures. Data protection laws, such as the European Union's General Data Protection Regulation (GDPR), thereby enforce the need for privacy. Although many privacy-preserving NLP methods have been proposed in recent years, no categories to organize them have been introduced yet, making it hard to follow the progress of the literature. To close this gap, this article systematically reviews over sixty DL methods for privacy-preserving NLP published between 2016 and 2020, covering theoretical foundations, privacy-enhancing technologies, and analysis of their suitability for real-world scenarios. First, we introduce a novel taxonomy for classifying the existing methods into three categories: data safeguarding methods, trusted methods, and verification methods. Second, we present an extensive summary of privacy threats, datasets for applications, and metrics for privacy evaluation. Third, throughout the review, we describe privacy issues in the NLP pipeline in a holistic view. Further, we discuss open challenges in privacy-preserving NLP regarding data traceability, computation overhead, dataset size, the prevalence of human biases in embeddings, and the privacy-utility tradeoff. Finally, this review presents future research directions to guide successive research and development of privacy-preserving NLP models.

show abstract

The Myth of Double-Blind Review Revisited: ACL vs. EMNLP

Cited by 10 publications

References 17 publications

What Can We Do to Improve Peer Review in NLP?

What Can We Do to Improve Peer Review in NLP?

What Can We Do to Improve Peer Review in NLP?

How to keep text private? A systematic review of deep learning methods for privacy-preserving natural language processing

Contact Info

Product

Resources

About