A Study on Manual and Automatic Evaluation for Text Style Transfer: The Case of Detoxification

Logacheva, Varvara; Dementieva, Daryna; Кротова, И. Н.; Fenogenova, Alena; Nikishina, Irina; Shavrina, Tatiana; Panchenko, Alexander

doi:10.18653/v1/2022.humeval-1.8

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yet, in each work, the comparison between models is made by automatic metrics that are not unified, and their choice may be arbitrary (Ostheimer et al, 2023). There are several recent works that studied the correlation between automatic and manual evaluation for text style transfer tasks -formality (Lai et al, 2022a) and toxicity (Logacheva et al, 2022a). Our work presents a new set of metrics for automatic evaluation for English and Russian languages, confirming our choice with correlations with manual metrics.…”

Section: Evaluation Setupssupporting

confidence: 64%

See 1 more Smart Citation

Methods for Detoxification of Texts for the Russian Language

Dementieva¹,

Moskovskiy²,

Logacheva³

et al. 2021

Computational Linguistics and Intellectual Technologies

View full text Add to dashboard Cite

We introduce the first study of automatic detoxification of Russian texts to combat offensive language. Such a kind of textual style transfer can be used, for instance, for processing toxic content in social media. While much work has been done for the English language in this field, it has never been solved for the Russian language yet. We test two types of models -unsupervised approach based on BERT architecture that performs local corrections and supervised approach based on pretrained language GPT-2 model -and compare them with several baselines. In addition, we describe evaluation setup providing training datasets and metrics for automatic evaluation. The results show that the tested approaches can be successfully used for detoxification, although there is room for improvement.

show abstract

Section: Evaluation Setupssupporting

confidence: 64%

“…Here, we present the explanation of labels that annotators had to assign for each of the three evaluation parameters. We adapt the manual annotation process described in (Logacheva et al, 2022a):…”

Section: E Manual Evaluation Instructionsmentioning

confidence: 99%