Human Perception in Natural Language Generation

Abstract

We take a collection of short texts, some of which are human-written, while others are automatically generated, and ask subjects, who are unaware of the texts' source, whether they perceive them as human-produced. We use this data to fine-tune a GPT-2 model to push it to generate more human-like texts, and observe that the production of this fine-tuned model is indeed perceived as more humanlike than that of the original model. Contextually, we show that our automatic evaluation strategy correlates well with human judgements. We also run a linguistic analysis to unveil the characteristics of human-vs machineperceived language.

A text style transfer system for reducing the physician–patient expertise gap: An analysis with automatic and human evaluations

“…This highlights the difficulty for humans in assessing the style strength, separating it from the structure and semantics. These findings are in line with recent studies in the field (De Mattei, Cafagna, Dell'Orletta, & Nissim, 2020).…”

From theories on styles to their transfer in text: Bridging the gap with a hierarchical survey

“…De Mattei et al (2020) put forward the idea that news styles are more difficult to judge than others (e.g., sentiment), and that humans are not as reliable judges of said styles as machines. They proposed a framework for the automatic evaluation of style-aware generation that seems handy for style transfer as well.…”

“…A useful newspaper dataset for style transfer was created by De Mattei et al (2020), even though their work regarded style-aware generation rather than transfer. They collected news that are lexically similar from two newspapers, a subset of which are topic-aligned.…”

What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability

“…Reference-free evaluation. A popular, referencefree alternative is to train evaluation models that discriminate human from model output (e.g., Bruni and Fernández, 2017;Gehrmann et al, 2019;Hashimoto et al, 2019), score the appropriateness of input-output pairs (e.g., Sinha et al, 2020;Fomicheva et al, 2020), or model human judgements directly (e.g., Lowe et al, 2017;De Mattei et al, 2021;Rei et al, 2021). Neural language models themselves have been proposed as evaluators (e.g., Yuan et al, 2021;Deng et al, 2021) and used to assess generations along interpretable evaluation dimensions (Zhong et al, 2022), yet they have been criticised for being biased (toward models similar to the evaluator) and thus limited in their ability to evaluate generated text (Deutsch et al, 2022).…”