Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metric

Yamshchikov, Ivan P.; Shibaev, Viacheslav; Khlebnikov, Nikolay; Tikhonov, Alexey

doi:10.1609/aaai.v35i16.17672

Cited by 14 publications

(4 citation statements)

References 20 publications

(17 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is worth mentioning that Yamshchikov et al (2021) proved in a recent study that fastText and Word2Vec pre-trained embedding vectors should not be used to evaluate text style transfer approaches in terms of content preservation. They demonstrated how such evaluation pipelines suffer from inaccurate content prediction, in analogy to similar human judgements.…”

Section: Methodsmentioning

confidence: 99%

“…To evaluate fluency and content preservation, Hu et al (2022) introduced metrics such as perplexity score (PPL), style transfer accuracy (ACC), Word Overlap (WO) and self-BLEU, further summarizing these with Geometric Mean (G-Score) and Harmonic Mean (H-Score). Conversely, Yamshchikov et al (2021) argued against the usage of fastText and Word2Vec embeddings for the evaluation of content preservation in text style transfer.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

PGST: A Persian gender style transfer method

Khanmohammadi

Mirroshandel

2023

Nat. Lang. Eng.

View full text Add to dashboard Cite

Recent developments in text style transfer have led this field to be more highlighted than ever. There are many challenges associated with transferring the style of input text such as fluency and content preservation that need to be addressed. In this research, we present PGST, a novel Persian text style transfer approach in the gender domain, composed of different constituent elements. Established on the significance of parts of speech tags, our method is the first that successfully transfers the gendered linguistic style of Persian text. We have proceeded with a pre-trained word embedding for token replacement purposes, a character-based token classifier for gender exchange purposes, and a beam search algorithm for extracting the most fluent combination. Since different approaches are introduced in our research, we determine a trade-off value for evaluating different models’ success in faking our gender identification model with transferred text. Our research focuses primarily on Persian, but since there is no Persian baseline available, we applied our method to a highly studied gender-tagged English corpus and compared it to state-of-the-art English variants to demonstrate its applicability. Our final approach successfully defeated English and Persian gender identification models by 45.6% and 39.2%, respectively.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

PGST: A Persian gender style transfer method

Khanmohammadi

Mirroshandel

2023

Nat. Lang. Eng.

View full text Add to dashboard Cite

show abstract

“…Mir et al 41 constructed style-specific dictionaries to remove or mask style-related words to improve this shortcoming of BLEU, while Pang et al 19 argued that for complex tasks, the masking process is not conducive to retaining content or semantic information. Yamshchikov et al 44 argue that none of the current measures used for semantic similarity assessment are consistent with human understanding of semantic similarity, and that the method of using WMD to calculate content retention has the highest correlation with human evaluation.…”

Section: Automatic Evaluationmentioning

confidence: 99%

A review of unsupervised text style transfer based on deep learning

Guo,

Rao

2023

Fourth International Conference on Artificial Intelligence and Electromechanical Automation (AIEA 2023)

View full text Add to dashboard Cite

Text style transfer is mainly to modify the text style to suit various application scenarios without changing the semantic meaning of the text, which is a great significant issue in natural language processing. To expedite research progress, this survey conducts a systematic review of existing literature. Drawing from a vast body of research, this survey first extracts the essential connotations of style information in text of varying granularity across different tasks, and then provide a clear definition of text style transfer, summarize the main challenges at present and comprehensively codify and discuss the current datasets used for evaluation, as well as their indicators. Moreover, this survey compares the mechanisms, advantages, and shortcomings of unsupervised classical methods across word, sentence, and paragraph levels. Finally, the future directions of fighting are also given, hoping to facilitate more comprehensive solutions.

show abstract

“…Our framework continues this line of research to produce interpretable metrics for multiple aspects. While recent evaluation frameworks each discussed the key evaluation aspects of one NLG task (Venkatesh et al, 2018;Mir et al, 2019;Yamshchikov et al, 2020;Fabbri et al, 2021), our framework provides a unified methodology that facilitates metric design for all the three main categories of tasks. We also highlight that all of metrics (except for the relevance metric for summarization) are reference-free once trained.…”

Section: Related Workmentioning

confidence: 99%

Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation

Deng¹,

Tan²,

Liu³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Natural language generation (NLG) spans a broad range of tasks, each of which serves for specific objectives and desires different properties of generated text. The complexity makes automatic evaluation of NLG particularly challenging. Previous work has typically focused on a single task and developed individual evaluation metrics based on specific intuitions. In this paper, we propose a unifying perspective based on the nature of information change in NLG tasks, including compression (e.g., summarization), transduction (e.g., text rewriting), and creation (e.g., dialog). Information alignment between input, context, and output text plays a common central role in characterizing the generation. With automatic alignment prediction models, we develop a family of interpretable metrics that are suitable for evaluating key aspects of different NLG tasks, often without need of gold reference data. Experiments show the uniformly designed metrics achieve stronger or comparable correlations with human judgement compared to state-of-the-art metrics in each of diverse tasks, including text summarization, style transfer, and knowledgegrounded dialog. 1

show abstract

Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metric

Cited by 14 publications

References 20 publications

PGST: A Persian gender style transfer method

PGST: A Persian gender style transfer method

A review of unsupervised text style transfer based on deep learning

Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation

Contact Info

Product

Resources

About