X-FACTOR: A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization

Chaudhury, Subhajit; Swaminathan, Sarathkrishna; Gunasekara, Chulaka; Crouse, Maxwell; Ravishankar, Srinivas; Kimura, Daiki; Murugesan, Keerthiram; Astudillo, Ramón Fernandez; Naseem, Tahira; Kapanipathi, Pavan; Gray, Alexander G.

doi:10.18653/v1/2022.emnlp-main.478

Cited by 2 publications

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For generative downstream tasks, preserving the consistency and completeness of the data is essential. Thus, the fact-checking systems may also be used as an essential tool for the evaluation of the large language models (Tam et al, 2022), (Chaudhury et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

Fact-checking benchmark for the Russian Large Language Models

Kozlova,

Shevelev,

Fenogenova

2023

Computational Linguistics and Intellectual Technologies”

View full text Add to dashboard Cite

Modern text-generative language models are rapidly developing. They produce text of high quality and are used in many real-world applications. However, they still have several limitations, for instance, the length of the context, degeneration processes, lack of logical structure, and facts consistency. In this work, we focus on the fact-checking problem applied to the output of the generative models on classical downstream tasks, such as paraphrasing, summarization, text style transfer, etc. We define the task of internal fact-checking, set the criteria for factual consistency, and present the novel dataset for this task for the Russian language. The benchmark for internal fact-checking and several baselines are also provided. We research data augmentation approaches to extend the training set and compare classification methods on different augmented data sets.

show abstract

Section: Introductionmentioning

confidence: 99%