TS-ANNO: An Annotation Tool to Build, Annotate and Evaluate Text Simplification Corpora

Stodden, Regina; Kallmeyer, Laura

doi:10.18653/v1/2022.acl-demo.14

Cited by 5 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yamaguchi et al (2023) annotates simplifications of earlier models such as DRESS (Zhang and Lapata, 2017) and SUC (Sun et al, 2020) using a taxonomy of 62 error categories, but do not analyze the SOTA, MUSS, or LLMs. Stodden and Kallmeyer (2022) proposes an interactive linguistic inspection interface, but this interface is not designed for human evaluation of model outputs and does not provide ratings for measuring performance.…”

Section: Related Workmentioning

confidence: 99%

“…The relativizer 'that' creates no syntactic or conceptual simplicity, but adds clarity as to the identify of the subject. Trivial changes have previously been described with finer granularity, including subcategories like abbreviation, filler words, compound segmentation, anaphora (Stodden and Kallmeyer, 2022) or even changes in number/date formatting (Cardon et al, 2022) but we exclude these groups due to their sparsity and our focus on evaluating performance.…”

Section: A13 Lexical Simplificationmentioning

confidence: 99%

“…An attempted paraphrase where the exact meaning is retained, but the replacement uses more complex semantics (also referred to as a hyponym, e.g. Stodden and Kallmeyer, 2022).…”

Section: A23 Lexical Errorsmentioning

confidence: 99%

“…We include morphological changes (e.g., tense change) as structure edits since these typically require multiple disconnected edits to perform and impact sentence-level meaning. Additionally, other work (Barancikova and Bojar, 2020), specifically Stodden and Kallmeyer (2022) annotate with a larger array of structural changes, notably including separate directions as distinct categories (e.g. singular → plural and plural → singular) and including change in sentiment and personal/impersonal form.…”

Section: B Structural Edit Examplesmentioning

confidence: 99%

See 3 more Smart Citations

Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA

Heineman,

Dou,

Maddela

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Large language models (e.g., GPT-4) are uniquely capable of producing highly rated text simplification, yet current human evaluation methods fail to provide a clear understanding of systems' specific strengths and weaknesses. To address this limitation, we introduce SALSA, an edit-based human annotation framework that enables holistic and fine-grained text simplification evaluation. We develop twenty one linguistically grounded edit types, covering the full spectrum of success and failure across dimensions of conceptual, syntactic and lexical simplicity. Using SALSA, we collect 19K edit annotations on 840 simplifications, revealing discrepancies in the distribution of simplification strategies performed by fine-tuned models, prompted LLMs and humans, and find GPT-3.5 performs more quality edits than humans, but still exhibits frequent errors. Using our finegrained annotations, we develop LENS-SALSA, a reference-free automatic simplification metric, trained to predict sentence-and word-level quality simultaneously. Additionally, we introduce word-level quality estimation for simplification and report promising baseline results. Our data, new metric, and annotation toolkit are available at https://salsa-eval.com.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: A13 Lexical Simplificationmentioning

confidence: 99%

“…An attempted paraphrase where the exact meaning is retained, but the replacement uses more complex semantics (also referred to as a hyponym, e.g. Stodden and Kallmeyer, 2022).…”

Section: A23 Lexical Errorsmentioning

confidence: 99%

Section: B Structural Edit Examplesmentioning

confidence: 99%

See 2 more Smart Citations

Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA

Heineman,

Dou,

Maddela

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…These results have been further improved with a sentence-based approach (Ebling et al, 2022). Most recently, the first detailed surveys about German text simplification have been released (Anschütz et al, 2023;Stodden et al, 2023;Schomacker et al, 2023).…”

Section: Related Workmentioning

confidence: 99%

Language Representation Models: An Overview

Schomacker

Tropmann-Frick

2021

Entropy

View full text Add to dashboard Cite

In the last few decades, text mining has been used to extract knowledge from free texts. Applying neural networks and deep learning to natural language processing (NLP) tasks has led to many accomplishments for real-world language problems over the years. The developments of the last five years have resulted in techniques that have allowed for the practical application of transfer learning in NLP. The advances in the field have been substantial, and the milestone of outperforming human baseline performance based on the general language understanding evaluation has been achieved. This paper implements a targeted literature review to outline, describe, explain, and put into context the crucial techniques that helped achieve this milestone. The research presented here is a targeted review of neural language models that present vital steps towards a general language representation model.

show abstract