Small but Mighty: New Benchmarks for Split and Rephrase

Abstract

Split and Rephrase is a text simplification task of rewriting a complex sentence into simpler ones. As a relatively new task, it is paramount to ensure the soundness of its evaluation benchmark and metric. We find that the widely used benchmark dataset universally contains easily exploitable syntactic cues caused by its automatic generation process. Taking advantage of such cues, we show that even a simple rule-based model can perform on par with the state-of-the-art model. To remedy such limitations, we collect and release two crowdsourced benchmark datasets. We not only make sure that they contain significantly more diverse syntax, but also carefully control for their quality according to a welldefined set of criteria. While no satisfactory automatic metric exists, we apply fine-grained manual evaluation based on these criteria using crowdsourcing, showing that our datasets better represent the task and are significantly more challenging for the models. 1 * Work done during internship at IBM Research.

BiSECT: Learning to Split and Rephrase Sentences with Bitexts

“…When compared with WIK-ISPLIT, BISECT contains significantly more highquality pairs, while containing fewer pairs with significant errors. Pairs containing unsupported and deleted details are comparable across corpora, though WIKISPLIT skews more towards adding unsupported information, which is consistent with previous work (Zhang et al, 2020a). Moreover, we take 100 random samples from the German BISECT corpus and perform manual inspection.…”

“…Thus, proposed the Split and Rephrase task, and introduced the WEBSPLIT corpus, created by aligning sentences in WebNLG . WEBSPLIT contains duplicate instances and phrasal repetitions (Aharoni and Goldberg, 2018;Botha et al, 2018), and most splitting operations can be trivially classified (Zhang et al, 2020a), so subsequent Split and Rephrase corpora have been created to improve training (Botha et al, 2018) and evaluation (Sulem et al, 2018;Zhang et al, 2020a). The main work we compare against is WIKISPLIT, a corpus created by extracting split sentences from Wikipedia edit histories (Botha et al, 2018).…”

“…Besides corpus size, we are interested in the amount of rephrasing (indicated by %new) and the syntactic complexity of sentences (approximated by length). In Table 2, we compare BISECT with previous split and rephrase corpora, including WIKISPLIT (Botha et al, 2018), WEB-SPLIT Aharoni and Goldberg, 2018), HSplit-Wiki (Sulem et al, 2018), Contract and Wiki-BM (Zhang et al, 2020a). BISECT is comparable in size with WIKISPLIT, while impor- We compute the number of aligned pairs (#pairs); number of unique long sentences l (#unique); the percentage of new words added to s compared to l (%new), and the average token Length of l and that of the individual split sentences.…”

“…introduced the WEBSPLIT corpus based on decomposing a long sentence into RDF triples (a form of semantic representation), and generating shorter sentences from subsets of these triples. However, the reliance on RDF triples and a limited vocabulary results in unnatural expressions (Botha et al, 2018) and repeated syntactic patterns (Zhang et al, 2020a).…”

Controllable Text Simplification with Explicit Paraphrasing

“…fluency and adequacy ratings with binary questions described in Zhang et al (2020a) for the second evaluation over another 100 simplifications from the NEWSELA-AUTO split-focused test set. We asked if the output sentence exhibits spitting and if the splitting occurs at the correct place.…”