Underreporting of errors in NLG output, and what to do about it

van Miltenburg, Emiel; Clinciu, Miruna; Dušek, Ondřej; Gkatzia, Dimitra; Inglis, Stephanie; Leppänen, Leo; Mahamood, Saad; Manning, Emma; Schoch, Stephanie; Thomson, Craig; Wen, Luou

doi:10.18653/v1/2021.inlg-1.14

Cited by 4 publications

(4 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Evaluation in low-resource NLG In addition to the specific challenges and mitigation strategies for system development above, evaluation has its own challenges in the low-resource setting and is a promising direction for future work in itself. For instance, having less validation and test data reduces the applicability of automated, referencebased evaluations, necessitating alternative evaluation strategies such as an emphasis on error analysis (van Miltenburg et al, 2021) or standardised human evaluations (Howcroft et al, 2020). Methods for maximising the efficiency of input from domain and language experts will also be necessary for human evaluations when access to these persons is more limited than usual.…”

Section: Discussion and Promising Directionsmentioning

confidence: 99%

Most NLG is Low-Resource: here’s what we can do about it

Howcroft¹,

Gkatzia²

2022

Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)

View full text Add to dashboard Cite

Many domains and tasks in natural language generation (NLG) are inherently 'lowresource', where training data, tools and linguistic analyses are scarce. This poses a particular challenge to researchers and system developers in the era of machine-learning-driven NLG. In this position paper, we initially present the challenges researchers & developers often encounter when dealing with low-resource settings in NLG. We then argue that it is unsustainable to collect large aligned datasets or build large language models from scratch for every possible domain due to cost, labour, and time constraints, so researching and developing methods and resources for low-resource settings is vital. We then discuss current approaches to low-resource NLG, followed by proposed solutions and promising avenues for future work in NLG for low-resource settings.

show abstract

Section: Discussion and Promising Directionsmentioning

confidence: 99%

Most NLG is Low-Resource: here’s what we can do about it

Howcroft¹,

Gkatzia²

2022

Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)

View full text Add to dashboard Cite

show abstract

“…For a detailed discussion of good practices in error analysis, see e.g. van Miltenburg et al (2021).…”

Section: Limitationsmentioning

confidence: 99%

Introducing VULCAN: A Visualization Tool for Understanding Our Models and Data by Example

Groschwitz

2023

Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP

View full text Add to dashboard Cite

Examples are a powerful tool that help us understand complex concepts and connections. In computational linguistics research, looking at example system output and example corpus entries can offer a wealth of insights that are not otherwise accessible. This paper describes the open-source software VULCAN, a visualization tool for strings, graphs, trees, alignments, attention and more. VULCAN's unique ability to visualize both linguistic structures and properties of neural models make it particularly relevant for neuro-symbolic models. Neurosymbolic models, combining neural networks with often linguistically grounded structures, offer a promise of increased interpretability in an age of purely neural black-box end-to-end models. VULCAN aims to facilitate this interpretability in practice. VULCAN is designed to be both easy to use and powerful in its capabilities.

show abstract

“…sign choices. A suitable interface can also encourage researchers to step away from unreliable automatic metrics (Gehrmann et al, 2022) and focus on manual error analysis (van Miltenburg et al, 2021(van Miltenburg et al, , 2023.…”

Section: Web Interfacementioning

confidence: 99%

TabGenie: A Toolkit for Table-to-Text Generation

Kasner¹,

Garanina²,

Plátek³

et al. 2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)

View full text Add to dashboard Cite

Heterogenity of data-to-text generation datasets limits the research on data-to-text generation systems. We present TABGENIE -a toolkit which enables researchers to explore, preprocess, and analyze a variety of data-to-text generation datasets through the unified framework of table-to-text generation. In TABGENIE, all inputs are represented as tables with associated metadata. The tables can be explored through a web interface, which also provides an interactive mode for debugging table-to-text generation, facilitates side-by-side comparison of generated system outputs, and allows easy exports for manual analysis. Furthermore, TAB-GENIE is equipped with command line processing tools and Python bindings for unified dataset loading and processing. We release TABGENIE as a PyPI package 1 and provide its open-source code and a live demo at https: //github.com/kasnerz/tabgenie.

show abstract

Underreporting of errors in NLG output, and what to do about it

Cited by 4 publications

References 41 publications

Most NLG is Low-Resource: here’s what we can do about it

Most NLG is Low-Resource: here’s what we can do about it

Introducing VULCAN: A Visualization Tool for Understanding Our Models and Data by Example

TabGenie: A Toolkit for Table-to-Text Generation

Contact Info

Product

Resources

About