Efficiency Metrics for Data-Driven Models: A Text Summarization Case Study

Çano, Erion; Bojar, Ondřej

doi:10.18653/v1/w19-8630

Cited by 2 publications

(3 citation statements)

References 27 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experimenting with keyphrases of scientific papers seems an ongoing trend that is greatly motivated by the availability of data in online academic repositories. Following the examples of [59] and [60], we took the initiative to produce an even larger collection of scientific paper keywords, titles and abstracts. Exploiting the whole data of Open Academic Graph (described in [61] and [62]), we retrieved keywords, title and abstract data wherever they were available.…”

Section: B a Novel And Huge Data Collectionmentioning

confidence: 99%

Keyphrase Generation: A Multi-Aspect Survey

Çano

Bojar

2019

2019 25th Conference of Open Innovations Association (FRUCT)

View full text Add to dashboard Cite

Extractive keyphrase generation research has been around since the nineties, but the more advanced abstractive approach based on the encoder-decoder framework and sequenceto-sequence learning has been explored only recently. In fact, more than a dozen of abstractive methods have been proposed in the last three years, producing meaningful keyphrases and achieving state-of-the-art scores. In this survey, we examine various aspects of the extractive keyphrase generation methods and focus mostly on the more recent abstractive methods that are based on neural networks. We pay particular attention to the mechanisms that have driven the perfection of the later. A huge collection of scientific article metadata and the corresponding keyphrases is created and released for the research community. We also present various keyphrase generation and text summarization research patterns and trends of the last two decades.

show abstract

Section: B a Novel And Huge Data Collectionmentioning

confidence: 99%

Keyphrase Generation: A Multi-Aspect Survey

Çano

Bojar

2019

2019 25th Conference of Open Innovations Association (FRUCT)

View full text Add to dashboard Cite

show abstract

“…It is useful to define a metric that measures accuracy loss per utterance, relative to the no-pruning baseline. We report relative data-score efficiency, originally proposed by Çano and Bojar (2019):…”

Section: Experiments Metricsmentioning

confidence: 99%

“…In Çano and Bojar (2019), the relative data-score efficiency metric σ was used to evaluate how well model accuracy scaled as the amount of training data was increased. We use use σ in a slightly different but analogous manner to quantify how well model errors are avoided as the training data is pruned using a given sampling technique.…”

mentioning

confidence: 99%

Influence Scores at Scale for Efficient Language Data Sampling

Anand,

Tan,

Minakova

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Modern ML systems ingest data aggregated from diverse sources, such as synthetic, humanannotated, and live customer traffic. Understanding which examples are important to the performance of a learning algorithm is crucial for efficient model training. Recently, a growing body of literature has given rise to various "influence scores," which use training artifacts such as model confidence or checkpointed gradients to identify important subsets of data. However, these methods have primarily been developed in computer vision settings, and it remains unclear how well they generalize to language-based tasks using pretrained models.In this paper, we explore the applicability of influence scores in language classification tasks. We evaluate a diverse subset of these scores on the SNLI dataset by quantifying accuracy changes in response to pruning training data through random and influence-score-based sampling. We then stress-test one of the scores -"variance of gradients" (VoG) from Agarwal and Hooker (2022) -in an NLU model stack that was exposed to dynamic user speech patterns in a voice assistant type of setting. Our experiments demonstrate that in many cases, encoder-based language models can be finetuned on roughly 50% of the original data without degradation in performance metrics. Along the way, we summarize lessons learned from applying out-of-the-box implementations of influence scores, quantify the effects of noisy and class-imbalanced data, and offer recommendations on score-based sampling for better accuracy and training efficiency.

show abstract

Efficiency Metrics for Data-Driven Models: A Text Summarization Case Study

Cited by 2 publications

References 27 publications

Keyphrase Generation: A Multi-Aspect Survey

Keyphrase Generation: A Multi-Aspect Survey

Influence Scores at Scale for Efficient Language Data Sampling

Contact Info

Product

Resources

About