Large-Scale Evaluation of Keyphrase Extraction Models

Abstract

Keyphrase extraction models are usually evaluated under different, not directly comparable, experimental setups. As a result, it remains unclear how well proposed models actually perform, and how they compare to each other. In this work, we address this issue by presenting a systematic large-scale analysis of state-ofthe-art keyphrase extraction models involving multiple benchmark datasets from various sources and domains. Our main results reveal that state-of-the-art models are in fact still challenged by simple baselines on some datasets. We also present new insights about the impact of using author-or reader-assigned keyphrases as a proxy for gold standard, and give recommendations for strong baselines and reliable benchmark datasets.

WikiCSSH: Extracting Computer Science Subject Headings from Wikipedia

“…However, this test set still allow for evaluation of how much important information these vocabularies can extract. Our performance evaluation uses a widely used exact match evaluation method [3,21], where only the extract matches with the gold standard are considered as true positives. Specifically, P recision = N umber of M atched in Abstract i N umber of Extracted in Abstract i , Recall = N umber of M atched in Abstract i N umber of Annotated in Abstract i .…”

TNT-KID: Transformer-based neural tagger for keyword identification

“…We follow these guidelines strictly, when it comes to the use of identical datasets and gold-standard keyword sets, but somewhat deviate from them when it comes to the employment of identical preprocessing techniques and parameter settings employed for different approaches. Since all unsupervised approaches operate on a set of keyphrase candidates, extracted from the input document, Gallina et al (2020) argues that the extraction of these candidates and other parameters should be identical (e.g., they select the sequences of adjacent nouns with one or more preceding adjectives of length up to five words in order to extract keyword candidates) for a fair comparison between algorithms. On the other hand, we are more interested in comparison between keyword extraction approaches instead of algorithms alone and argue that the distinct keyword candidate extraction techniques are inseparable from the overall approach and should arguably be optimized for each distinct algorithm.…”

Keyphrase extraction techniques

“…Πολλές διαδικασίες διαφόρων πεδίων μπορούν να ωφεληθούν από την επιτυχημένη εξαγωγή λέξεων ή φράσεων κλειδιών, όπως η ομαδοποίηση κειμένων (document clustering) (Shubankar et al, 2011;Kim and Gil, 2019;Karpagam and Saradha, 2019), η κατηγοριοποίηση/ταξινόμηση κειμένων (text classification) (Hulth and Megyesi, 2006;Meng et al, 2019), προβλήματα ανάκτησης πληροφορίας (information retrieval) (Ji et al, 2019;Boudin et al, 2020), όπως η διεύρυνση ερωτημάτων (query expansion) (Song et al, 2006) και η πολυσύνθετη αναζήτηση (faceted search) , η εξαγωγή περίληψης από κείμενα (text summarization) (Zhang et al, 2004;Litvak and Last, 2008;Song et al, 2019), η αναγνώριση οντοτήτων (entity recognition) (Du et al, 2018) και ο εντοπισμός γεγονότων (event detection) (Hossny et al, 2020). Ο σημαντικός ρόλος των φράσεων κλειδιών σε μεθόδους διαφόρων πεδίων (όπως οι παραπάνω) σε συνδυασμό με την αύξηση της ποσότητας της ψηφιακής πληροφορίας κειμένου στο Διαδίκτυο (διαδικτυακές ψηφιακές βιβλιοθήκες, ηλεκτρονικές εφημερίδες, διαδικτυακά περιοδικά, κριτικές πελατών σε πλατφόρμες ηλεκτρονικού εμπορίου, κ.α.)…”

“…Several tasks can benefit from accurate keyword or keyphrase extraction outcomes, including document clustering (Shubankar et al, 2011;Kim and Gil, 2019;Karpagam and Saradha, 2019), text classification (Hulth and Megyesi, 2006;Meng et al, 2019), information retrieval tasks (Ji et al, 2019;Boudin et al, 2020), such as query expansion (Song et al, 2006) and faceted search ), text summarization (Zhang et al, 2004;Litvak and Last, 2008;Song et al, 2019), entity recognition (Du et al, 2018), and event detection (Hossny et al, 2020). The crucial role of keyphrases in these tasks along with the increasing online digital textual information (e.g., scientific digital libraries, e-newspapers, online magazines, customer reviews on e-commerce platforms, blogs, etc.)…”

“…The crucial role of keyphrases in these tasks along with the increasing online digital textual information (e.g., scientific digital libraries, e-newspapers, online magazines, customer reviews on e-commerce platforms, blogs, etc.) has excited the interest of both the research community (Nasar et al, 2019;Çano and Bojar, 2019;Vega-Oliveros et al, 2019;Firoozeh et al, 2020;Gallina et al, 2020;Papagiannopoulou and Tsoumakas, 2020;Merrouni et al, 2019) and companies. Automatic keyphrase extraction has been used to improve customer service and monitor time-evolving product opportunities in real-time (Choi et al, 2020).…”