An Assessment of Online Semantic Annotators for the Keyword Extraction Task

Semantic Measures for Keywords Extraction

“…Specifically, in Table 1(a) we present the results obtained by comparing the keywords extracted to the top 15 keywords in the Crowd500 dataset, while the results obtained by considering all of the gold standard keywords are provided in Table 1(a). Regarding the first experiment, over the top 15 keywords, we note that in 3 out of 5 of the considered metrics (namely, NASARIE, UCI and UMASS), the F 1 score is higher than those reported in the paper by [6]. Also in the second experiment NASARIE, UCI and UMASS obtained highest F 1 score, whilst the results of NASARI and ttcs E are featured by the highest precision.…”

“…In the following, for the sake of self-containedness, we report the experimental results obtained by [6], where the authors performed a systematic assessment of an array of keyword extractors and online semantic annotators. In particular, we report the results obtained by 2 keyword extractors that participated in the "SemEval-2010 Task 5: Automatic Keyphrase Extraction from Scientific Articles" (namely, KP-Miner [3] and Maui [20]), and 5 semantic annotators (AlchemyAPI, Zemanta, OpenCalais, TagMe, and TextRazor 7 ).…”

“…With regards to Alchemy, both the keyword extraction (Alch Key) and concept tagging (Alch Con) services were considered. More details can be found in [6].…”

“…We adopted the same setting as in [6], where two experiments have been carried out: in the first one the authors restricted to considering the top 15 keywords for each document in the dataset, while in the second one they considered all annotated keywords. Given the diversity of the metrics employed, some of them typically return a centrality score for each concept in the document (NASARIE, UCI, UMASS), while the other ones (NASARI and TTCS) are only able to express a centrality score for some of the concepts in the document.…”

“…The ttcs E is compliant with the Conceptual Spaces, a geometric framework for common-sense knowledge representation and reasoning, and contains a novel mixture of common-sense and encyclopedic knowledge [7,10]. 6 Concept representation and similarity computation with the ttcs E . Let D be the set of N dimensions.…”