Human evaluation of Kea, an automatic keyphrasing system

Abstract

This paper describes an evaluation of the Kea automatic keyphrase extraction algorithm. Tools that automatically identify keyphrases are desirable because document keyphrases have numerous applications in digital library systems, but are costly and time consuming to manually assign. Keyphrase extraction algorithms are usually evaluated by comparison to author-specified keywords, but this methodology has several well-known shortcomings. The results presented in this paper are based on subjective evaluations of the quality and appropriateness of keyphrases by human assessors, and make a number of contributions. First, they validate previous evaluations of Kea that rely on author keywords. Second, they show Kea's performance is comparable to that of similar systems that have been evaluated by human assessors. Finally, they justify the use of author keyphrases as a performance metric by showing that authors generally choose good keywords.

Automatic Web Service Tagging Using Machine Learning and WordNet Synsets

“…This phenomenon has also been observed in several human evaluations of Kea [16,17], that inspired our approach. It occurs because tags assigned by the users are not the absolute truth.…”

Towards an NLP-Based Topic Characterization of Social Relations

“…However, Turney [17] remarks that a particular document might be represented equally well by more than one set of keywords, and recommends having the output of the system rated by human judges. This is consistent with the results of Jones and Paynter [18], who find a statistically significant agreement on the quality of keywords between different human assessors. Barker and Cornacchia [19] attribute this effect to keyword coherence: "Judges did not prefer keyphrase sets based simply on the individual keyphrases they contained.…”

Adding Lexical Chain to Keyphrase Extraction

“…Second, authors rarely provide more than a few keyphrases-far fewer than may be extracted automatically. Fourth, authors' keyphrases are available for a limited number and type of documents [19]. We will do more research on looking for a more scientific and objective way to evaluate the automatic extraction result.…”