Evaluation of cutoff policies for term extraction

Abstract

Background: This paper presents a policy to choose cutoff points to identify potentially relevant terms in a given domain. Term extraction methods usually generate term lists ordered according to a relevance criteria, and the literature is abundant to offer different relevance indices. However, very few studies turn their attention to how many terms should be kept, i.e., to a cutoff policy. Methods: Our proposed policy provides an estimation of the portion of this list which preserves a good balance between recall and precision, adopting a refined term extraction and tf-dcf relevance index. Results: A practical study was conducted based on terms extracted from a Brazilian Portuguese corpus, and the results were quantitatively analyzed according to a previously defined reference list. Conclusions: Even thou different extraction procedures and different relevance indices could brought a different outcome, our policy seems to deliver a good balance for the method adopted in our experiments and it is likely to be able to be generalized to other methods. Background Automatic identification of relevant terms for a given domain is an extremely important task for a myriad of natural language processing applications. For instance, any ontology learning effort is doomed to fail if the concept identification step has a poor performance. In fact, any other steps to automatically build an ontology rely on the concept identification [1-4]. Also, text categorization applications can be much more effective if a good relevant term identification is available. An important part in the process of identifying relevant terms is the extraction of terms and the computation of their frequencies of use as term relevance index. For the term extraction itself, many software tools are available [5, 6]. These tools usually offer high-quality extraction, regardless being implemented on the basis of linguistic or statistical approaches. As for relevance indices, many theoretical formulations are available [7-11] and it is safe to assume that a reliable relevance-based rank of extracted terms is not difficult to obtain. Unfortunately, even assuming a nearly perfect ranked list of terms, it is still difficult to decide how many relevant

Analysing the Impact of Supervised Machine Learning on Automatic Term Extraction: HAMLET vs TermoStat

“…This has been a standard methodology for some time (Daille, 1994) and is still used by state-ofthe-art systems such as TermoStat (Drouin, 2003) and TExSIS (Macken et al, 2013). However, the problem with these methodologies is determining the cut-off point (Lopes and Vieira, 2015) and combining multiple features (e.g., separate measures for termhood and unithood). It has become clear that multiple evidence (i.e.…”

Approaches, tools, algorithms, and methods for automatic term extraction: A systematic literature mapping

“…Approaches, tools, algorithms, and methods for automatic term extraction: A systematic literature mapping 2015 Gaizauskas et al (2015), Khumalo (2015), Saneifar et al (2015), Gupta (2015), Pan and Zhao (2015), Kochetkova (2015), Lopes and Vieira (2015), Periñán-Pascual (2015), Gonçalves et al (2015), Guo et al (2015), Lahbib et al (2015), Astrakhantsev et al (2015), Bakar et al (2015), Liu et al (2015) 12,39% 2016…”

Semantics-based approach for generating partial views from linked life-cycle highway project data

“…Manual evaluation of the entire sorted list would avoid the removal of real terms with low C-values, but it might be too laborious especially for large corpora. To minimize both laborious work and the number of true terms wrongly discarded, this study adopts a relative cut-o policy proposed by Lopes and Vieira (2015) which is based on the optimal trade-o point between wrong discard of true domain terms and wrong inclusion of irrelevant ones. The policy suggests that the bottom 85% of the ranked list should be discarded.…”