Evaluation of machine learning-based information extraction algorithms: criticisms and recommendations

Abstract

We survey the evaluation methodology adopted in information extraction (IE), as defined in a few different efforts applying machine learning (ML) to IE. We identify a number of critical issues that hamper comparison of the results obtained by different researchers. Some of these issues are common to other NLP-related tasks: e.g., the difficulty of exactly identifying the effects on performance of the data (sample selection and sample size), of the domain theory (features selected), and of algorithm parameter settings. Some issues are specific to IE: how leniently to assess inexact identification of filler boundaries, the possibility of multiple fillers for a slot, and how the counting is performed. We argue that, when specifying an IE task, these issues should be explicitly addressed, and a number of methodological characteristics should be clearly defined. To empirically verify the practical impact of the issues mentioned above, we perform a survey of the results of different algorithms when applied to a few standard datasets. The survey shows a serious lack of consensus on these issues, which makes it difficult to draw firm conclusions on a comparative evaluation of the algorithms. Our aim is to elaborate a clear and detailed experimental methodology and propose it to the IE community. Widespread agreement on this proposal should lead to future IE comparative evaluations that are fair and reliable. To demonstrate the way the methodology is to be applied we have organized and run a comparative evaluation of ML-based IE systems (the Pascal Challenge on ML-based IE) where the principles described in this article are put into practice. In this article we describe the proposed methodology and its motivations. The Pascal evaluation is then described and its results presented.

Enhancing Opinion Extraction by Automatically Annotated Lexical Resources

“…The typical approach (Lavelli et al, 2008) is to define a predicate match (g, p), with values in {T rue, F alse}, which determines if there is a match between two annotations g ∈ G and p ∈ P , and then use this predicate to compute an approximate version of precision (π) and recall (ρ):…”

Evaluating Information Extraction

“…is obtained by (i) computing the category-specific values T P i , F P i and F N i , (ii) obtaining T P as the sum of the T P i 's (same for F P and F N), and then (iii) applying Equation (1). F M 1 is instead obtained by first computing the categoryspecific F 1 values and then averaging them across the c i 's.…”

“…A recent review paper on the evaluation of IE systems [1], while discussing in detail other undoubtedly important evaluation issues (such as datasets, training set / test set splits, and evaluation campaigns), devotes surprisingly little space to discussing the mathematical measures used in evaluating IE systems; and the same happens for a recent survey on information extraction methods and systems [2]. That the issue is far from solved is witnessed by a long discussion 1 , appeared on a popular NLP-related blog, in which prominent members of the NLP community voice their discontent with the evaluation measures currently used in the IE literature, and come to the conclusion that no satisfactory measure has been found yet.…”

An architecture for web information agents

“…Rule learning algorithms can be trained to achieve higher precision and recall, but using cross validation over a data set might give us an estimation of how an algorithm will behave in practice. Despite its importance, the majority of proposals in literature do not use this validation method [17].…”