An extensive review of tools for manual annotation of documents

Abstract

Motivation Annotation tools are applied to build training and test corpora, which are essential for the development and evaluation of new natural language processing algorithms. Further, annotation tools are also used to extract new information for a particular use case. However, owing to the high number of existing annotation tools, finding the one that best fits particular needs is a demanding task that requires searching the scientific literature followed by installing and trying various tools. Methods We searched for annotation tools and selected a subset of them according to five requirements with which they should comply, such as being Web-based or supporting the definition of a schema. We installed the selected tools (when necessary), carried out hands-on experiments and evaluated them using 26 criteria that covered functional and technical aspects. We defined each criterion on three levels of matches and a score for the final evaluation of the tools. Results We evaluated 78 tools and selected the following 15 for a detailed evaluation: BioQRator, brat, Catma, Djangology, ezTag, FLAT, LightTag, MAT, MyMiner, PDFAnno, prodigy, tagtog, TextAE, WAT-SL and WebAnno. Full compliance with our 26 criteria ranged from only 9 up to 20 criteria, which demonstrated that some tools are comprehensive and mature enough to be used on most annotation projects. The highest score of 0.81 was obtained by WebAnno (of a maximum value of 1.0).

A computational ecosystem to support eHealth Knowledge Discovery technologies in Spanish

“…An extensive analysis and comparison of several annotation tools is provided in Neves and Ŝeva [18]. Table 2 summarizes the main characteristics we considered relevant for this research and identifies the most appropriate annotation tool among a subset of popular alternatives.…”

“…Qualitative comparison of popular annotation tools. Adapted fromTable 3in Neves and Ŝeva[18],Table 3. A symbol ≈ indicates that the corresponding feature is only partially supported.…”

An empirical evaluation of electronic annotation tools for Twitter data

“…Prior to the publication of an extensive review of 78 annotation tools by Neve and Seva [12], we started a review of annotation tools for Twitter data. The inclusion criteria for our review were the availability and the ease of installation of the tools, or otherwise, a demonstration of the tool online.…”

Extending TextAE for annotation of non-contiguous entities

“…Many annotation tools have been developed for manual tagging of entities within a document for the biological domain and other domains. A detailed recent review of the strengths and weaknesses of different methods can be found in Neves and Seva's study [4]. To gauge the support for non-contiguous entities, we manually tested the 15 tools selected in that review with an overview shown in Table 2.…”

“…An analysis of the annotation tools reviewed in Neves and Seva's study[4] for their capabilities to annotate non-contiguous entities…”