Assessing the benefits of partial automatic pre-labeling for frame-semantic annotation

Rehbein, Ines; Ruppenhofer, Josef; Sporleder, Caroline

doi:10.3115/1698381.1698384

Cited by 9 publications

(9 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Inaccurate pre-annotations may require deletion or correction, and evidence indicates that time-savings correlate with preannotation accuracy. 25 26 For some tasks, pre-annotation may not alter annotation time, 27 and the presence of multiple, inaccurate pre-annotations may instead increase annotation time. 25 28 Also, pretrained systems capable of pre-annotating for a specific task or medical realm either may not exist or may not be sufficiently accurate when used within a new domain.…”

Section: Introductionmentioning

confidence: 99%

Assisted annotation of medical free text using RapTAT

Gobbel

Garvin

Reeves

et al. 2014

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

show abstract

Section: Introductionmentioning

confidence: 99%

Assisted annotation of medical free text using RapTAT

Gobbel

Garvin

Reeves

et al. 2014

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

show abstract

“…Once each annotator has annotated half of the corpus, pre-annotation has been applied, as a proven way to obtain significant gains in both annotation time and quality of annotation [Fort and Sagot, 2010;Marcus et al, 1993;Rehbein et al, 2009]. Since we were targeting errors of omission (segments missed by the annotators), an "overly-general" CRF model was trained on all 300 documents, and applied on the corpus.…”

Section: Methodsmentioning

confidence: 99%

Argument Extraction from News

Sardianos

Katakis²,

Petasis

et al. 2015

Proceedings of the 2nd Workshop on Argumentation Mining

View full text Add to dashboard Cite

Argument extraction is the task of identifying arguments, along with their components in text. Arguments can be usually decomposed into a claim and one or more premises justifying it. The proposed approach tries to identify segments that represent argument elements (claims and premises) on social Web texts (mainly news and blogs) in the Greek language, for a small set of thematic domains, including articles on politics, economics, culture, various social issues, and sports. The proposed approach exploits distributed representations of words, extracted from a large non-annotated corpus. Among the novel aspects of this work is the thematic domain itself which relates to social Web, in contrast to traditional research in the area, which concentrates mainly on law documents and scientific publications. The huge increase of social web communities, along with their user tendency to debate, makes the identification of arguments in these texts a necessity. In addition, a new manually annotated corpus has been constructed that can be used freely for research purposes. Evaluation results are quite promising, suggesting that distributed representations can contribute positively to the task of argument extraction.

show abstract

“…Rehbein et al [31] performed quite thorough experiments in the field of semantic frame assignment annotation. Although in this case, the results of the experiments are a bit disappointing as they could not find a direct improvement of annotation time using pre-annotation, they found that noisy and low-quality pre-annotation does not overall corrupt human judgment.…”

Section: Pre-annotation Techniquementioning

confidence: 99%

EmoLabel: Semi-Automatic Methodology for Emotion Annotation of Social Media Text

Canales

Daelemans

Boldrini

et al. 2022

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

The exponential growth of the amount of subjective information on the Web 2.0. has caused an increasing interest from researchers willing to develop methods to extract emotion data from these new sources. One of the most important challenges in textual emotion detection is the gathering of data with emotion labels because of the subjectivity of assigning these labels. Basing on this rationale, the main objective of our research is to contribute to the resolution of this important challenge. This is tackled by proposing EmoLabel: a semi-automatic methodology based on pre-annotation, which consists of two main phases: (1) an automatic process to pre-annotate the unlabelled English sentences; and (2) a manual process of refinement where human annotators determine which is the dominant emotion. Our objective is to assess the influence of this automatic pre-annotation method on manual emotion annotation from two points of view: agreement and time needed for annotation. The evaluation performed demonstrates the benefits of pre-annotation processes since the results on annotation time show a gain of near 20% when the pre-annotation process is applied (Pre-ML) without reducing annotator performance. Moreover, the benefits of pre-annotation are higher in those contributors whose performance is low (inaccurate annotators).

show abstract

Assessing the benefits of partial automatic pre-labeling for frame-semantic annotation

Abstract: In this paper, we present the results of an experiment in which we assess the usefulness of partial semi-automatic annotation for frame labeling. While we found no conclusive evidence that it can speed up human annotation, automatic pre-annotation does increase its overall quality.

Cited by 9 publications

References 9 publications

Assisted annotation of medical free text using RapTAT

Assisted annotation of medical free text using RapTAT

Argument Extraction from News

EmoLabel: Semi-Automatic Methodology for Emotion Annotation of Social Media Text

Contact Info

Product

Resources

About