Event Nugget Annotation: Processes and Issues

Abstract

This paper describes the processes and issues of annotating event nuggets based on DEFT ERE Annotation Guidelines v1.3 and TAC KBP Event Detection Annotation Guidelines 1.7. Using Brat Rapid Annotation Tool (brat), newswire and discussion forum documents were annotated. One of the challenges arising from human annotation of documents is annotators' disagreement about the way of tagging events. We propose using Event Nuggets to help meet the definitions of the specific type/subtypes which are part of this project. We present case studies of several examples of event annotation issues, including discontinuous multi-word events representing single events. Annotation statistics and consistency analysis is provided to characterize the interannotator agreement, considering single term events and multi-word events which are both continuous and discontinuous. Consistency analysis is conducted using a scorer to compare first pass annotated files against adjudicated files.

A Multi-Axis Annotation Scheme for Event Temporal Relations

“…Event modality have been discussed in many existing event annotation schemes, e.g., Event Nugget (Mitamura et al, 2015), Rich ERE , and RED. Generally, an event is classified as Actual or Non-Actual, a.k.a.…”

Unsupervised Event Clustering and Aggregation from Newswire and Web Articles

“…This definition fits with definitions proposed in other contexts such as the ACE 2005 and TAC KBP Event evaluations or work such as (Cybulska and Vossen, 2014;Mitamura et al, 2015), which generally view each event as "something that happens at a particular place and time", implying changes in the state of the world and involving participants. In accordance with ontologies about events such as the Simple Event Model (SEM) ontology (van Hage et al, 2011), events can be categorized into different types, for example "elections" or "earthquakes", gathering multiple real-life instances, for example the "2017 UK General Election" or the "2012 French Presidential Election".…”

One, no one and one hundred thousand events: Defining and processing events in an inter-disciplinary perspective

“…Besides, answers highlighted the importance of states and conditions (I feel that the state/condition is important.). In ACE, ERE and Event Nugget, states that result from actions, such as being dead, married or retired, are included in the annotation, but disagreement is an open issue for human annotators (Mitamura et al, 2015). On the other hand, in TimeML, only states that are temporally relevant (e.g., that are bound to a specific point or period of time) have to be annotated.…”

“…Moreover, the number of attributes is expanded and more attention is devoted to event CO. These DEFT ERE standards are the basis of the novel Event Nugget annotation scheme (Mitamura et al 2015). An event nugget is a semantically meaningful unit referring to an event and linguistically represented not only by a single word but also by a continuous or discontinuous multi-token expression.…”