Embedding Time Expressions for Deep Temporal Ordering Models

Abstract

Data-driven models have demonstrated stateof-the-art performance in inferring the temporal ordering of events in text. However, these models often overlook explicit temporal signals, such as dates and time windows. Rule-based methods can be used to identify the temporal links between these time expressions (timexes), but they fail to capture timexes' interactions with events and are hard to integrate with the distributed representations of neural net models. In this paper, we introduce a framework to infuse temporal awareness into such models by learning a pre-trained model to embed timexes. We generate synthetic data consisting of pairs of timexes, then train a character LSTM to learn embeddings and classify the timexes' temporal relation. We evaluate the utility of these embeddings in the context of a strong neural model for event temporal ordering, and show a small increase in performance on the MATRES dataset and more substantial gains on an automatically collected dataset with more frequent event-timex interactions. 1

Severing the Edge Between Before and After: Neural Architectures for Temporal Ordering of Events

“…The MATRES dataset is our primary dataset for training and validation. As in previous work, we use TimeBank and AQUAINT (256 articles) for training, 25 articles of which are selected at random for validation and Platinum (20 articles) as a held-out test set (Ning et al, 2018;Goyal and Durrett, 2019;Ning et al, 2019). Articles from TimeBank and AQUAINT at full length are about 400 tokens long on average.…”

“…The sigmoid and exponent schedulers perform better than the constant scheduler, suggesting that the model needs to first learn about temporality, and then learn to be more specialized on predicting temporal ordering relations later. We believe this timex multi-tasking setup to be an implicit yet effective way to teach our model about timexes in general without timex embeddings used in (Goyal and Durrett, 2019). When we use the ACE relation extraction dataset as an auxiliary task, none of the schedulers produce improvements while the sigmoid and exponent scheduler fare significantly worse.…”

“…We also use a closer and complementary temporal annotation dataset, i.e. the Timebank and Aquaint annotations involving timex relations (timex-event, event-timex, timex-timex) (Ning et al, 2018;Goyal and Durrett, 2019). 6 We expect the model to greatly benefit from being exposed to the timex relations in an MTL framework by learning about temporality in general and by adding specificity of the event-event temporal relations from the MATRES annotations.…”

“…In this paper, we present a set of neural architectures for temporal ordering of events. Our main model (Section 2) is similar to the temporal ordering models designed by Goyal and Durrett (2019), Liu et al (2019a) and Ning et al (2019).…”

“…Given the scarcity of training data, (2) we explore the application of an existing framework for Scheduled Multitask-Learning (henceforth SMTL) (Kiperwasser and Ballesteros, 2018) by leveraging complementary (temporal and non temporal) information to our models; this imitates pretraining and finetuning. This consumes timex information in a different way than Goyal and Durrett (2019). (3) A self-training method that incorporates the predictions of our model and learns from them; we test it jointly with the SMTL method.…”