Don’t Let Discourse Confine Your Model: Sequence Perturbations for Improved Event Language Models

Abstract

Event language models represent plausible sequences of events. Most existing approaches train autoregressive models on text, which successfully capture event co-occurrence but unfortunately constrain the model to follow the discourse order in which events are presented. Other domains may employ different discourse orders, and for many applications, we may care about different notions of ordering (e.g., temporal) or not care about ordering at all (e.g., when predicting related events in a schema). We propose a simple yet surprisingly effective strategy for improving event language models by perturbing event sequences so we can relax model dependence on text order. Despite generating completely synthetic event orderings, we show that this technique improves the performance of the event language models on both applications and outof-domain events data.

SAGEViz: SchemA GEneration and Visualization

“…The current version of the tool has two event generation models described below, however SAGEViz is designed such that incorporating new models is very easy as long as it uses a similar input/output structure. Question-guided event language model: The first model used in the tool, is a finetuned event language model that takes a set of events and a question about an entity of interest as the context and generates the next event (Koupaee et al, 2023). The users can have control over the entities and can ask the system to generate the events with respect to the desired entities.…”

“…Backend The backend APIs are developed using Flask (Grinberg, 2018), a lightweight Python web framework, which makes it an ideal choice for hosting our Question-guided event language model. We load the language model (Koupaee et al, 2023) and SpanBERT model (for coreference resolution) during application startup. For the GPT-3.5 few-shot event generator we use the OpenAI python library which gives us access to the gpt-3.5-turbo model.…”

“…We aimed to study the effectiveness and efficiency of SAGEViz at generating schemas using the process outlined in Koupaee et al (2023) and compared it with the schema generation interactive system (CLI) used in this study. For a fair comparison, we selected only the event generator component of SAGEViz.…”

“…For each 4-minute interaction, a user started with a seed event as the schema context and a participating entity to use in the question to the event generator, and generated events using the Question Guided Event Language Model (QGELM) (Koupaee et al, 2023). From the generated events, the user selected and added each event that satisfied all 4 of the selection criteria (each is a binary judgment).…”

“…They provide a way to organize and represent how complex events unfold (Schank and Abelson, 1975;Mooney and DeJong, 1985). Schema-based reasoning enables reliable and explainable prediction of next events, inference of missing events or entities (Chambers andJurafsky, 2008, 2009;Manshadi et al, 2008;Chambers, 2013;Balasubramanian et al, 2013;Pichotta and Mooney, 2016;Weber et al, 2018;Koupaee et al, 2021;Rezaee et al, 2021), and drawing connections between events that have already occurred (Kwon et al, 2020). For example, when a disease outbreak happens, it is likely that an investigation into the outbreak and mitigation steps will follow.…”