Machine-Assisted Script Curation

Abstract

We describe Machine-Aided Script Curator (MASC), a system for human-machine collaborative script authoring. Scripts produced with MASC include (1) English descriptions of sub-events that comprise a larger, complex event;(2) event types for each of those events;(3) a record of entities expected to participate in multiple sub-events; and (4) temporal sequencing between the sub-events. MASC automates portions of the script creation process with suggestions for event types, links to Wikidata, and sub-events that may have been forgotten. We illustrate how these automations are useful to the script writer with a few case-study scripts.

Human-in-the-loop Schema Induction

“…In those cases, we showed that human curation can step in timely and improve the quality of event schema when LLM-based models make mistakes 7 . In addition, our interface is easy to use, with much shorter time required to complete each event schema task compared to previous work (Ciosici et al, 2021).…”

“…Another area related to our work is human-in-theloop schema generation, where annotators collaborate with computational models to create highquality event schema. In this field, one of the closest approachs is the Machine-Assisted Script Curation (Ciosici et al, 2021) created for script induction. With a fully interactive interface, they have shown the feasibility of realtime interaction between humans and pre-trained LLMs (e.g.…”

“…We followed the evaluation methodology used by Ciosici et al (2021) with slight modifications to assess our system. Evaluation is done by researchers in the field of NLP who have experience in handwriting event schemas but have not used the interface before.…”

“…We will include this graph view in our next version. Forth, our current evaluation is experimental and probably subjective, we will develop more robust evaluation metrics comparing manual, Ciosici et al (2021)'s and our schema and test on downstream tasks in the next step.…”

“…In fact, the majority of existing approaches under the KAIROS program are fully-automated IR and IE systems over large collections of news articles (Li et al, 2020(Li et al, , 2021. Only some of limited human post-processing on schemas (Ciosici et al, 2021) have been explored. Further discussion of the advantages and limitations of existing systems can be found in Related Work.…”