ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning

Han, Rujun; Ren, Xiang; Peng, Nanyun

doi:10.18653/v1/2021.emnlp-main.436

Cited by 20 publications

(17 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some methods focus on more specific aspects of events and their correlations. DEER (Han et al, 2020b) performs temporal and event masking predictions for temporal relations. Lin et al (2021) propose to recover a temporally-disordered or event-missing sequence for temporal and causal relations.…”

Section: Related Workmentioning

confidence: 99%

ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification

Zhou¹,

Shen²,

Geng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Generating new events given context with correlated ones plays a crucial role in many eventcentric reasoning tasks. Existing works either limit their scope to specific scenarios or overlook event-level correlations. In this paper, we propose to pre-train a general Correlationaware context-to-Event Transformer (ClarET) for event-centric reasoning. To achieve this, we propose three novel event-centric objectives, i.e., whole event recovering, contrastive event-correlation encoding and prompt-based event locating, which highlight event-level correlations with effective training. The proposed ClarET is applicable to a wide range of eventcentric reasoning scenarios, considering its versatility of (i) event-correlation types (e.g., causal, temporal, contrast), (ii) application formulations (i.e., generation and classification), and (iii) reasoning types (e.g., abductive, counterfactual and ending reasoning). Empirical fine-tuning results, as well as zero-and fewshot learning, on 9 benchmarks (5 generation and 4 classification tasks covering 4 reasoning types with diverse event correlations), verify its effectiveness and generalization ability.

show abstract

Section: Related Workmentioning

confidence: 99%

ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification

Zhou¹,

Shen²,

Geng³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…As previously mentioned, the direction of an entitytrace-induced linkage is determined by the narrated order of text segments within contexts, however, in circumstances such as the fourth row in Table 1, the narrative order can be inconsistent with the actual temporal order of the associated events. To alleviate such inconsistency, we apply an event temporal relation prediction model (Han et al, 2021b) to fix the linkage directions. 7 The utilized model predicts 7 These do not include linkages decided by the keywords.…”

Section: Incorporating Temporal Relationsmentioning

confidence: 99%

“…preconditions should occur prior to an action), the narrated order of events may not be consistent with their actual temporal order. We thus adopt a temporal relation resolution module (Han et al, 2021b) to alleviate such an issue.…”

Section: Introductionmentioning

confidence: 99%

Learning Action Conditions from Instructional Manuals for Instruction Understanding

Wu¹,

Zhang²,

Hu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

The ability to infer pre-and postconditions of an action is vital for comprehending complex instructions, and is essential for applications such as autonomous instruction-guided agents and assistive AI that supports humans to perform physical tasks. In this work, we propose a task dubbed action condition inference, and collecting a high-quality, human annotated dataset of preconditions and postconditions of actions in instructional manuals. We propose a weakly supervised approach to automatically construct large-scale training instances from online instructional manuals, and curate a densely human-annotated and validated dataset to study how well the current NLP models can infer action-condition dependencies in the instruction texts. We design two types of models differ by whether contextualized and global information is leveraged, as well as various combinations of heuristics to construct the weak supervisions. Our experimental results show a >20% F1-score improvement with considering the entire instruction contexts and a > 6% F1-score benefit with the proposed heuristics.

show abstract

“…Second, some other approaches mine event information in raw corpus instead of the KGs. DEER (Han et al, 2020) performs continual pre-training via temporal and event masking predictions, i.e., new masking schemes for MLM, to focus on event temporal relations. Lin et al (2020) leverage BART structure and propose to recover a temporally-disordered or event-missing sequence to the original one by a denoising autoencoder.…”

Section: Commonsense-centricmentioning

confidence: 99%

“…Limitation. First, EventBERT is focused on correlation-based event reasoning, and not general enough to every event correlation reasoning task (e.g., event temporal reasoning as in (Han et al, 2020;Lin et al, 2020)). Second, we evaluate EventBERT on deterministic tasks, e.g., multichoice and Cloze-type question answering, due to their stable metrics and widely-available baselines.…”

Section: Case Study Error Analysis and Limitationmentioning

confidence: 99%

EventBERT: A Pre-Trained Model for Event Correlation Reasoning

Zhou¹,

Geng²,

Shen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Event correlation reasoning infers whether a natural language paragraph containing multiple events conforms to human common sense. For example, "Andrew was very drowsy, so he took a long nap, and now he is very alert" is sound and reasonable. In contrast, "Andrew was very drowsy, so he stayed up a long time, now he is very alert" does not comply with human common sense. Such reasoning capability is essential for many downstream tasks, such as script reasoning, abductive reasoning, narrative incoherence, story cloze test, etc. However, conducting event correlation reasoning is challenging due to a lack of large amounts of diverse event-based knowledge and difficulty in capturing correlation among multiple events. In this paper, we propose EventBERT, a pre-trained model to encapsulate eventuality knowledge from unlabeled text. Specifically, we collect a large volume of training examples by identifying natural language paragraphs that describe multiple correlated events and further extracting event spans in an unsupervised manner. We then propose three novel event-and correlation-based learning objectives to pre-train an event correlation model on our created training corpus. Empirical results show EventBERT outperforms strong baselines on four downstream tasks, and achieves SoTA results on most of them. Besides, it outperforms existing pre-trained models by a large margin, e.g., 6.5∼23%, in zero-shot learning of these tasks.

show abstract

ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning

Cited by 20 publications

References 29 publications

ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification

ClarET: Pre-training a Correlation-Aware Context-To-Event Transformer for Event-Centric Generation and Classification

Learning Action Conditions from Instructional Manuals for Instruction Understanding

EventBERT: A Pre-Trained Model for Event Correlation Reasoning

Contact Info

Product

Resources

About