Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks

Mohiuddin, Tasnim; Jwalapuram, Prathyusha; Lin, Xiang; Joty, Shafiq

doi:10.18653/v1/2021.eacl-main.308

Cited by 4 publications

(9 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On LMVLM, the UNC model has a better performance; we suspect that its explicit conditional language modeling loss might provide an additional advantage for this particular task. Overall, our results are consistent with observations fromMohiuddin et al (2021) that show poor generalizability in the previous SOTA model.…”

supporting

confidence: 91%

“…We report the results obtained by Mohiuddin et al (2021) and Pishdad et al (2020) on their evaluation tasks for SOTA neural coherence models in Table 6. Mesgar & Strube (2018).…”

Section: A5 Comparison Of Existing State-of-the-art Coherence Modelsmentioning

confidence: 99%

“…The Unified Coherence model (Moon et al, 2019) uses bi-linear layer and lightweight convolution-pooling in a Siamese framework to capture discourse relations and topic structures, along with an explicit language model loss to capture syntactic patterns. Mohiuddin et al (2021) recently tested these state-of-the-art models by conducting coherence evaluations on the WSJ permuted document task, machine translation, summarization and next utterance ranking tasks. They found that while models performed well on the permuted document task, when tested off-the-shelf, models generalized poorly to downstream evaluation tasks.…”

Section: Introductionmentioning

confidence: 99%

“…. Results from evaluation of existing coherence models by bothPishdad et al (2020) andMohiuddin et al (2021) indicate that the Unified Coherence model or UNC(Moon et al, 2019) is overall the best-performing model. We retrain their model with our training data for comparison 2 .Results.…”

mentioning

confidence: 99%

“…Results reported byMohiuddin et al (2021) andPishdad et al (2020) on various tasks and datasets that compare the UNC model to two other SOTA neural coherence models proposed by Xu et al (2019) and…”

mentioning

confidence: 99%

See 4 more Smart Citations

Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling

Jwalapuram¹,

Joty²,

Lin³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Although large-scale pre-trained neural models have shown impressive performances in a variety of tasks, their ability to generate coherent text that appropriately models discourse phenomena is harder to evaluate and less understood. Given the claims of improved text generation quality across various systems, we consider the coherence evaluation of machine generated text to be one of the principal applications of coherence models that needs to be investigated. We explore training data and self-supervision objectives that result in a model that generalizes well across tasks and can be used off-the-shelf to perform such evaluations. Prior work in neural coherence modeling has primarily focused on devising new architectures, and trained the model to distinguish coherent and incoherent text through pairwise self-supervision on the permuted documents task. We instead use a basic model architecture and show significant improvements over state of the art within the same training regime. We then design a harder self-supervision objective by increasing the ratio of negative samples within a contrastive learning setup, and enhance the model further through automatic hard negative mining coupled with a large global negative queue encoded by a momentum encoder. We show empirically that increasing the density of negative samples improves the basic model, and using a global negative queue further improves and stabilizes the model while training with hard negative samples. We evaluate the coherence model on task-independent test sets that resemble real-world use cases and show significant improvements in coherence evaluations of downstream applications.

show abstract

supporting

confidence: 91%

“…We report the results obtained by Mohiuddin et al (2021) and Pishdad et al (2020) on their evaluation tasks for SOTA neural coherence models in Table 6. Mesgar & Strube (2018).…”

Section: A5 Comparison Of Existing State-of-the-art Coherence Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling

Jwalapuram¹,

Joty²,

Lin³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

BBScore: A Brownian Bridge Based Metric for Assessing Text Coherence

Sheng,

Zhang,

Jiang

et al. 2024

AAAI

View full text Add to dashboard Cite

Measuring the coherence of text is a vital aspect of evaluating the quality of written content. Recent advancements in neural coherence modeling have demonstrated their efficacy in capturing entity coreference and discourse relations, thereby enhancing coherence evaluation. However, many existing methods heavily depend on static embeddings or focus narrowly on nearby context, constraining their capacity to measure the overarching coherence of long texts. In this paper, we posit that coherent texts inherently manifest a sequential and cohesive interplay among sentences, effectively conveying the central theme, purpose, or standpoint. To explore this abstract relationship, we introduce the "BB Score," a novel reference-free metric grounded in Brownian bridge theory for assessing text coherence. Our findings showcase that when synergized with a simple additional classification component, this metric attains a performance level comparable to state-of-the-art techniques on standard artificial discrimination tasks. We also establish in downstream tasks that this metric effectively differentiates between human-written documents and text generated by large language models within specific domains. Furthermore, we illustrate the efficacy of this approach in detecting written styles attributed to various large language models, underscoring its potential for generalizability. In summary, we present a novel Brownian bridge coherence metric capable of measuring both local and global text coherence, while circumventing the need for end-to-end model training. This flexibility allows for its application in various downstream tasks.

show abstract

Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling

Jwalapuram¹,

Joty²,

Lin

2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Self Cite

View full text Add to dashboard Cite

Given the claims of improved text generation quality across various pre-trained neural models, we consider the coherence evaluation of machine generated text to be one of the principal applications of coherence models that needs to be investigated. Prior work in neural coherence modeling has primarily focused on devising new architectures for solving the permuted document task. We instead use a basic model architecture and show significant improvements over state of the art within the same training regime. We then design a harder self-supervision objective by increasing the ratio of negative samples within a contrastive learning setup, and enhance the model further through automatic hard negative mining coupled with a large global negative queue encoded by a momentum encoder. We show empirically that increasing the density of negative samples improves the basic model, and using a global negative queue further improves and stabilizes the model while training with hard negative samples. We evaluate the coherence model on task-independent test sets that resemble real-world applications and show significant improvements in coherence evaluations of downstream tasks. 1

show abstract

Rethinking Coherence Modeling: Synthetic vs. Downstream Tasks

Cited by 4 publications

References 37 publications

Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling

Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling

BBScore: A Brownian Bridge Based Metric for Assessing Text Coherence

Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling

Contact Info

Product

Resources

About