Evaluating Topic Quality with Posterior Variability

Abstract

Probabilistic topic models such as latent Dirichlet allocation (LDA) are popularly used with Bayesian inference methods such as Gibbs sampling to learn posterior distributions over topic model parameters. We derive a novel measure of LDA topic quality using the variability of the posterior distributions. Compared to several existing baselines for automatic topic evaluation, the proposed metric achieves state-of-the-art correlations with human judgments of topic quality in experiments on three corpora. 1 We additionally demonstrate that topic quality estimation can be further improved using a supervised estimator that combines multiple metrics.

Ranking coherence in topic models using statistically validated networks

“…We evaluated our estimator of topic quality on a dataset of articles extracted from the New York Times, which was already analysed by 47 .…”

Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling

“…Unsupervised methods typically design features based on the assumption that segments in the same topic are more coherent than those that belong to different topics, such as lexical cohesion (Hearst, 1997;Choi, 2000;Riedl and Biemann, 2012b), topic models (Misra et al, 2009;Riedl and Biemann, 2012a;Jameel and Lam, 2013;Du et al, 2013) and semantic embedding (Glavaš et al, 2016;Solbiati et al, 2021;Xing and Carenini, 2021). In contrast, supervised models can achieve more precise predictions by automatically mining clues of topic shift from large amounts of labeled data, either by classification on the pairs of sentences or chunks (Wang et al, 2017;Lukasik et al, 2020) or sequence labeling on the whole input sequence (Koshorek et al, 2018;Badjatiya et al, 2018;Xing et al, 2020;. However, the memory consumption and efficiency of neural models such as BERT (Kenton and Toutanova, 2019) can be limiting factors for modeling long documents as their length increases.…”

“…However the fluency of the constructed document is too low so that the semantic information is basically lost. Xing et al (2020) proposed to add the Consecutive Sentence-pair Coherence (CSC) task by computing the cosine similarity as coherence score. But no more incoherent sentence pairs are considered in CSC, except for those located at segment boundaries.…”

“…Supervised models have achieved large gains on topic segmentation through pre-training language models (PLMs) (e.g., BERT) and fine-tuning on large-scale supervised datasets (Kenton and Toutanova, 2019;Lukasik et al, 2020;Inan et al, 2022). Recently, (Arnold et al, 2019;Xing et al, 2020;Somasundaran et al, 2020;Lo et al, 2021) improve topic segmentation performance by explicitly modeling text coherence. However, these approaches either neglect context modeling beyond adjacent sentences (Wang et al, 2017), or require additional label information (Arnold et al, 2019;Barrow et al, 2020;Lo et al, 2021;Inan et al, 2022), or impede learning sentence-pair coherence without considering both coherent and incoherent pairs (Xing et al, 2020).…”

“…Recently, (Arnold et al, 2019;Xing et al, 2020;Somasundaran et al, 2020;Lo et al, 2021) improve topic segmentation performance by explicitly modeling text coherence. However, these approaches either neglect context modeling beyond adjacent sentences (Wang et al, 2017), or require additional label information (Arnold et al, 2019;Barrow et al, 2020;Lo et al, 2021;Inan et al, 2022), or impede learning sentence-pair coherence without considering both coherent and incoherent pairs (Xing et al, 2020). Moreover, compared to short documents, topic segmentation becomes more critical for understanding long documents, and coherence modeling for long document topic segmentation is more crucial.…”