Towards Better Evaluation of Topic Model Quality

Khodorchenko, Maria; Butakov, Nikolay; Nasonov, Denis

doi:10.23919/fruct56874.2022.9953874

Cited by 2 publications

(2 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…4. The dataset with evaluation of topics (Khodorchenko et al, 2022a) contains automatic and human scores for a variety of sampled topics outputted by 100 variously configured ARTM models with different amounts of topics built on the datasets 1-3 from this list. To measure the quality of topics, they were presented as tasks in Toloka (Tol, 2023) crowdsourcing platform interface.…”

Section: Datasetsmentioning

confidence: 99%

“…Furthermore, the metrics may reflect only a particular side of the produced model quality (Hoyle et al, 2021). Additionaly, best evaluation metrics can differ from dataset to dataset (Khodorchenko et al, 2022a). The suboptimal decision of the best topic model may cause an inaccurate representation of data and, therefore, its biased understanding.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Attention-based estimation of topic model quality

Kataeva,

Khodorchenko

2023

Computational Linguistics and Intellectual Technologies”

View full text Add to dashboard Cite

Topic modeling is an essential instrument for exploring and uncovering latent patterns in unstructured textual data, that allows researchers and analysts to extract valuable understanding of a particular domain. Nonetheless, topic modeling lacks consensus on the matter of its evaluation. The estimation of obtained insightful topics is complicated by several obstacles, the majority of which are summarized by the absence of a unified system of metrics, the one-sidedness of evaluation, and the lack of generalization. Despite various approaches proposed in the literature, there is still no consensus on the aspects of effective examination of topic quality. In this research paper, we address this problem and propose a novel framework for evaluating topic modeling results based on the notion of attention mechanism and Layer-wise Relevance Propagation as tools for discovering the dependencies between text tokens. One of our proposed metrics achieved a 0.71 Pearson correlation and 0.74 𝜑𝜑𝐾𝐾 correlation with human assessment. Additionally, our score variant outperforms other metrics on the challenging Amazon Fine Food Reviews dataset, suggesting its ability to capture contextual information in shorter texts.

show abstract

Section: Datasetsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%