Topic Modelling Meets Deep Neural Networks: A Survey

Zhao, He; Phung, Dinh; Huynh, Viet; Jin, Yuan; Du, Lan; Buntine, Wray L.

doi:10.48550/arxiv.2103.00498

Cited by 16 publications

(17 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, neural topic models have increasingly shown success in leveraging neural networks to improve upon existing topic modeling techniques (Terragni et al, 2021;Cao et al, 2015;Zhao et al, 2021;Larochelle and Lauly, 2012). The incorporation of word embeddings into classical models, such as LDA, demonstrated the viability of using these powerful representations Nguyen et al, 2015;Shi et al, 2017;Qiang et al, 2017).…”

Section: Related Workmentioning

confidence: 99%

BERTopic: Neural topic modeling with a class-based TF-IDF procedure

Grootendorst¹

2022

Preprint

329

281

View full text Add to dashboard Cite

Topic models can be useful tools to discover latent topics in collections of documents. Recent studies have shown the feasibility of approach topic modeling as a clustering task. We present BERTopic, a topic model that extends this process by extracting coherent topic representation through the development of a class-based variation of TF-IDF. More specifically, BERTopic generates document embedding with pre-trained transformer-based language models, clusters these embeddings, and finally, generates topic representations with the class-based TF-IDF procedure. BERTopic generates coherent topics and remains competitive across a variety of benchmarks involving classical models and those that follow the more recent clustering approach of topic modeling.

show abstract

Section: Related Workmentioning

confidence: 99%

BERTopic: Neural topic modeling with a class-based TF-IDF procedure

Grootendorst¹

2022

Preprint

329

281

View full text Add to dashboard Cite

show abstract

“…Other advances could be made through the use of unsupervised methods, which thus far have also relied on traditional machine learning only. More recent methods such as Neural Topic Models (NTM) have become increasingly popular for different tasks, including document summarisation and text generation [ 83 ] due to their flexibility and capability. These methods could also be applied to occupational exposure research to uncover new topics and concepts at a larger scale or draw new connections between exposures and work environments.…”

Section: Discussionmentioning

confidence: 99%

“…These methods could also be applied to occupational exposure research to uncover new topics and concepts at a larger scale or draw new connections between exposures and work environments. Similarly, NTM methods could also be coupled with pre-trained language models to further boost performance and result in more accurate representations of new topics [ 83 ]. Extrapolating existing research to other domains of exposure research Most of the research explored in this review is specific to a particular type of exposure, databases or enhancement of literature reviews.…”

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

A Narrative Literature Review of Natural Language Processing Applied to the Occupational Exposome

Schoene

Basinas

Tongeren

et al. 2022

IJERPH

View full text Add to dashboard Cite

The evolution of the Exposome concept revolutionised the research in exposure assessment and epidemiology by introducing the need for a more holistic approach on the exploration of the relationship between the environment and disease. At the same time, further and more dramatic changes have also occurred on the working environment, adding to the already existing dynamic nature of it. Natural Language Processing (NLP) refers to a collection of methods for identifying, reading, extracting and untimely transforming large collections of language. In this work, we aim to give an overview of how NLP has successfully been applied thus far in Exposome research. Methods: We conduct a literature search on PubMed, Scopus and Web of Science for scientific articles published between 2011 and 2021. We use both quantitative and qualitative methods to screen papers and provide insights into the inclusion and exclusion criteria. We outline our approach for article selection and provide an overview of our findings. This is followed by a more detailed insight into selected articles. Results: Overall, 6420 articles were screened for the suitability of this review, where we review 37 articles in depth. Finally, we discuss future avenues of research and outline challenges in existing work. Conclusions: Our results show that (i) there has been an increase in articles published that focus on applying NLP to exposure and epidemiology research, (ii) most work uses existing NLP tools and (iii) traditional machine learning is the most popular approach.

show abstract

“…As a hierarchical neural topic model, WHAI shows attractive qualities in multi-layer document representation learning and hierarchical explainable topic discovery. Compared with traditional Bayesian probabilistic topic models, these NTMs usually enjoy better flexibility and scalability, which are important for modeling large-scale data and performing downstream tasks (Zhang et al, 2019;Wang et al, 2020c;chen et al, 2020;Wang et al, 2020a;Duan et al, 2021;Zhao et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network

Duan

Wang

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Hierarchical topic models such as the gamma belief network (GBN) have delivered promising results in mining multi-layer document representations and discovering interpretable topic taxonomies. However, they often assume in the prior that the topics at each layer are independently drawn from the Dirichlet distribution, ignoring the dependencies between the topics both at the same layer and across different layers. To relax this assumption, we propose sawtooth factorial topic embedding guided GBN, a deep generative model of documents that captures the dependencies and semantic similarities between the topics in the embedding space. Specifically, both the words and topics are represented as embedding vectors of the same dimension. The topic matrix at a layer is factorized into the product of a factor loading matrix and a topic embedding matrix, the transpose of which is set as the factor loading matrix of the layer above. Repeating this particular type of factorization, which shares components between adjacent layers, leads to a structure referred to as sawtooth factorization. An auto-encoding variational inference network is constructed to optimize the model parameter via stochastic gradient descent. Experiments on big corpora show that our models outperform other neural topic models on extracting deeper interpretable topics and deriving better document representations.

show abstract

Topic Modelling Meets Deep Neural Networks: A Survey

Cited by 16 publications

References 35 publications

BERTopic: Neural topic modeling with a class-based TF-IDF procedure

BERTopic: Neural topic modeling with a class-based TF-IDF procedure

A Narrative Literature Review of Natural Language Processing Applied to the Occupational Exposome

Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network

Contact Info

Product

Resources

About