Analyzing Entities and Topics in News Articles Using Statistical Topic Models

Abstract

Abstract. Statistical language models can learn relationships between topics discussed in a document collection and persons, organizations and places mentioned in each document. We present a novel combination of statistical topic models and named-entity recognizers to jointly analyze entities mentioned (persons, organizations and places) and topics discussed in a collection of 330,000 New York Times news articles. We demonstrate an analytic framework which automatically extracts from a large collection: topics; topic trends; and topics that relate entities.

Natural language processing for aviation safety reports: From classification to interactive analysis

“…It is this specific use that led us to applying them to incident reports. Former successful experiments have been run on collection of scientific publications (Hall et al, 2008), newspaper articles (Newman et al, 2006) and encyclopedia entries (Blei, 2012).…”

Clinical progress of human papillomavirus genotypes and their persistent infection in subjects with atypical squamous cells of undetermined significance cytology: Statistical and latent Dirichlet allocation analysis

“…A topic is then associated with five clinical measurements, including age, progress status, genotype, clearance (natural healing) and duration of infection with varying proportions, to show the probability distributions of the terms to constitute the topic (for example, age between 20-30 with a probability of 20%, progress 'a' with 10% and genotype 16 with 40%, clear '0' with 10% and duration with 20%). A large body of works exists that have applied LDA to a number of tasks, including news article analysis (17), cancer prediction (18) and analysis of scientific ideas (19). However, limited studies have been directed towards identifying complex interactions of clinical measurements, as proposed in the present study, in association with HPV-positive patients.…”

A text categorisation tool for open source communities based on semantic analysis

“…A key feature of this model is that it is an unsupervised learning technique, which means that the often human-intensive task of finding labelled examples is completely eliminated. Unsupervised also means that one can model a collection of documents through topics without being a domain expert -in fact one can even model a collection of documents through topics in other languages, without needing to know much about the language (Newman et al 2006). Up to 7482 different messages were considered in this study.…”

“…After pre-processing the content of 12,022 posts, we have extracted an initial vocabulary of more than 120,000 words. The final length of the vocabulary (W ¼ 410) has been selected considering the occurrence frequency rate (Ng et al 2001).…”

“…The number of ITER has been chosen equal to 200. This value is large enough to guarantee the convergence of the algorithm (Ng et al 2001). The number of topics has been selected using perplexity.…”