Identifying Co-referential Names Across Large Corpora

Lloyd, Levon; Mehler, Andrew; Skiena, Steven

doi:10.1007/11780441_3

Cited by 12 publications

(6 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lydia [2,3,9,14,13,15,18] Table 4: Precision, Recall, and F-Score of the two best classifiers, and % change in F-Score for various experiments with the hierarchical classifier Ward, et.al. [23] have used our classifier in the course of a significant study on differences in news coverage between cultural groups.…”

Section: Applications To News Streamsmentioning

confidence: 99%

Name-ethnicity classification from open sources

Ambekar

Ward

Mohammed

et al. 2009

Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Self Cite

214

185

View full text Add to dashboard Cite

The problem of ethnicity identification from names has a variety of important applications, including biomedical research, demographic studies, and marketing. Here we report on the development of an ethnicity classifier where all training data is extracted from public, non-confidential (and hence somewhat unreliable) sources. Our classifier uses hidden Markov models (HMMs) and decision trees to classify names into 13 cultural/ethnic groups with individual group accuracy comparable accuracy to earlier binary (e.g., Spanish/non-Spanish) classifiers. We have applied this classifier to over 20 million names from a large-scale news corpus, identifying interesting temporal and spatial trends on the representation of particular cultural/ethnic groups.

show abstract

Section: Applications To News Streamsmentioning

confidence: 99%

Name-ethnicity classification from open sources

Ambekar

Ward

Mohammed

et al. 2009

Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Self Cite

214

185

View full text Add to dashboard Cite

show abstract

“…Our work is based on the Lydia text analysis system (Godbole, Srinivasaiah, & Skiena 2007;Kil, Lloyd, & Skiena 2005;Lloyd, Kechagias, & Skiena 2005;Lloyd, Mehler, & Skiena 2006;Mehler et al 2006). The Lydia system recognizes named entities in text and extracts their temporal and spatial distribution.…”

Section: The Lydia Sentiment Analysis Systemmentioning

confidence: 99%

International Sentiment Analysis for News and Blogs

Bautin

Vijayarenu

Skiena

2021

ICWSM

View full text Add to dashboard Cite

There is a growing interest in mining opinions using sentiment analysis methods from sources such as news, blogs and product reviews. Most of these methods have been developed for English and are difficult to generalize to other languages. We explore an approach utilizing state-of-the-art machine translation technology and perform sentiment analysis on the English translation of a foreign language text. Our experiments indicate that (a) entity sentiment scores obtained by our method are statistically significantly correlated across nine languages of news sources and five languages of a parallel corpus; (b) the quality of our sentiment analysis method is largely translator independent; (c) after applying certain normalization techniques, our entity sentiment scores can be used to perform meaningful cross-cultural comparisons.

show abstract

“…Our entity-search engine is built on top of our Lydia news analysis system [4,[6][7][8][9]]. The Lydia system automatically builds an entity database from online U.S. newspapers.…”

Section: Related Workmentioning

confidence: 99%

Concordance-based entity-oriented search

Bautin

Skiena

2009

Web Intelligence and Agent Systems: An International Journal

Self Cite

View full text Add to dashboard Cite

We consider the problem of finding relevant named entities in response to a search query over a given text corpus. Entity search can readily be used to augment conventional web search engines for a variety of applications. We use entity concordance documents to generate lists of relevant entities for arbitrary text queries. To assess the significance of entity search, we analyzed the AOL dataset of 36 million web search queries with respect to two different sets of entities: namely (a) 2.3 million distinct entities extracted from a news text corpus and (b) 2.9 million Wikipedia article titles. The results clearly indicate that search engines should be aware of entities, for under various criteria of matching between 18-39% of all web search queries can be recognized as specifically searching for entities, while 73-87% of all queries contain entities. Our entity search engine creates a concordance document for each entity, consisting of all the sentences in the corpus containing that entity. We then index and search these documents using open-source search software. This gives a ranked list of entities as the result of search. Visit http://www.textmap.com for a demonstration of our entity search engine over a large news corpus. In the case where the query is a named entity, we evaluate the performance of our system by comparing the results of our search engine to the list of entities that have highest statistical juxtaposition scores with the queried entity. Juxtaposition score is a measure of how strongly two entities are related in terms of a probabilistic upper bound. The results show excellent performance, particularly over well-characterized classes of entities such as people.

show abstract

Identifying Co-referential Names Across Large Corpora

Cited by 12 publications

References 9 publications

Name-ethnicity classification from open sources

Name-ethnicity classification from open sources

International Sentiment Analysis for News and Blogs

Concordance-based entity-oriented search

Contact Info

Product

Resources

About