News Across Languages - Cross-Lingual Document Similarity and Event Tracking

Abstract

In today's world, we follow news which is distributed globally. Significant events are reported by different sources and in different languages. In this work, we address the problem of tracking of events in a large multilingual stream. Within a recently developed system Event Registry we examine two aspects of this problem: how to compare articles in different languages and how to link collections of articles in different languages which refer to the same event. Taking a multilingual stream and clusters of articles from each language, we compare different cross-lingual document similarity measures based on Wikipedia. This allows us to compute the similarity of any two articles regardless of language. Building on previous work, we show there are methods which scale well and can compute a meaningful similarity between articles from languages with little or no direct overlap in the training data. Using this capability, we then propose an approach to link clusters of articles across languages which represent the same event. We provide an extensive evaluation of the system as a whole, as well as an evaluation of the quality and robustness of the similarity measure and the linking algorithm.

Multilingual Clustering of Streaming News

“…The conclusions from our experiments are: (a) the weighting of the similarity metric features using SVM significantly outperforms unsupervised baselines such as CluStream ( Rupnik et al (2016), as explained in §5. "Size" denotes the number of documents in the collection, "Avg.…”

“…However, these are not "truly" online crosslingual clustering systems since they only decide on the linking of already-built monolingual clusters. In particular, Rupnik et al (2016) compute distances of document pairs across clusters using nearest neighbors, which might not scale well in an online setting. As detailed before, we adapted the cluster-linking dataset from Rupnik et al (2016) to evaluate our online crosslingual clustering approach.…”

“…Furthermore, for both the monolingual and crosslingual cases, we also experiment with using document timestamp features, as explained in §4.1. We use a new set of diverse timestamp features in addition to the simple absolute difference (in hours) between timestamps used by Rupnik et al (2016).…”

On Cross-Lingual Text Similarity Using Neural Translation Models

“…Another approach is based on the family of matrix decomposition [8], [24], [30]. This type of approach requires a parallel translation corpus and constructs a term-document matrix for each language.…”

Cross-lingual link discovery with TR-ESA

“…The experimental results demonstrate that taking into account the semantic aspect of news increases performance and improves linking. In the news domain there is another work that compares different cross-lingual document similarity measures based on Wikipedia to establish link connections of articles in different languages [46]. Also this work is mainly based on the identification of named entities in the articles by annotating the entity with the corresponding Wikipedia page.…”