Unsupervised Multilingual Sentence Boundary Detection

Abstract

In this article, we present a language-independent, unsupervised approach to sentence boundary detection. It is based on the assumption that a large number of ambiguities in the determination of sentence boundaries can be eliminated once abbreviations have been identified. Instead of relying on orthographic clues, the proposed system is able to detect abbreviations with high accuracy using three criteria that only require information about the candidate type itself and are independent of context: Abbreviations can be defined as a very tight collocation consisting of a truncated word and a final period, abbreviations are usually short, and abbreviations sometimes contain internal periods. We also show the potential of collocational evidence for two other important subtasks of sentence boundary disambiguation, namely, the detection of initials and ordinal numbers. The proposed system has been tested extensively on eleven different languages and on different text genres. It achieves good results without any further amendments or language-specific resources. We evaluate its performance against three different baselines and compare it to other systems for sentence boundary detection proposed in the literature.

Information Extraction over Structured Data: Question Answering with Freebase

“…The learning task can be framed in the following short steps: 1. We split each HTML document by sentences (Kiss and Strunk, 2006) using NLTK (Bird and Loper, 2004) and extracted those with at least two Freebase entities which has at least one direct established relation according to Freebase. 2.…”

Rule-Based Sentence Detection Method (RBSDM) for Turkish

“…The well-known highest success rate for Turkish sentence boundary method was denoted by Kiss and Strunk [8] about multilingual sentence boundary detection including Turkish, and it was measured as 98.74% mean value of all languages' test results. It was tested on the METU Turkish Corpus [9], which only included Turkish newspaper Milliyet.…”

Approximating Style by N-gram-based Annotation

“…This preprocessing followed rather simple heuristics and while the results are not perfect, they are sufficient for a quantitative analysis based on this amount of data. We processed the data using the following tools: the system Punkt ( Kiss and Strunk, 2006) 3 for tokenization and an off-the-shelf version of MATE dependency parser (Bohnet, 2010) trained on the TIGER Corpus (Seeker and Kuhn, 2012) for lemma, pos and dependency annotation. We evaluated the parser's annotations against a gold standard consensually created by two annotators for a sample of 22 sentences (600 tokens).…”