Dynamic N-Gram System Based on an Online Croatian Spellchecking Service

Abstract

As an infrastructure able to accelerate the development of natural language processing applications, large-scale lexical n-gram databases are at present important data systems. However, deriving such systems for world minority languages as it was done in the Google n-gram project leads to many obstacles. This paper presents an innovative approach to large-scale n-gram system creation applied to the Croatian language. Instead of using the Web as the world's largest text repository, our process of n-gram collection relies on the Croatian online academic spellchecker Hascheck, a language service publicly available since 1993 and popular worldwide. Our n-gram filtering is based on dictionary criteria, contrary to the publicly available Google n-gram systems in which cutoff criteria were applied. After 12 years of collecting, the size of the Croatian n-gram system reached the size of the largest Google Version 1 n-gram systems. Due to reliance on a service in constant use, the Croatian n-gram system is a dynamic one. System dynamics allowed modeling of n-gram count behavior through Heaps' law, which led to interesting results. Like many minority languages, the Croatian language suffers from a lack of sophisticated language processing systems in many application areas. The importance of a rich lexical n-gram infrastructure for rapid breakthroughs in new application areas is also exemplified in the paper.

N-gram Based Croatian Language Network: Application in a Smart Environment

“…However, the n-gram collection used in our research greatly exceeds the corpora available for training machine learning models. Hascheck is an online spell checking service that has been collecting n-grams for more than two decades, and its processed corpora are estimated to total more than 7 billion tokens [4]. Therefore, we assumed that using an n-gram collection of such volume could provide better results than expected.…”

“…Currently, N-gram search service is based on 3-gram system, which contains word sequences comprised of three words. However, this can be changed in future research since Hascheck contains n-gram collections of different lengths, with 2 <= n <= 7 [4]. N-gram system used in scope of our work is represented by a directed graph.…”

“…Moreover, there are still no suitable open datasets for research purposes. A rare positive example is the n-gram system collected in scope of Hascheck, a Croatian online academic spell checking service [3] [4].…”

Spoken notifications in smart environments using Croatian language

“…Currently, the n-gram service is based on 3-gram system, which contains word sequences comprised of three words. However, Hascheck contains n-gram collections of different lengths, with 2 <= n <= 7 [34].…”

“…Morphological lexicon was constructed using a segment of the morphological database available in Hascheck [34], summing up to approximately 700 000 entries. The implemented service provides the morphological descriptor for the given word, thus determining its exact morphological form.…”