An Economic Approach to Big Data in a Minority Language

Abstract

Google's n-gram project brought recently big data benefits to several main world languages, like English, Chinese etc. Any attempt to derive such systems, aimed to accelerate the development of NLP applications for world minority languages, in the manner in which it has been done in the project, encounters many obstacles. This paper presents an innovative and economic approach to large-scale n-gram system creation applied to the Croatian language case. Instead of using the Web as the world's biggest text repository, our process of n-gram collection relies on the Croatian academic online spellchecker Hascheck, a language service publicly available since 1993 and popular worldwide. The service has already processed a corpus whose size exceeds the size of the Croatian web-corpus created in recent years. Contrary to the Google n-gram systems, where cutoff criteria were applied, our n-gram filtering is based on dictionary criteria. This resulted in a system comparable in size to the largest ngram systems of today. Because of the reliance on a service in constant use, the Croatian n-gram system is a dynamic one, unique among the systems compared. The importance of having an n-gram infrastructure for rapid breakthroughs in new application areas is also exemplified in the paper.

Models of lexical meaning

“…Second, results from psycholinguistic research (based on the frequency data of the terms used in the partition of the color spectrum) enabled comparison to the data collected via other empirically-based methods. For example, the EOSS data for Croatian were compared to the frequency data from the Croatian n-gram system (based on the Web as Corpus approach) consisting of 1.72 billion tokens (Dembitz et al, 2014). The 165 different Croatian color terms (types) from the EOSS project were checked in the Croatian n-gram system in order to provide evidence about their attestation in a large language resource.…”

Dynamic N-Gram System Based on an Online Croatian Spellchecking Service

“…The update of the n-gram database is performed monthly. Further details about n-gram system creation and maintenance are given in [8].…”

“…In [8], we have presented a figure, copied here as Fig. 1, that implies the 4-grams are the richest n-grams in Croatian, and no n-grams, n > 4, can ever overcome them.…”

“…We wanted to see what would happen with them because we expected divergence based on data from [3] and [4]. Our approach proved to be an economic one [8] because it did not require extra manpower or other resources, as opposed to the Web as Corpus approach described in [9].…”

“…FIGURE 1. Behavior of n-gram counts derived from the higher-order n-gram system in the year 2014[8]; x-axis: corpus size, y-axis: number of n-grams.…”