N-Gram Collection from a Large-Scale Corpus of Polish Internet

Roziewski, Szymon; Stokowiec, Wojciech; Sobkowicz, Antoni

doi:10.1007/978-3-319-30315-4_3

Cited by 2 publications

(2 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…7 present the occurrences of the top 100 n-grams for each ngram rank analysed. The shapes of curves and the general characteristics are in accordance with results published in work (Roziewski et al, 2016) and remain intact.…”

Section: N-gram Language Modelsupporting

confidence: 87%

See 1 more Smart Citation

LanguageCrawl: a generic tool for building language models upon common Crawl

Roziewski

Kozłowski

2021

Lang Resources & Evaluation

Self Cite

View full text Add to dashboard Cite

The exponential growth of the internet community has resulted in the production of a vast amount of unstructured data, including web pages, blogs and social media. Such a volume consisting of hundreds of billions of words is unlikely to be analyzed by humans. In this work we introduce the tool LanguageCrawl, which allows Natural Language Processing (NLP) researchers to easily build web-scale corpora using the Common Crawl Archive—an open repository of web crawl information, which contains petabytes of data. We present three use cases in the course of this work: filtering of Polish websites, the construction of n-gram corpora and the training of a continuous skipgram language model with hierarchical softmax. Each of them has been implemented within the LanguageCrawl toolkit, with the possibility to adjust specified language and n-gram ranks. This paper focuses particularly on high computing efficiency by applying highly concurrent multitasking. Our tool utilizes effective libraries and design. LanguageCrawl has been made publicly available to enrich the current set of NLP resources. We strongly believe that our work will facilitate further NLP research, especially in under-resourced languages, in which the lack of appropriately-sized corpora is a serious hindrance to applying data-intensive methods, such as deep neural networks.

show abstract

Section: N-gram Language Modelsupporting

confidence: 87%

“…Figure 6 shows n-gram occurrences for each n-gram type with respect to its length. The chart is based on much broader data than in paper (Roziewski, Stokowiec, & Sobkowicz, 2016), though the characteristics remain similar.…”

Section: N-gram Language Modelmentioning

confidence: 99%