Lessons Learned from Creating a Balanced Corpus from Online Data

Abstract

This paper describes lessons learned from developing the most recent Balanced Corpus of Modern Latvian (LVK2018) from various online sources. Most of the new corpora are created from data obtained from various text holders, which requires cooperation agreements with each of the text holders. Reaching these cooperation agreements is a difficult and time consuming task and may not be necessary if the resource to be created is not of hundred millions of size. Although there are many different resources available on the Internet today for a particular language, finding viable online resources to create a balanced corpus is still a challenging task. Developing a balanced corpus from various online sources does not require agreements with text holders, but it presents many more technical challenges, including text extraction, cleaning and validation.

Experience of design and development of a new open access interactive multifunctional database management system for special lexis of biology

“…Also, a large data set is necessary for ML algorithms to learn and it is not always possible to collect enough data for learning purposes, especially in languages with a relatively small number of users, such as Latvian (Jasmonts et al, 2022). Integration with other systems is possible, but this specific set of biology data may not be sufficient for neural network training (Darģis et al, 2020). In addition, not all information systems permit access via application programming interface to collect data dynamically and more effectively.…”