Optimization of data segments and number of cores for defining popularity of kazakh words using apache spark

Abstract

Kazakh is an agglutinative language which has complex structure. In this work Apache Spark was used to specify the popularity of Kazakh words in 3 popular kazakh compositions. The main goal was to find the optimal number of data segments for a specific number of cores in order to find the best computational speed. To do so, the data was divided into several segments and ran on a cluster with a different number of cores each time. Results show that the amount of data segments directly affects the computing speed.