Performance Analysis of Apache Spark MLlib Clustering on Batch Data Stored in Cassandra

Abstract

With the tremendous increase in the amount of data being generated from variety of sources there is a need of efficient data storage and processing techniques. Some of the sources generating this large amount of data are Weather Sensors, Scientific experiments, etc. This huge voluminous data is termed as BigData. Due to ever-increasing amount of data there is a demand for faster data ingestion and processing. Apache Spark, a dominant processing tool is a publicly available platform for processing outsized data and is mostly intended for iterative machine learning jobs. In this study, an integrated approach i.e., Spark MLlib Clustering on batch weather data stored in Cassandra database is proposed. This helps to analyze our data into number of Clusters which is required and useful for further examination of data. The main idea of this study is to evaluate Batch Processing performance of an integrated approach with two popular clustering algorithms.