Data Mining and Revealing Hidden Sentiment in Tweets Using Spark

Abstract

Data science is important and scientific value in our lives because it is multi-fields, it's the science that uses scientific methods, processes, algorithms, and systems for the purpose of extracting knowledge and ideas from data whether this data is organized or not. Data science is called 21st century oil to highlight its importance and scientific value in our lives. We paid great attention in this research paper, where we achieved three main steps in the field of data analysis, collecting data from different sources in the Internet and then storing the data within the system, the second step cleaning the data in order to obtain structured data and then applying the algorithms that are responsible for classifying the data. In pursuit of development, we have collected more than 1600000 tweets about the educational process and the possibility of future online education. We give great attention to this field of Sentiment Analysis (SA) and the use of modern Spark technology, which has achieved great success since its emergence. We have succeeded in using Spark in getting a good result, handling data quickly and accurately, which encouraged us to test it on two algorithms of Machine Learning, Support Vector Machine (SVM) & Maximum Entropy (Max Ent).