Effective processing of unstructured data using python in Hadoop map reduce

Abstract

In present scenario, the growing data are naturally unstructured. In this case to handle the wide range of data, is difficult. The proposed paper is to process the unstructured text data effectively in Hadoop map reduce using Python. Apache Hadoop is an open source platform and it widely uses Map Reduce framework. Map Reduce is popular and effective for processing the unstructured data in parallel manner. There are two stages in map reduce, namely transform and repository. Here the input splits into small blocks and worker node process individual blocks in parallel. This map reduce generally is based on java. While Hadoop Streaming allows writing mapper and reducer in other languages like Python. In this paper, we are going to show an alternative way of processing the growing unstructured content data by using python. We will also compare the performance between java based and non-java based programs.

Text Classification Using Genetic Programming with Implementation of Map Reduce and Scraping

“…Research related to the use of map-reduce for preprocessing has been carried out to review the algorithmic aspects of parallel processing [25], Scalable Distributed Data Processing [26]- [28], to Effective processing for unstructured data using python [29]. The proposed research uses the python programming language and parallel processing; however, it uses a different kind of pre-processing and algorithm.…”