Big data clustering with varied density based on MapReduce

Abstract

With the recent growth and advancement in Information Technology, data has produced at a very high rate in a variety of fields, which have presented to users in a structured, semi-structured, and non-structured mode [1]. New technologies for storing and extracting useful information from this volume of data (big data) have needed because the discovery and extraction of useful information and knowledge from this data volume are difficult, hence, other traditional relational databases cannot meet the needs of users [2]. If you are dealing with data beyond the capabilities of existing software, you are, in fact, dealing with big data. Large data is commonly referred to as a set of data that exceeds the extent to which it can be extracted, refined, managed, and processed by standard management tools and databases. In other words, the term "big data" refers to data that is complex in terms of volume and variety; however, it is not possible to manage them with traditional tools, and therefore, they cannot extract their hidden knowledge and knowledge at predetermined times [3, 4]. Big data is, therefore, defined with three attributes of volume, velocity, and variety that are called Gartner's commentary; some scholars have in addition; IBM cited the

Big data monetization throughout Big Data Value Chain: a comprehensive review

“…It aims to analyze pre-processed stored data in order to find correlations, identify patterns and create actionable insights. There are mainly four categories through which Big Data analysis could be designed and conducted: prescriptive, predictive, diagnostic and descriptive [32,36,[52][53][54][55][56]. In next, we describe each of these categories:…”

Estimating runtime of a job in Hadoop MapReduce

“…To estimate the runtime of a job in Hadoop MapReduce, first, we investigated the anatomy of Hadoop's job and the stages of running a job precisely [1][2][3][4][5]. Since Hadoop works on the repetitive application on the same data type [17], we use the profiling method, which means that there is some separate table in the database for each application.…”

“…Therefore, the time of each stage must be calculated and summated. Since Hadoop run on a distributed system, many factors and parameters affect T map and T reduce [1][2][3][4][5]22]. So, we investigate the parameters with more impact on runtime.…”

“…According to the Hadoop MapReduce operation [1][2][3][4][5], when the request to execute a job is issuing, the input file breaks as predefined blocks (in Hadoop, version 2, the size of each block or split is by default 128 MB) and divided up into nodes, each block is called a task or a map. The number of maps determines by dividing the input data size by the size of the data block (Eq.…”

Decreasing the execution time of reducers by revising clustering based on the futuristic greedy approach

“…These models can be applied for evaluating several methods in Map-Reduce. Heidari et al [26] discussed clustering with variable density based on huge data. They presented MR-VDBSCAN in this method.…”