Big Data with Cloud Computing: an insight on the computing environment, MapReduce, and programming frameworks

Abstract

The term 'Big Data' has spread rapidly in the framework of Data Mining and Business Intelligence. This new scenario can be defined by means of those problems that cannot be effectively or efficiently addressed using the standard computing resources that we currently have. We must emphasize that Big Data does not just imply large volumes of data but also the necessity for scalability, i.e., to ensure a response in an acceptable elapsed time. When the scalability term is considered, usually traditional parallel-type solutions are contemplated, such as the Message Passing Interface or high performance and distributed Database Management Systems. Nowadays there is a new paradigm that has gained popularity over the latter due to the number of benefits it offers. This model is Cloud Computing, and among its main features we has to stress its elasticity in the use of computing resources and space, less management effort, and flexible costs. In this article, we provide an overview on the topic of Big Data, and how the current problem can be addressed from the perspective of Cloud Computing and its programming frameworks. In particular, we focus on those systems for large-scale analytics based on the MapReduce scheme of data are recorded everyday resulting in a large volume of information; this incoming information arrives at a high rate and its processing involves real-time requirements implying a high velocity; we may find a wide variety of structured, semi-structured, and unstructured data; and data have to be cleaned before the integration into the system in order to maintain veracity.1 This 4V property is one of the most widespread definitions of what is known as the Big Data problem, 2,3 which has become a hot topic of interest within academia and corporations. The current explosion of data that is being generated is due to three main reasons 4 : (1) hundreds of applications such as mobile sensors, social media services, and other related devices are collecting information continuously; (2) storage capacity has improved so much that collecting data is cheaper than ever, making preferable to buy more storage space rather than deciding what to delete; (3) Machine Learning and information retrieval approaches have reached a significant improvement in the last years, thus enabling the acquisition of a higher degree of knowledge from data. 5,6 Corporations are aware of these developments. Gaining critical business insights by querying and analyzing such massive amounts of data is becoming a necessity. This issue is known as Business Intelligence (BI), 7,8 which refers to decision support systems that combine data gathering, data storage, and knowledge management with analysis to provide input to the decision process. 9 Regarding the former issues, a new concept appears as a more general field, integrating data warehousing, Data Mining (DM), and data visualization for Business Analytics. This topic is known as Data Science. 10,11The data management and analytics carried out in conventional database systems (and other rela...

Access methods for Big Data: current status and future directions

“…A lot of platforms for large-scale processing have tried to meet the challenges of Big Data in the last years [17]. These platforms attempt to carry closer the distributed technologies to the benchmark user (engineers and data scientists) by hiding the technical difficulties derived from distributed environments.…”

Hadoop-Based Word Count Simulation on Amazon Cloud

“…They proposed many alternatives to MapReduce programming model for large scale data processing needs. Madasamy et.al [22] have presented a paper on the performance evaluation of word frequency count in single node and multinode setup under Hadoop environment. They have concluded that multinode configuration yields comparatively better results as the input text file size scales.…”

Big Data Supervised Pairwise Ortholog Detection in Yeasts

“…Nevertheless, considering implementation hazards of the parallel models, scalability can be improved when these models are executed in a cloud within a big data framework as MapReduce [27,28]. This framework guarantees reliability, availability, and a good execution performance in a cloud with a distributed file system.…”