Performance evaluation of a MongoDB and hadoop platform for scientific data analysis

Abstract

Scientific facilities such as the Advanced Light Source (ALS) and Joint Genome Institute and projects such as the Materials Project have an increasing need to capture, store, and analyze dynamic semi-structured data and metadata. A similar growth of semi-structured data within large Internet service providers has led to the creation of NoSQL data stores for scalable indexing and MapReduce for scalable parallel analysis. MapReduce and NoSQL stores have been applied to scientific data. Hadoop, the most popular open source implementation of MapReduce, has been evaluated, utilized and modified for addressing the needs of different scientific analysis problems. ALS and the Materials Project are using MongoDB, a document oriented NoSQL store. However, there is a limited understanding of the performance trade-offs of using these two technologies together. In this paper we evaluate the performance, scalability and fault-tolerance of using MongoDB with Hadoop, towards the goal of identifying the right software environment for scientific data analysis.

Application and Realization of Improved Apriori Algorism in Hadoop Simulation Platform for Mass Data Process

“…HDFS provides the users with corresponding file namespace for storing data in a file format. Generally, HDFS divides these files into several file blocks, which are stored in a group of data services [2][3][4][5][6]. Then NameNode provides fundamental functions such as opening, closing and renaming the files and directories, while being responsible for mapping the file blocks to the DataNodes.…”

Evaluation Of Nosql Dbms Scalability For Storing Data For Psychology Web Platform

“…Within the scope of this work, the necessity of applying NoSQL solutions (Han, Haihong, Guan, & Jian, 2011;Nikulchev et al, 2015) will be substantiated and the experiment on the scalability analysis of MongoDB (Abramova & Bernardino, 2013;Dede, Govindaraju, Gunter, Canon, & Ramakrishnan, 2013) and Cassandra (Abramova & Bernardino, 2013;Lakshman & Malik, 2010) will be carried out.…”

Document-Oriented Data Warehouses: Complex Hierarchies and Summarizability

“…There is an increasing interest in implementing data warehouses with NoSQL systems [19] including document-oriented systems such as MongoDB [5]. NoSQL systems are an interesting alternative to relational databases (RDBMS), because they offer interesting scaling, replication and flexibility features.…”

“…They support atomic attributes as well as the complex attributes (nested records, arrays, …) for storing the data. Document-oriented systems are one of the most popular classes of NoSQL approaches [5]. Data is stored in documents and documents are grouped in collections [5,3].…”

“…Document-oriented systems are one of the most popular classes of NoSQL approaches [5]. Data is stored in documents and documents are grouped in collections [5,3]. Documents have a flexible schema.…”