High dimensional biological data retrieval optimization with NoSQL technology

Abstract

BackgroundHigh-throughput transcriptomic data generated by microarray experiments is the most abundant and frequently stored kind of data currently used in translational medicine studies. Although microarray data is supported in data warehouses such as tranSMART, when querying relational databases for hundreds of different patient gene expression records queries are slow due to poor performance. Non-relational data models, such as the key-value model implemented in NoSQL databases, hold promise to be more performant solutions. Our motivation is to improve the performance of the tranSMART data warehouse with a view to supporting Next Generation Sequencing data.ResultsIn this paper we introduce a new data model better suited for high-dimensional data storage and querying, optimized for database scalability and performance. We have designed a key-value pair data model to support faster queries over large-scale microarray data and implemented the model using HBase, an implementation of Google's BigTable storage system. An experimental performance comparison was carried out against the traditional relational data model implemented in both MySQL Cluster and MongoDB, using a large publicly available transcriptomic data set taken from NCBI GEO concerning Multiple Myeloma. Our new key-value data model implemented on HBase exhibits an average 5.24-fold increase in high-dimensional biological data query performance compared to the relational model implemented on MySQL Cluster, and an average 6.47-fold increase on query performance on MongoDB.ConclusionsThe performance evaluation found that the new key-value data model, in particular its implementation in HBase, outperforms the relational model currently implemented in tranSMART. We propose that NoSQL technology holds great promise for large-scale data management, in particular for high-dimensional biological data such as that demonstrated in the performance evaluation described in this paper. We aim to use this new data model as a basis for migrating tranSMART's implementation to a more scalable solution for Big Data.

Operational Efficiencies and Simulated Performance of Big Data Analytics Platform over Billions of Patient Records of a Hospital System

“…It's processing and storage methods must adhere to data privacy at a high level and also the accessibility of the data for public disclosure [82][83][84]. One method of ensuring that patient data privacy/security is to use indexes generated from HBase, which can securely encrypt KV stores [36,[85][86][87], and HBase can further encrypt with integration with Hive [35]. Scott [62] also stated that Drill is already setup for encryption for HIPPA but we did not find this out-of-the-box and attempting to encrypt was time consuming.…”

Optimal Binning for Genomics

“…Several cloud-based systems are concerned with the management of secondary analysis pipelines [2], [21] and several cloud-based libraries provide the methods for those pipelines [35] [31] [46]. Effective metadata management for selecting samples using key-based NoSQL storage for referring to genomic datasets is described in [50]. We next concentrate on region processing, the most critical aspect of ternary data management, and specifically with the technology used for massive region-based processing, with the libraries supporting low-level region operations and with high-level query languages.…”

Usability and Suitability of the Omics-Integrating Analysis Platform tranSMART for Translational Research and Education

“…7,11,12 Others describe their usage in studies 2,3,[13][14][15] or in connection with other platforms or technologies. [16][17][18] This indicates a substantial need for further research.…”