An Interface for Biomedical Big Data Processing on the Tianhe-2 Supercomputer

Yang, Xi; Wu, Chengkun; Lü, Kai; Fang, Lin; Zhang, Yong; Li, Shengkang; Guo, Gui-Xin; Du, Yunfei

doi:10.3390/molecules22122116

Cited by 3 publications

(3 citation statements)

References 18 publications

(18 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Supercomputing and cloud computing are two major components of data infrastructure ( Yang et al, 2017b ). To address the challenges associated with big data, supercomputing must be considered.…”

Section: Introductionmentioning

confidence: 99%

Agricultural big data and methods and models for food security analysis—a mini-review

Ammar

Kheir

Manikas

2022

PeerJ

View full text Add to dashboard Cite

Background Big data and data analysis methods and models are important tools in food security (FS) studies for gap analysis and preparation of appropriate analytical frameworks. These innovations necessitate the development of novel methods for collecting, storing, processing, and extracting data. Methodology The primary goal of this study was to conduct a critical review of agricultural big data and methods and models used for FS studies published in peer-reviewed journals since 2010. Approximately 130 articles were selected for full content review after the pre-screening process. Results There are different sources of data collection, including but not limited to online databases, the internet, omics, Internet of Things, social media, survey rounds, remote sensing, and the Food and Agriculture Organization Corporate Statistical Database. The collected data require analysis (i.e., mining, neural networks, Bayesian networks, and other ML algorithms) before data visualization using Python, R, Circos, Gephi, Tableau, or Cytoscape. Approximately 122 models, all of which were used in FS studies worldwide, were selected from 130 articles. However, most of these models addressed only one or two dimensions of FS (i.e., availability and access) and ignored the other dimensions (i.e., stability and utilization), creating a gap in the global context. Conclusions There are certain FS gaps both worldwide and in the United Arab Emirates that need to be addressed by scientists and policymakers. Following the identification of the drivers, policies, and indicators, the findings of this review could be used to develop an appropriate analytical framework for FS and nutrition.

show abstract

Section: Introductionmentioning

confidence: 99%

Agricultural big data and methods and models for food security analysis—a mini-review

Ammar

Kheir

Manikas

2022

PeerJ

View full text Add to dashboard Cite

show abstract

“…Another direction where we believe our results can be used is the convergence Big Data-HPC. Several supercomputing centers have started to implement a convergence between Big-Data/Cloud and HPC, where numerous small applications are run on a supercomputer [3,4,5]. In this context, cloud applications are treated as second class citizen, where they can use the computing power not being used by actual HPC applications.…”

Section: Introductionmentioning

confidence: 99%

Scheduling on Two Unbounded Resources with Communication Costs

Ait

Kordon

Pallez

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Heterogeneous computing systems became a popular and powerful platform, containing several heterogeneous computing elements (e.g. CPU+GPU). In this paper, we consider that we have two platforms, each with an unbounded number of processors. We want to execute an application represented as a Directed acyclic Graph (DAG) using these two platforms. Each task of the application has two possible execution times, depending on the platform it is executed on. Also, there is a cost to transfer data from one platform to another between successive tasks. The goal here is to minimize the finish execution time of the last task of the application (usually called makespan). We show that the problem is NP-complete for graphs of depth at least 3 but polynomial for graphs of depth at most 2. Finally, we focus on particular classes of graphs, by providing polynomial-time algorithms for bi-partite graphs, trees and 2-series-parallel graphs with different assumptions on communication delays.Résumé : Les systèmes de calculs hétérogènes (par exemple CPU+GPU) sont des plateformes populaires. Dans ce travail, nous considérons une machine avec deux plateformes homogènes de calcul, chacune contenant un nombre illimité de ressources de calcul. Nous cherchons à exécuter une application représentée par un graphe de dépendance dirigé et acyclique sur ces plateformes. Chaque tâche de l'application a deux possible modèle d'exécution en fonction de la plateforme sur laquelles elles sont exécutées. En plus nous considérons un coût de communication entre deux tâches successives si elles ne sont pas exécutées sur la même plateforme. Nous travaillons à minimiser le temps d'exécution de l'application.Nous montrons que le problème est NP-complet pour les graphes de profondeur au moins trois, mais polynomial pour les graphes de profondeur au plus deux. En plus, nous montrons qu'il est possible de calculer des solutions optimales en temps polynomial pour certaines classes de graphes définies récursivement (arbres, graphes série-parallèles).

show abstract

“…This results in focusing on the use of various scalable platforms that would support efficient data processing and calculations [8,9,10]. Taking into account the complex nature of biological data, including macromolecular data of proteins, various storage formats for the data, the growing amount of the data, and finally, the complexity of some calculation processes performed over the data, we may find out that we meet challenges of Big Data processing and analysis [11,12,13]. Performing many calculations with 3D structures of proteins and nucleic acids meets the 5V model of Big Data at least in terms of the volume and the variety of data, and maybe for some calculations (e.g., those related to drug design) also the value .…”

Section: Introductionmentioning

confidence: 99%

Scalable Extraction of Big Macromolecular Data in Azure Data Lake Environment

Mrozek

Dąbek

2019

Molecules

View full text Add to dashboard Cite

Calculation of structural features of proteins, nucleic acids, and nucleic acid-protein complexes on the basis of their geometries and studying various interactions within these macromolecules, for which high-resolution structures are stored in Protein Data Bank (PDB), require parsing and extraction of suitable data stored in text files. To perform these operations on large scale in the face of the growing amount of macromolecular data in public repositories, we propose to perform them in the distributed environment of Azure Data Lake and scale the calculations on the Cloud. In this paper, we present dedicated data extractors for PDB files that can be used in various types of calculations performed over protein and nucleic acids structures in the Azure Data Lake. Results of our tests show that the Cloud storage space occupied by the macromolecular data can be successfully reduced by using compression of PDB files without significant loss of data processing efficiency. Moreover, our experiments show that the performed calculations can be significantly accelerated when using large sequential files for storing macromolecular data and by parallelizing the calculations and data extractions that precede them. Finally, the paper shows how all the calculations can be performed in a declarative way in U-SQL scripts for Data Lake Analytics.

show abstract

An Interface for Biomedical Big Data Processing on the Tianhe-2 Supercomputer

Cited by 3 publications

References 18 publications

Agricultural big data and methods and models for food security analysis—a mini-review

Agricultural big data and methods and models for food security analysis—a mini-review

Scheduling on Two Unbounded Resources with Communication Costs

Scalable Extraction of Big Macromolecular Data in Azure Data Lake Environment

Contact Info

Product

Resources

About