Exascale computing and data architectures for brownfield applications

Bobák, Martin; Belloum, Adam; Nowakowski, Piotr; Meizner, Jan; Bubak, Marian; Heikkurinen, Matti; Habala, Ondrej; Hluchý, Ladislav

doi:10.1109/fskd.2018.8686900

Cited by 6 publications

(4 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As the MEE is a generic architecture, capable of supporting a wide range of simulations and HPC studies, not necessarily limited to medical sciences, work is underway to extend its usage to applications which call for simultaneous processing of data across a range of HPC centres. This is being done in research investigating the major challenges and the basic architecture of the scalable storage and compute platform for exascale processing of extremely large datasets [30,31], in the context of the PROCESS project [32], and involves developing support for simulation steps encapsulated in application containers (e.g. Docker and Singularity), which can be freely distributed to multiple computing infrastructures for better parallelization of computational tasks.…”

Section: Discussionmentioning

confidence: 99%

The EurValve model execution environment

et al. 2020

View full text Add to dashboard Cite

The goal of this paper is to present a dedicated high-performance computing (HPC) infrastructure which is used in the development of a so-called reduced-order model (ROM) for simulating the outcomes of interventional procedures which are contemplated in the treatment of valvular heart conditions. Following a brief introduction to the problem, the paper presents the design of a model execution environment, in which representative cases can be simulated and the parameters of the ROM fine-tuned to enable subsequent deployment of a decision support system without further need for HPC. The presentation of the system is followed by information concerning its use in processing specific patient cases in the context of the EurValve international collaboration.

show abstract

Section: Discussionmentioning

confidence: 99%

The EurValve model execution environment

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Figure 2 depicts the changes of the initial PROCESS architecture [6,2] needed to involve the micro-infrastructure approach into the initial architecture. All the changes are highlighted in magenta.…”

Section: Process Architecture -An Example Of a Technical Exascale Architecturementioning

confidence: 99%

Reference Exascale Architecture (Extended Version)

Bobák

Hluchý

Habala

et al. 2020

cai

Self Cite

View full text Add to dashboard Cite

While political commitments for building exascale systems have been made, turning these systems into platforms for a wide range of exascale applications faces several technical, organisational and skills-related challenges. The key technical challenges are related to the availability of data. While the first exascale machines are likely to be built within a single site, the input data is in many cases impossible to store within a single site. Alongside handling of extreme-large amount of data, the exascale system has to process data from different sources, support accelerated computing, handle high volume of requests per day, minimize the size of data flows, and be extensible in terms of continuously increasing data as well as an increase in parallel requests being sent. These technical challenges are addressed by the general reference exascale architecture. It is divided into three main blocks: virtualization layer, distributed virtual file system, and manager of computing resources. Its main property is modularity which is achieved by containerization at two levels: 1) application containers -containerization of scientific workflows, 2) micro-infrastructure -containerization of extreme-large data serviceoriented infrastructure. The paper also presents an instantiation of the reference architecture -the architecture of the PROCESS project (PROviding Computing solutions for ExaScale ChallengeS) and discusses its relation to the reference exascale architecture. The PROCESS architecture has been used as an exascale platform within various exascale pilot applications. This paper also presents performance modelling of exascale platform with its validation 1 .

show abstract

“…The underlying data management plays a key role in these WfMSs. Many different approaches have been developed to address the problem of delivering of extensive data sets across geographically distributed and heterogeneous storage systems [8]. Most of such approaches aim to deal mostly with data transfer, data sharing and synchronization, authorisation, security and privacy regarding data compliance (e.g., GDPR 1 and HIPAA 2 ), provenance, and reproducibility.…”

Section: Introductionmentioning

confidence: 99%

“…Correspondingly, decentralised storage systems and protocols such as IPFS [12], Storj [13], SWARM [14], and Sia [15] are widely working together with blockchain-based solutions or frameworks to address the inefficient and costly problem of storing large volumes of data [16]- [18]. Some researchers have pointed out that IPFS, if built right, could complement or replace HTTP and some other protocols in the discussion about data management tools and technologies [8]. However, the study of decentralised data management (e.g., IPFS) in WfMSs has not been taken seriously and published research in this field is countable.…”

Section: Introductionmentioning

confidence: 99%

Towards a Service-based Adaptable Data Layer for Cloud Workflows

Wang,

Janse,

Bianchi

et al. 2023

2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC)

View full text Add to dashboard Cite

Many scientific workflows are data-driven and need to be continuously executed for the large volume of datasets transferred from distributed data sources. The overhead arising from data transfers must be considered when optimizing workflow performance. Many workflow systems support various data transfer protocols (DTPs) and file systems, which lack flexibility for adopting new options. The major problem is to determine which protocol or file system should be selected. In this paper, we prototype a container-native workflow data layer that supports multiple DTPs, e.g., FTP, WebDAV, and IPFS, and assess their data transfer performance. We base this tool has demonstrated the feasibility of using combinations of Docker, CWL, and Argo to provide performance analysis for workflow scenarios. Our results show that: 1) IPFS outperforms WebDAV in uploading large files; 2) the makespan via IPFS executed in Argo is comparable with WebDAV, and IPFS's total data transferred is always less than WebDAV.

show abstract

Exascale computing and data architectures for brownfield applications

Cited by 6 publications

References 20 publications

The EurValve model execution environment

The EurValve model execution environment

Reference Exascale Architecture (Extended Version)

Towards a Service-based Adaptable Data Layer for Cloud Workflows

Contact Info

Product

Resources

About