Scalable Parallel Programming in Python with Parsl

Babuji, Yadu; Woodard, A.; Li, Zhuozhao; Clifford, Ben; Foster, Ian; Wilde, Michael; Chard, Kyle

doi:10.1145/3332186.3332231

Cited by 89 publications

(100 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Table 1 shows a qualitative comparison between the proposed model, based on blocks and microblocks, and different solutions from the state-of-the-art focused on building workflows and pipelines. Table 1 includes traditional workflow engines for the building of processing solutions (i.e., Comps [118], Pycomps [119], Sacbe [86], Parsl [121], and DagOnStar [124]), and software for building distributed processing IoT dataflows based on message exchange (i.e., Apache Kafka [122] and Amazon kinesis [123]). The qualitative comparison was performed considering reliability, security, and efficiency features and the different applications of these non-functional requirements.…”

Section: A Qualitative Comparison Of Continuity Toolsmentioning

confidence: 99%

On the Continuous Processing of Health Data in Edge-Fog-Cloud Computing by Using Micro/Nanoservice Composition

et al. 2020

View full text Add to dashboard Cite

The edge, the fog, the cloud, and even the end-user's devices play a key role in the management of the health sensitive content/data lifecycle. However, the creation and management of solutions including multiple applications executed by multiple users in multiple environments (edge, the fog, and the cloud) to process multiple health repositories that, at the same time, fulfilling non-functional requirements (NFRs) represents a complex challenge for health care organizations. This paper presents the design, development, and implementation of an architectural model to create, on-demand, edge-fog-cloud processing structures to continuously handle big health data and, at the same time, to execute services for fulfilling NFRs. In this model, constructive and modular blocks, implemented as microservices and nanoservices, are recursively interconnected to create edge-fog-cloud processing structures as infrastructureagnostic services. Continuity schemes create dataflows through the blocks of edge-fog-cloud structures and enforce, in an implicit manner, the fulfillment of NFRs for data arriving and departing to/from each block of each edge-fog-cloud structure. To show the feasibility of this model, a prototype was built using this model, which was evaluated in a case study based on the processing of health data for supporting critical decision-making procedures in remote patient monitoring. This study considered scenarios where end-users and medical staff received insights discovered when processing electrocardiograms (ECGs) produced by sensors in wireless IoT devices as well as where physicians received patient records (spirometry studies, ECGs and tomography images) and warnings raised when online analyzing and identifying anomalies in the analyzed ECG data. A scenario where organizations manage multiple simultaneous each edge-fog-cloud structure for processing of health data and contents delivered to internal and external staff was also studied. The evaluation of these scenarios showed the feasibility of applying this model to the building of solutions interconnecting multiple services/applications managing big health data through different environments. INDEX TERMS Big health data, edge-fog-cloud, health-IoT processing, Internet of Things, microservice architecture.

show abstract

Section: A Qualitative Comparison Of Continuity Toolsmentioning

confidence: 99%

On the Continuous Processing of Health Data in Edge-Fog-Cloud Computing by Using Micro/Nanoservice Composition

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Each manager can consume either the entire or a fraction of the physical resource, depending on configuration, through a unified cluster interface. Mangers use existing task execution systems like Dask, 51 Parsl, 52 RADICAL, 53 and Fireworks 54 to accomplish high-throughput distributed computing within a given resource, leveraging large amounts of software infrastructure work by the community. In particular, for traditional HPC resources, these task execution systems hook into queuing systems like SLURM 55 to submit "workers" to nodes, which in turn evaluate tasks.…”

Section: Qcfractalmentioning

confidence: 99%

The MolSSI QCArchive Project: An Open-Source Platform to Compute, Organize, and Share Quantum Chemistry Data

Smith

Altarawy²,

Burns³

et al. 2020

Preprint

View full text Add to dashboard Cite

<div>The Molecular Sciences Software Institute's (MolSSI) Quantum Chemistry Archive (QCArchive) project is an umbrella name that covers both a central server hosted by MolSSI for community data and the Python-based software infrastructure that powers automated computation and storage of quantum chemistry results.</div><div>The MolSSI-hosted central server provides the computational molecular sciences community a location to freely access tens of millions of quantum chemistry computations for machine learning, methodology assessment, force-field fitting, and more through a Python interface.</div><div>Facile, user-friendly mining of the centrally archived quantum chemical data also can be achieved through web applications found at https://qcarchive.molssi.org.</div><div>The software infrastructure can be used as a standalone platform to compute, structure, and distribute hundreds of millions of quantum chemistry computations for individuals or groups of researchers at any scale.</div><div>The QCArchive Infrastructure is open-source (BSD-3C), code repositories can be found at https://github.com/MolSSI, and releases can be downloaded via PyPI and Conda.</div>

show abstract

“…A large body of research has explored distributed scheduling [27,29,48,49,51]. However, these solutions all target serverful scheduling with serverful deployment specific optimization objectives.…”

Section: Related Workmentioning

confidence: 99%

Wukong

Carver

Zhang

Wang

et al. 2020

Proceedings of the 11th ACM Symposium on Cloud Computing

View full text Add to dashboard Cite

Executing complex, burst-parallel, directed acyclic graph (DAG) jobs poses a major challenge for serverless execution frameworks, which will need to rapidly scale and schedule tasks at high throughput, while minimizing data movement across tasks. We demonstrate that, for serverless parallel computations, decentralized scheduling enables scheduling to be distributed across Lambda executors that can schedule tasks in parallel, and brings multiple benefits, including enhanced data locality, reduced network I/Os, automatic resource elasticity, and improved cost effectiveness. We describe the implementation and deployment of our new serverless parallel framework, called Wukong, on AWS Lambda. We show that Wukong achieves near-ideal scalability, executes parallel computation jobs up to 68.17× faster, reduces network I/O by multiple orders of magnitude, and achieves 92.96% tenantside cost savings compared to numpywren.

show abstract

Scalable Parallel Programming in Python with Parsl

Cited by 89 publications

References 9 publications

On the Continuous Processing of Health Data in Edge-Fog-Cloud Computing by Using Micro/Nanoservice Composition

On the Continuous Processing of Health Data in Edge-Fog-Cloud Computing by Using Micro/Nanoservice Composition

The MolSSI QCArchive Project: An Open-Source Platform to Compute, Organize, and Share Quantum Chemistry Data

Wukong

Contact Info

Product

Resources

About