Swift/T: Large-Scale Application Composition via Distributed-Memory Dataflow Processing

Wozniak, Justin M.; Armstrong, Timothy G.; Wilde, Michael; Katz, Daniel S.; Lusk, Ewing L.; Foster, Ian

doi:10.1109/ccgrid.2013.99

Cited by 116 publications

(72 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…By simulating the discovery process hundreds of times for each parameter setting, we identify strategies that, on average, discover a given proportion of the network with greatest efficiency. This exploration was computationally intensive and could not be conducted on the full network; in fact, our project was a use case for the development of novel parallel programming approaches (35,36). Recall that each strategy (i.e., set of parameters) prioritizes different kinds of experiments.…”

Section: Resultsmentioning

confidence: 99%

Choosing experiments to accelerate collective discovery

Rzhetsky

Foster

et al. 2015

Proc. Natl. Acad. Sci. U.S.A.

176

161

View full text Add to dashboard Cite

A scientist's choice of research problem affects his or her personal career trajectory. Scientists' combined choices affect the direction and efficiency of scientific discovery as a whole. In this paper, we infer preferences that shape problem selection from patterns of published findings and then quantify their efficiency. We represent research problems as links between scientific entities in a knowledge network. We then build a generative model of discovery informed by qualitative research on scientific problem selection. We map salient features from this literature to key network properties: an entity's importance corresponds to its degree centrality, and a problem's difficulty corresponds to the network distance it spans. Drawing on millions of papers and patents published over 30 years, we use this model to infer the typical research strategy used to explore chemical relationships in biomedicine. This strategy generates conservative research choices focused on building up knowledge around important molecules. These choices become more conservative over time. The observed strategy is efficient for initial exploration of the network and supports scientific careers that require steady output, but is inefficient for science as a whole. Through supercomputer experiments on a sample of the network, we study thousands of alternatives and identify strategies much more efficient at exploring mature knowledge networks. We find that increased risk-taking and the publication of experimental failures would substantially improve the speed of discovery. We consider institutional shifts in grant making, evaluation, and publication that would help realize these efficiencies.complex networks | computational biology | science of science | innovation | sociology of science A scientist's choice of research problem directly affects his or her career. Indirectly, it affects the scientific community. A prescient choice can result in a high-impact study. This boosts the scientist's reputation, but it can also create research opportunities across the field. Scientific choices are hard to quantify because of the complexity and dimensionality of the underlying problem space. In formal or computational models, problem spaces are typically encoded as simple choices between a few options (1, 2) or as highly abstract "landscapes" borrowed from evolutionary biology (3-5). The resulting insight about the relationship between research choice and collective efficiency is suggestive, but necessarily qualitative and abstract.We obtain concrete, quantitative insight by representing the growth of knowledge as an evolving network extracted from the literature (2, 6). Nodes in the network are scientific concepts and edges are the relations between them asserted in publications. For example, molecules-a core concept in chemistry, biology, and medicine-may be linked by physical interaction (7) or shared clinical relevance (8). Variations of this network metaphor for knowledge have appeared in philosophy (9), social studies of science (10-12), artific...

show abstract

Section: Resultsmentioning

confidence: 99%

Choosing experiments to accelerate collective discovery

Rzhetsky

Foster

et al. 2015

Proc. Natl. Acad. Sci. U.S.A.

176

161

View full text Add to dashboard Cite

show abstract

“…Some approaches store monitoring data on log files [36] while others use relational databases. The organization of the monitoring data also varies because the information for each monitoring level may be stored on separate repositories with different formats.…”

Section: Monitoringmentioning

confidence: 99%

Dynamic steering of HPC scientific workflows: A survey

Mattoso

Dias

Ocaña

et al. 2015

Future Generation Computer Systems

View full text Add to dashboard Cite

“…Swift/T is a new implementation of Swift that aims for extreme scalability up to the largest supercomputers [9]. In pursuit of this goal, this paper explores two questions: what execution model and runtime primitive operations are needed to support extremely scalable data-driven task parallel applications, and how can a high-level declarative language be compiled to make efficient use of these lower-level primitives.…”

Section: Introductionmentioning

confidence: 99%

Compiler Optimization for Extreme-Scale Scripting

Armstrong

Wozniak

Wilde

et al. 2014

2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

View full text Add to dashboard Cite

The data-driven task parallelism execution model can support parallel programming models that are well suited for large-scale distributed-memory parallel computing, for example, simulations and analysis pipelines running on clusters and clouds. We describe a novel compiler intermediate representation and optimizations for this execution model, including adaptions of standard techniques alongside novel techniques. These techniques are applied to Swift/T, a high-level scripting language for flexible data flow composition of functions, which may be serial or use lower-level parallel programming models such as MPI and OpenMP. This paper presents preliminary results, indicating that our compiler optimizations reduce communication overhead by 70% to 93% on distributed-memory systems.

show abstract

Swift/T: Large-Scale Application Composition via Distributed-Memory Dataflow Processing

Cited by 116 publications

References 24 publications

Choosing experiments to accelerate collective discovery

Choosing experiments to accelerate collective discovery

Dynamic steering of HPC scientific workflows: A survey

Compiler Optimization for Extreme-Scale Scripting

Contact Info

Product

Resources

About