Data Management in Dynamic Environment-driven Computational Science

Simmhan, Yogesh; Pallickara, Shrideep; Vijayakumar, Nithya N.; Plale, Beth

doi:10.1007/978-0-387-73659-4_17

Cited by 9 publications

(8 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Workflow provenance [108,109,110,111,112,113] has been used as a way to capture and track lineage information (more discussion in §9). Domain specific workspaces have been developed to share workflows [114,115] and communities have developed workflow repositories [116,117]. However, workflow and knowledge transfer in scientific communities largely happens in ad-hoc ways.…”

Section: State Of the Artmentioning

confidence: 99%

Report of the DOE Workshop on Management, Analysis, and Visualization of Experimental and Observational data – The Convergence of Data and Computing

Bethel¹,

eds²

2016

View full text Add to dashboard Cite

The Department of Energy (DOE) Office of Science (SC) operates dozens of national science user facilities that span many different disciplines. These facilities include accelerators, colliders, supercomputers, light sources and neutron sources, as well as facilities for studying the nanoworld, the environment, and the atmosphere. In Fiscal Year 2014 over 33,000 researchers from academia, industry, and government laboratories, spanning all fifty states and the District of Columbia, utilized these unique facilities to perform new scientific research. Each of these facilities generates vast amounts of scientific data, and the rate, size, and complexity of this data is rapidly increasing, thanks to advances in technology. A growing concern, which motivates this workshop, is the likely significant adverse impact on science programs that will result without significant advances in the capabilities needed to manage and gain knowledge from these collections of data. The purpose of this workshop, held 29 September 2015 through 1 October 2015 in Bethesda, MD, is to help the Advanced Scientific Computing Research (ASCR) and research community better understand needs related to the management, analysis, and visualization of experimental and observational data (EOD) collected and generated by experimental and observational science projects (EOS) at Office of Science user facilities. The science needs articulated in this report, along with the findings, recommendations, and detailed discussion of issues, collectively are consistent with and show opportunity for cultivating a research, development, and deployment path that takes steps towards realizing the vision articulated in the National Strategic Computing Initiative (NCSI) [1, 2] and the Big Data Research and Development Initiative [3, 4]. Specifically, the science use cases reveal a trend towards the convergence of data and computing: data-and compute-centric needs and opportunities are increasingly intertwined, interrelated, and symbiotic. Advances in our ability to collect data in turn require advances in computational capabilities to understand, preserve, share and make optimal use of data, and can even favorably impact the quality and value of science we perform by improving the quality of data we collect. This workshop report consists of input from a set of representatives from DOE EOS facilities and researchers in mathematics and computer science. The findings, drawn from use cases that articulate science drivers, indicate acute and urgent needs. This report articulates a path forward for meeting those needs, a path that includes advances in mathematics, computer and data science, as well as advances in and the use of HPC computational and networking infrastructures. One major theme recurring in the use cases is that individual EOS projects are presently pursuing their own independent paths towards meeting data-centric challenges, which results in the duplication of effort and increased costs across the entire program. EOS projects, and the EO community as a whole, ...

show abstract

Section: State Of the Artmentioning

confidence: 99%

Report of the DOE Workshop on Management, Analysis, and Visualization of Experimental and Observational data – The Convergence of Data and Computing

Bethel¹,

eds²

2016

View full text Add to dashboard Cite

show abstract

“…While it is possible to perform incremental SQL queries combined with service API calls to answer this query, such queries are not scalable and deemed outside the scope of the Karma provenance system. Karma is part of the LEAD Cyberinfrastructure project and, as in other such projects, there are information services such as the myLEAD personal catalog and Resource Catalog [9] that record generic metadata about data products and services. We expect such systems to answer the bulk of the annotation queries and the provenance is just one piece in the information integration landscape of the virtual organization.…”

Section: Query Capabilitiesmentioning

confidence: 99%

“…Generic annotations are considered beyond the purview of a provenance system. There are several general purpose metadata catalogs available for various resources on the grid 7–9 and the provenance system can play a pivotal role in the integration of information from these information services—without being tasked with storing, managing, and querying generic annotations. We had a chance to pit this philosophy against the challenge queries, several of which deal with queries over annotations, and found our system to answer all but one of the challenge queries.…”

Section: Introductionmentioning

confidence: 99%

Query capabilities of the Karma provenance framework

Simmhan¹,

Plale²,

Gannon³

2007

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

SUMMARYProvenance metadata in e-Science captures the derivation history of data products generated from scientific workflows. Provenance forms a glue linking workflow execution with associated data products, and finds use in determining the quality of derived data, tracking resource usage, and for verifying and validating scientific experiments. In this article, we discuss the scope of provenance collected in the Karma provenance framework used in the LEAD Cyberinfrastructure project, distinguishing provenance metadata from generic annotations. We further describe our approaches to querying for different forms of provenance in Karma in the context of queries in the first provenance challenge. We use an incremental, building-block method to construct provenance queries based on the fundamental querying capabilities provided by the Karma service centered on the provenance data model. This has the advantage of keeping the Karma service generic and simple, and yet supports a wide range of queries. Karma successfully answers all but one challenge query.

show abstract

“…This has led to an explosion in the availability of scientific datasets , including the raw data directly extracted with measuring instruments and data derived from computational models and simulations . These datasets can be stored online in large volume in public or private repositories that can be made accessible to users within the scientific community or beyond, in order to foster inter‐organizational and interdisciplinary research, which can accelerate scientific discovery .…”

Section: Introductionmentioning

confidence: 99%

Spatio‐temporal pseudo relevance feedback for scientific data retrieval

Takeuchi

Sugiura

Akahoshi

et al. 2016

IEEJ Transactions Elec Engng

View full text Add to dashboard Cite

We consider the problem of searching scientific data from vast heterogeneous scientific data repositories. This problem is challenging because scientific data contain relatively little text information compared to other search targets such as web pages. On the other hand, the metadata in scientific data contain other characteristic information such as spatio‐temporal information. Although using this information make it possible to improve the search performance, many widely adopted scientific data search engines use this information exclusively for narrowing down search results. In this paper, we propose a novel query generation method using spatial, temporal, and text information based on pseudo relevance feedback. The proposed method generates new spatio‐temporal queries from the initial search results. By using these queries, the search results are reranked such that more related results obtain higher rank. The experimental results show that the proposed method outperforms a baseline method when search targets do not have rich text information. © 2016 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

show abstract

Data Management in Dynamic Environment-driven Computational Science

Cited by 9 publications

References 32 publications

Report of the DOE Workshop on Management, Analysis, and Visualization of Experimental and Observational data – The Convergence of Data and Computing

Report of the DOE Workshop on Management, Analysis, and Visualization of Experimental and Observational data – The Convergence of Data and Computing

Query capabilities of the Karma provenance framework

Spatio‐temporal pseudo relevance feedback for scientific data retrieval

Contact Info

Product

Resources

About