Ensuring reliable datasets for environmental models and forecasts

Abstract

At the dawn of the 21 st century, environmental scientists are collecting more data more rapidly than at any time in the past. Nowhere is this change more evident than in the advent of sensor networks able to collect and process (in real time) simultaneous measurements over broad areas and at high sampling rates. At the same time there has been great progress in the development of standards, methods, and tools for data analysis and synthesis, including a new standard for descriptive metadata for ecological datasets (Ecological Metadata Language) and new workflow tools that help scientists to assemble datasets and to diagram, record, and execute analyses. However these developments (important as they are) are not yet sufficient to guarantee the reliability of datasets created by a scientific process -the complex activity that scientists carry out in order to create a dataset. We define a dataset to be reliable when the scientific process used to create it is (1) reproducible and (2) analyzable for potential defects.To address this problem we propose the use of an analytic web, a formal representation of a scientific process that consists of three coordinated graphs (a data-flow graph, a dataset-derivation graph, and a process-derivation graph) originally developed for use in software engineering. An analytic web meets the two key requirements for ensuring dataset reliability: (1) a complete audit trail of all artifacts (e.g., datasets, code, models) used or created in the execution of the scientific process that created the dataset, and (2) detailed process metadata that precisely describe all sub-processes of the scientific process. Construction of such metadata requires the semantic features of a high-level process definition language.In this paper we illustrate the use of an analytic web to represent the scientific process of constructing estimates of ecosystem water flux from data gathered by a complex, realtime multi-sensor network. We use Little-JIL, a high-level process definition language, to precisely and accurately capture the analytical processes involved. We believe that incorporation of this approach into existing tools and evolving metadata specifications (such as EML) will yield significant benefits to science. These benefits include: complete and accurate representations of scientific processes; support for rigorous evaluation of such processes for logical and statistical errors and for propagation of measurement error; and assurance of dataset reliability for developing sound models and forecasts of environmental change.3

Experience in using a process language to define scientific workflow and generate dataset provenance

“…Over time, the number, nature, and variety of sensors to be used in this research, and datasets resulting from it, will grow. The system is described in detail in [3].…”

“…The icons represent dataset types (boxes) or process types (ovals) and arrows indicate the flow of datasets into and out of processes (from [3]). …”

“…The totality of data and capabilities produced and consumed in this way can thus be thought of as a scientific data processing enterprise. In this work we refer to it as an analytic web [1][2][3]. There is a need for strong support of this enterprise by supporting both the production of such datasets (a particular focus of current Scientific Workflow systems) and their consumption by those wanting to access them for their own subsequent scientific investigations (a need that currently seems less well served).…”

“…A key objective of this paper, however, is to demonstrate that current Scientific Workflow technologies can be improved upon by drawing upon experiences and technologies developed to facilitate support for such key concepts as abstraction, exception management and concurrency control. In previous papers [1,3] we presented to the ecological research community an indication of the promise of our approach. This paper discusses the challenges and research opportunities that this problem domain presents for software engineering research.…”

Clear and Precise Specification of Ecological Data Management Processes and Dataset Provenance

“…The totality of data produced and consumed by a working scientific team, combined with all the processes used to transform and analyze those data, comprise a scientific data processing enterprise, that can be formally represented by what we call an analytic web [6], [12], [22]. An analytic web should provide facilities for producing and accessing all such data as well as its associated metadata.…”