Cooperating services for data-driven computational experimentation

Abstract

D ata-driven computational science is characterized by dynamic adaptation in response to external data. Applications of this type, which are often data-and I/O-intensive, run as parallel or distributed computations on high-end resources such as distributed clusters or symmetric multiprocessing machines. Ondemand weather forecasting is a canonical example of a data-driven application. Essentially, it is the automated process of invoking a forecast model run in response to the detection of a severe weather condition. The existing framework for running forecast models has drawbacks that we must overcome to bring about dynamic on-demand forecasting. For instance, the current investigation process requires a lot of human involvement, particularly in the staging and moving of the files required by the model and in the invocation of downstream tools to visualize and analyze model results. Although scripts exist to automate some of the process, a knowledgeable expert must properly configure them.The Linked Environments for Atmospheric Discovery (LEAD) project addresses the limitations of current weather forecast frameworks through a new, service-oriented architecture capable of responding to unpredicted weather events and response patterns in real time. These services are intended to support the execution of multimodel simulations of weather forecasts on demand across a distributed grid of resources while dynamically adapting resource allocation in response to the results. At the system's heart is a suite of core services that together provide the essential functionality needed to invoke and run a complex experiment with minimal human involvement. Specifically, it lets the user define an experiment workflow, execute the experiment, and store the results. (See the "Related Work in Grid Technology" sidebar for a discussion of other work in this area.)In this article, we focus on three MyLEAD services-the metadata catalog service, notification service, and workflow service-that together form the core services for managing complex experimental meteorological investigations and managing the data products used in and generated during the computational experimentation. We show how the services work together on the user's behalf, easing the technological burden on the scientists and freeing them to focus on more of the science that compels them. User interaction with the system is through the LEAD portal. We've shown the ser-

Science on the TeraGrid

“…The Linked Environments for Atmospheric Discovery II (LEAD II) [1,2] project is a multi-disciplinary collaboration that applies advanced cyberinfrastructure to bridge the gap between environmental scientists, weather data and tools, and large-scale computational resources needed to interpret the weather. LEAD II evolved from a National Science Foundation (NSF) funded grant (2003)(2004)(2005)(2006)(2007)(2008)(2009)) that involved nine institutions and more than 100 scientists, students and technical staff.…”

Service-oriented environments for dynamically interacting with mesoscale weather

“…The workflow instance, itself a Web service that runs in a workflow engine hosted on the LEAD grid, has a highly interactive relationship with the myLEAD personal metadata catalog. 2 The workflow shown in Figure 13, along with other workflows and the basic capabilities needed to create, edit, and execute them, are now running on the LEAD grid.…”

“…In a companion article, 2 we describe the organization and cataloging of metadata as LEAD workflows generate them. Here, we present a more holistic view of LEAD and focus on the rationale behind and structure of the architecture.…”

Data Management in Dynamic Environment-driven Computational Science

“…The myLEAD agent responds to activities being carried out on the LEAD Cyberinfrastructure by listening on an event notification channel [17] on which status events about workflow progress are published. The agent uses the notification messages to determine the current state of a particular workflow, and actively manages the user space by, for instance, creating a new collection when a major mode transition has taken place [29]. Users interact with the tools primarily through the LEAD Cyberinfrastructure [11].…”