Data access and analysis with distributed federated data servers in climate<i>prediction</i>.net

Abstract

Abstract. climateprediction.net is a large public resource distributed scientific computing project. Members of the public download and run a full-scale climate model, donate their computing time to a large perturbed physics ensemble experiment to forecast the climate in the 21st century and submit their results back to the project. The amount of data generated is large, consisting of tens of thousands of individual runs each in the order of tens of megabytes. The overall dataset is, therefore, in the order of terabytes. Access and analysis of the data is further complicated by the reliance on donated, distributed, federated data servers. This paper will discuss the problems encountered when the data required for even a simple analysis is spread across several servers and how webservice technology can be used; how different user interfaces with varying levels of complexity and flexibility can be presented to the application scientists, how using existing web technologies such as HTTP, SOAP, XML, HTML and CGI can engender the reuse of code across interfaces; and how application scientists can be notified of their analysis' progress and results in an asynchronous architecture.

Attribution of changes in precipitation patterns in African rainforests

“…The method as introduced by Stone & Allen [14] requires access to a sufficiently large number of simulations to enable sampling of the uncertainty, so that statistics of the occurrence of a rare event can be estimated with confidence. The weather@ home project provides such a large ensemble using publicly volunteered distributed computing [15,16]. Studies employing PEA rely heavily on climate modelling and have concentrated on European and US American events.…”

“…Also, a sufficiently long period of time is to be simulated to evaluate model bias (figure 1), and to determine whether the model captures the observed distribution of the relevant variables (not shown). To generate a sufficiently large ensemble, the model was run for several years many hundreds of times with different initial conditions within the climateprediction.net weather@ home project [15,16] via volunteer distributed computing. Output of the global model for the region of interest provides only monthly diagnostics, which is not ideal to look at precipitation extreme events relevant for flooding, for example, however, this is no disadvantage for the study since the deficit in rainfall accumulated over months is especially important for rainforests.…”

Reconciling two approaches to attribution of the 2010 Russian heat wave

“…The method requires access to a sufficiently large number of simulations so that statistics of the occurrence of a rare event can be estimated with confidence. The weather at home project provides such a large ensemble using publicly volunteered distributed computing [ Allen , 1999; Massey et al , 2006].…”

Accessibility-Based Resource Selection in Loosely-Coupled Distributed Systems

“…Such data-intensive applications encompass a variety of domains such as high energy physics [34], climate prediction [27], astronomy [3] and bioinformatics [5]. For example, in high energy physics applications such as the Large Hadron Collider (LHC), thousands of physicists worldwide will require access to shared, immutable data at the scale of petabytes [19].…”