A software architecture-based framework for highly distributed and data intensive scientific applications

Abstract

Modern scientific research is increasingly conducted by virtual communities of scientists distributed around the world. The data volumes created by these communities are extremely large, and growing rapidly. The management of the resulting highly distributed, virtual data systems is a complex task, characterized by a number of formidable technical challenges, many of which are of a software engineering nature. In this paper we describe our experience over the past seven years in constructing and deploying OODT, a software framework that supports large, distributed, virtual scientific communities. We outline the key software engineering challenges that we faced, and addressed, along the way. We argue that a major contributor to the success of OODT was its explicit focus on software architecture. We describe several large-scale, real-world deployments of OODT, and the manner in which OODT helped us to address the domainspecific challenges induced by each deployment.

A comparative analysis of software architecture recovery techniques

“…We assess the accuracy of these techniques on eight architectures from six different open-source systems: ArchStudio [16], Bash [13], Hadoop [4], Linux [12], Mozilla [2], and OODT [28]. In the case of two of the systems-Linux and Mozilla-we use a pair of architectures each, at different levels of detail.…”

DRAT: An Unobtrusive, Scalable Approach to Large Scale Software License Analysis

“…DRAT is a MapReduce (M/R) style [7] RAT workflow that runs on top of Apache Object Oriented Data Technology (OODT) [8], a scientific data processing, acquisition, and dissemination system. M/R, a methodology for processing a large amount of unstructured data with a parallel, distributed algorithm, consists of two components: "mapper" and "reducer".…”

Injecting software architectural constraints into legacy scientific applications

“…While we have met with success in applying software architectures to other software built at JPL, such as the Planetary Data System [5], the project described in this paper is the first that incorporates JPL's more recently developed software services built in a componentized fashion with well-defined interfaces, as well as multiple scientific processing codes known as PGEs. These PGEs include a number of legacy radar packages (imaging and interferometry) and trajectory packages.…”