DataFoundry: information management for scientific data

Abstract

Data warehouses and data marts have been successfully applied to a multitude of commercial business applications. They have proven to be invaluable tools by integrating information from distributed, heterogeneous sources and summarizing this data for use throughout the enterprise. Although the need for information dissemination is as vital in science as in business, working warehouses in this community are scarce because traditional warehousing techniques do not transfer to scientific environments. There are two primary reasons for this difficulty. First, schema integration is more difficult for scientific databases than for business sources, because of the complexity of the concepts and the associated relationships. While this difference has not yet been fully explored, it is an important consideration when determining how to integrate autonomous sources. Second, scientific data sources have highly dynamic data representations (schemata). When a data source participating in a warehouse changes its schema, both the mediator transferring data to the warehouse and the warehouse itself need to be updated to reflect these modifications. The cost of repeatedly performing these updates in a traditional warehouse, as is required in a dynamic environment, is prohibitive. This paper discusses these issues within the context of the DataFoundry project, an ongoing research effort at Lawrence Livermore National Laboratory. DataFoundry utilizes a unique integration strategy to identify corresponding instances while maintaining differences between data from different sources, and a novel architecture and an extensive meta-data infrastructure, which reduce the cost of maintaining a warehouse.

ANNODA: Tool for integrating Molecular-biological Annotation Data

“…On the other hand, the extraction, cleaning, transformation, and loading process can take considerable time and effort, which is a major drawback of Data warehousing. TSIMMIS [14] and DataFoundry [11] are examples of data warehousing system.…”

A standards-based framework to foster geospatial data and process interoperability

“…Two of the major issues to be dealt with in this layer are problems on what to store and how to fill in the gaps left by several types of acquisition errors. Selecting what is going to be stored is important since the amount of data acquired may be far too large to be stored in full 13,18 . Given that geospatial systems must also cope with streamed data, this raises the additional issue of providing query mechanisms that can cope with both stored and streamed data 3 .…”

Flexible Integration of Molecular-Biological Annotation Data: The GenMapper Approach

“…Current solutions mostly follow a data warehouse (e.g. IGD [ 30], GIMS [ 29], DataFoundry [ 16 ]) or federation approach (e.g. TAMBIS [ 21], P/FDM [ 24 ]) with a physical or virtual integration of data sources, respectively.…”