Managing scientific data

Abstract

68 communications of th e ac m | j u n e 2 0 1 0 | vo l . 5 3 | n o. 6 contributed articles DATA -orienTeD sC i e nT i f iC P ro C es se s depend on fast, accurate analysis of experimental data generated through empirical observation and simulation. However, scientists are increasingly overwhelmed by the volume of data produced by their own experiments. With improving instrument precision and the complexity of the simulated models, data overload promises to only get worse. The inefficiency of existing database management systems (DBMSs) for addressing the requirements of scientists has led to many application-specific systems. Unlike their general-purpose counterparts, these systems require more resources, hindering reuse of knowledge. Still, the data-management community aspires to generalpurpose scientific data management. Here, we explore the most important requirements of such systems and the techniques being used to address them.Observation and simulation of phenomena are keys for proving scientific theories and discovering facts of nature the human brain could otherwise never imagine. Scientists must be able to manage data derived from observations and simulations. Constant improvement of observational instruments and simulation tools give modern science effective options for abundant information capture, reflecting the rich diversity of complex life forms and cosmic phenomena. Moreover, the need for in-depth analysis of huge amounts of data relentlessly drives demand for additional computational support.Microsoft researcher and ACM Turing Award laureate Jim Gray once said, "A fourth data-intensive science is emerging. The goal is to have a world in which all of the science literature is online, all the science data is online, and they interoperate with each other."9 Unfortunately, today's commercial data-management tools are incapable of supporting the unprecedented scale, rate, and complexity of scientific data collection and processing.Despite its variety, scientific data does share some common features: Needed are generic, rather than one-off, DBMS solutions automating storage and analysis of data from scientific collaborations.BY anastasia aiLamaKi, VeRena KanteRe, anD DeBaBRata Dash managing scientific Data key insights managing the enormous amount of scientific data being collected is the key to scientific progress. though technology allows for the extreme collection rates of scientific data, processing is still performed with stale techniques developed for small data sets; efficient processing is necessary to be able to exploit the value of huge scientific data collections.Proposed solutions also promise to achieve efficient management for almost any other kind of data. Lack of complete solutions using commercial DBMSs has led scientists in all fields to develop or adopt application-specific solutions, though some have been added on top of commercial DBMSs; for example, the Sloan Digital Sky Survey (SDSS-1 and SDSS-2; http://www.sdss.org/) uses SQL Server as its backend. Moreover, the resulting sof...

"They're all going out to something weird"

“…This is to reduce what is seen to be a heavy manual overhead [49]. Particular interest has been paid to how metadata is preserved across workflows in the work of scientists where data may have to be re-used and curated by a number of different parties [2]. Here there is particular emphasis upon the possible role of automation (see [3,4]) rather than providing support for contingent interests.…”

Datom: Towards Modular Data Management

“…Modular data management will contribute to the efficient and automated processing of heterogeneous data, by offering decomposition of complex queries across various data sources and integration of query results. Scientific data management suffers from lack of automation, online processing, data and process integration [3]. Currently, scientists need to collaborate tightly with computer engineers to develop custom solutions that efficiently support data storage and analysis for different experiments [5].…”

VisInfo: a digital library system for time series research data based on exploratory search—a user-centered design approach

“…A classification of different types of research data is presented by Kehrer and Hauser [33], with an emphasis on visualization and visual data analysis. The process of passing research data through different phases is often referred to as the data-life-cycle [3,13,17]. The typical phases are data creation, data processing, data analysis, data preservation, data access, and data re-use.…”

“…The typical phases are data creation, data processing, data analysis, data preservation, data access, and data re-use. The VisInfo approach is primarily targeted towards the phases of data analysis, data access, and data re-use, and aims at tackling respective challenges related to search [13,24,40], and exploration [3,25,68] tasks (see Sect. 2.2 for more details).…”