Metadata Extraction and Management in Data LakesWith GEMMS

Abstract

In addition to volume and velocity, Big data is also characterized by its variety. Variety in structure and semantics requires new integration approaches which can resolve the integration challenges also for large volumes of data. Data lakes should reduce the upfront integration costs and provide a more flexible way for data integration and analysis, as source data is loaded in its original structure to the data lake repository. Some syntactic transformation might be applied to enable access to the data in one common repository; however, a deep semantic integration is done only after the initial loading of the data into the data lake. Thereby, data is easily made available and can be restructured, aggregated, and transformed as required by later applications. Metadata management is a crucial component in a data lake, as the source data needs to be described by metadata to capture its semantics. We developed a Generic and Extensible Metadata Management System for data lakes (called GEMMS) that aims at the automatic extraction of metadata from a wide variety of data sources. Furthermore, the metadata is managed in an extensible metamodel that distinguishes structural and semantical metadata. The use case applied for evaluation is from the life science domain where the data is often stored only in files which hinders data access and efficient querying. The GEMMS framework has been proven to be useful in this domain. Especially, the extensibility and flexibility of the framework are important, as data and metadata structures in scientific experiments cannot be defined a priori.

On the Logical Design of a Prototypical Data Lake System for Biological Resources

“…Thus, to ensure data accessibility, exploration, and exploitation, an efficient and effective metadata system becomes an indispensible component in data lakes (Quix et al, 2016). Yet, most of the research work on data lakes still concentrate on structured data, or semi-structured data only (Farid et al, 2016;Farrugia et al, 2016;Madera and Laurent, 2016;Quix et al, 2016;Klettke et al, 2017). So far, unstructured data have not received enough consideration in the relevant research literature, while more often than not unstructured heterogeneous data occur frequently (Miloslavskaya and Tolstoy, 2016).…”

Data Lakes: Trends and Perspectives

“…We only have partial solutions in the literature. Some works concentrate on the detection of relationships between different datasets [1,9,27]. Some other work focus on the extraction of metadata for unstructured data (mostly textual data) [27,29].…”

Big Provenance Stream Processing for Data Intensive Computations

“…Such data-intensive processing environments are hard to manage [9] [78] as the data lifecycle inside them is so complicated. Given a data product, tracing its sources and finding all the processing steps applied on it is challenging.…”