Constance

Hai, Rihan; Geisler, Sandra; Quix, Christoph

doi:10.1145/2882903.2899389

Cited by 155 publications

(51 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The framework is also extensible as new types of data sources can be easily integrated as we have shown in the evaluation. Metadata extraction is one of the core features of our data lake implementation [27]. Based on the extracted metadata and schema information, further methods for the semantic enrichment of the data lake are currently being implemented.…”

Section: Discussionmentioning

confidence: 99%

Metadata Extraction and Management in Data LakesWith GEMMS

Quix

Hai

Vatov

2016

CSIMQ

Self Cite

View full text Add to dashboard Cite

In addition to volume and velocity, Big data is also characterized by its variety. Variety in structure and semantics requires new integration approaches which can resolve the integration challenges also for large volumes of data. Data lakes should reduce the upfront integration costs and provide a more flexible way for data integration and analysis, as source data is loaded in its original structure to the data lake repository. Some syntactic transformation might be applied to enable access to the data in one common repository; however, a deep semantic integration is done only after the initial loading of the data into the data lake. Thereby, data is easily made available and can be restructured, aggregated, and transformed as required by later applications. Metadata management is a crucial component in a data lake, as the source data needs to be described by metadata to capture its semantics. We developed a Generic and Extensible Metadata Management System for data lakes (called GEMMS) that aims at the automatic extraction of metadata from a wide variety of data sources. Furthermore, the metadata is managed in an extensible metamodel that distinguishes structural and semantical metadata. The use case applied for evaluation is from the life science domain where the data is often stored only in files which hinders data access and efficient querying. The GEMMS framework has been proven to be useful in this domain. Especially, the extensibility and flexibility of the framework are important, as data and metadata structures in scientific experiments cannot be defined a priori.

show abstract

Section: Discussionmentioning

confidence: 99%

Metadata Extraction and Management in Data LakesWith GEMMS

Quix

Hai

Vatov

2016

CSIMQ

Self Cite

View full text Add to dashboard Cite

show abstract

“…Constance 84,85 is a Data Lake (DL) system with sophisticated metadata management over raw data extracted from heterogeneous data sources. Constance discovers, extracts, and summarizes the structural metadata from the data sources, and annotates data and metadata with semantic information to avoid ambiguities.…”

Section: Related Workmentioning

confidence: 99%

IPDS: A semantic mediator‐based system using Spark for the integration of heterogeneous proteomics data sources

Messaoudi

Fissoune

Badir

2020

Concurrency and Computation

View full text Add to dashboard Cite

Summary With the constant rise of data volumes in many disciplines, various new Big data management systems have emerged to provide scalable tools for efficient data integration, processing, and analysis. In this article, we provide an overview of biomedical data integration systems focusing on ontology‐based semantic systems and Big data technologies based systems such as Apache Spark. We also propose a new semantic data integration system, called Integrated Proteomics Data System (IPDS), which uses a mediator approach. IPDS provides users a unified interface for query processing and data exploration. This system takes advantage of the Apache Spark framework to perform the query transformation and execution needed to question the integrated data sources. We develop a domain ontology that allows the user to formulate its queries in terms defined in the ontology. IPDS is a case study of semantic proteomics data integration linking four data sources UniProt (protein annotation), String (protein‐protein interaction), PDB (protein structure), and Pubmed (biomedical citation).

show abstract

“…We identify in the literature six main functionalities that should ideally be provided by the metadata system of a data lake. Semantic enrichment (SE), also called semantic annotation [11] or semantic profiling [2], consists in generating a description of the context of data, e.g., with tags, to make them more interpretable and understandable [27]. It is done using knowledge bases such as ontologies.…”

Section: Expected Featuresmentioning

confidence: 99%

“…In this context, the concept of data lake [6] appears as a solution to big data heterogeneity problems. A data lake provides integrated data storage without predefined schema [11]. In the absence of a data schema, an effective metadata system becomes essential to make data queryable and thus prevent the lake from turning into a data swamp, i.e., an inexploitable data lake [1,11,26].…”

Section: Introductionmentioning

confidence: 99%

“…While the literature seems unanimous about the importance of the metadata system in a data lake, questions and uncertainties remain about its implementation methodology. Several approaches help organize metadata, but most concern only structured and semi-structured data [8,11,17,22]. Moreover, the effectiveness of a metadata system is difficult to measure because, to the best of our knowledge, there are no widely shared and accepted evaluation criteria.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Metadata Systems for Data Lakes: Models and Features

Sawadogo

Scholly

Favre

et al. 2019

Communications in Computer and Information Science

View full text Add to dashboard Cite

2 BIAL-X https://www.bial-x.com/ Abstract. Over the past decade, the data lake concept has emerged as an alternative to data warehouses for storing and analyzing big data. A data lake allows storing data without any predefined schema. Therefore, data querying and analysis depend on a metadata system that must be efficient and comprehensive. However, metadata management in data lakes remains a current issue and the criteria for evaluating its effectiveness are more or less nonexistent. In this paper, we introduce MEDAL, a generic, graph-based model for metadata management in data lakes. We also propose evaluation criteria for data lake metadata systems through a list of expected features. Eventually, we show that our approach is more comprehensive than existing metadata systems.

show abstract

Constance

Cited by 155 publications

References 7 publications

Metadata Extraction and Management in Data LakesWith GEMMS

Metadata Extraction and Management in Data LakesWith GEMMS

IPDS: A semantic mediator‐based system using Spark for the integration of heterogeneous proteomics data sources

Metadata Systems for Data Lakes: Models and Features

Contact Info

Product

Resources

About