A Data Lake Metadata Enrichment Mechanism via Semantic Blueprints

Pingos, Michalis; Andreou, Andreas S.

doi:10.5220/0011080400003176

Cited by 8 publications

(20 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figure 1 The five primary types of data sources This paper extends and enhances previous work on the topic [13] which adopts the basic principles of manufacturing blueprints [20] and modifies their purpose and meaning to reflect the description and characterization of sources and the data they produce via the utilization of the five Big Data characteristics (5Vs -see figure 2). These characteristics essentially describe data sources by means of specific types of blueprints through an ontology-based description representation.…”

Section: • Expandabilitymentioning

confidence: 81%

“…The work in [13] extended the aforementioned list of characteristics by comparing the metadata mechanism with the two most completed systems as presented in [14]: CoreKG [15] and MEDAL [14]. The new list of the characteristics include:…”

Section: • DL Accept Any Data Without Oversight and Governancementioning

confidence: 99%

“…The goal of the proposed framework is to enrich the existing novel DL pond metadata mechanism framework described in [13] with the Blockchain technology so as to provide more security and privacy and to ensure that the data in the DL have not been modified or altered.…”

Section: Dlmetachain Framework Architecturementioning

confidence: 99%

“…According to [13] each candidate source, including IoT data sources, needs to be characterized first by a metadata mechanism namely Data Source Βlueprint (DSB) before it becomes a member of a DL [13]. This characterization is performed with the contribution of the 5 basic characteristics of Big Data, namely Volume, Variety, Veracity, Value, and Velocity.…”

Section: A DL Blueprintmentioning

confidence: 99%

See 3 more Smart Citations

DLMetaChain: An IoT Data Lake Architecture Based on the Blockchain

Pingos

Christodoulou

Andreou

2022

2022 13th International Conference on Information, Intelligence, Systems &Amp; Applications (IISA)

Self Cite

View full text Add to dashboard Cite

Nowadays, the IoT ecosystem is evolving rapidly, with multiple heterogeneous sources producing high volumes of data and processes transforming this data into meaningful or "smart" information . These volumes of data, including IoT data, need to be stored in repositories that can host raw, unprocessed, relational and non-relational types of data, such as Data Lakes. Due to the weakness of metadata management, security & access control is one of the main challenges of Big Data storage architectures as Data Lakes can be replaced without oversight of the contents. Recently, the Blockchain technology has been introduced as an effective solution to build trust between different entities, where trust is either nonexistent or unproven, and to address security and privacy concerns. In this paper we introduce DLMetaChain, an extended Data Lake metadata mechanism that consists of data from heterogeneous data sources which interact with IoT data. The extended mechanism mainly focuses on developing an architecture to ensure that the data in the Data Lake is not modified or altered by taking into advantage the capabilities of the Blockchain.

show abstract

Section: • Expandabilitymentioning

confidence: 81%

Section: • DL Accept Any Data Without Oversight and Governancementioning

confidence: 99%

Section: Dlmetachain Framework Architecturementioning

confidence: 99%

Section: A DL Blueprintmentioning

confidence: 99%

See 2 more Smart Citations

DLMetaChain: An IoT Data Lake Architecture Based on the Blockchain

Pingos

Christodoulou

Andreou

2022

2022 13th International Conference on Information, Intelligence, Systems &Amp; Applications (IISA)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The set of SDB essentially describes properties of data via stable attributes, such as variety, value, velocity, veracity, and attributes that are not stable over time, such as volume, last source update and keywords. The proposed approach builds upon previous work on the topic that introduced a semantic metadata enrichment mechanism for Data Lakes [15], which allows for the efficient storing and retrieval of data belonging to dispersed and heterogenous data sources. The same concepts are extended, modified and adapted in this work to match the characteristics of Data Meshes.…”

Section: Introductionmentioning

confidence: 99%

Discovering Data Domains and Products in Data Meshes Using Semantic Blueprints

Pingos,

Andreou

2024

Preprint

View full text Add to dashboard Cite

Nowadays, one of the greatest challenges in Data Meshes revolves around detecting and creating Data Domains and Data Products for providing the ability to adapt easily and quickly to changing business needs. This requires a disciplined approach to identify, differentiate and prioritize distinct data sources according to their content and diversity. The current paper tackles this highly com-plicated issue and suggests a standardized approach that integrates the concept of Data Blueprints with Data Meshes. In essence, a novel standardization framework is proposed that creates Data Products using a metadata semantic enrichment mechanism, the latter also offering Data Domain readiness and alignment. The approach is demonstrated using real-world data produced by mul-tiple sources in a poultry meat production factory. A set of functional attributes is used to compare qualitatively the proposed approach against existing data structures utilized in storage architectures with quite promising results. Finally, experimentation with different scenarios varying in data product complexity and granularity suggests successful performance.

show abstract

Exploiting Metadata Semantics in Data Lakes Using Blueprints

Pingos

Andreou

2023

Communications in Computer and Information Science

View full text Add to dashboard Cite

A Data Lake Metadata Enrichment Mechanism via Semantic Blueprints

Cited by 8 publications

References 0 publications

DLMetaChain: An IoT Data Lake Architecture Based on the Blockchain

DLMetaChain: An IoT Data Lake Architecture Based on the Blockchain

Discovering Data Domains and Products in Data Meshes Using Semantic Blueprints

Exploiting Metadata Semantics in Data Lakes Using Blueprints

Contact Info

Product

Resources

About