DLMetaChain: An IoT Data Lake Architecture Based on the Blockchain

Pingos, Michalis; Christodoulou, Panayiotis; Andreou, Andreas S.

doi:10.1109/iisa56318.2022.9904404

Cited by 6 publications

(5 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The proposed approach expands earlier research on the subject that proposed SDB, a semantic metadata enrichment technique for data lakes that enables the effective storing and retrieval of data from distributed and heterogeneous data sources, as well as ensuring security in data lakes using Blockchain technology and NFTs [10][11][12]. The same concepts are employed in this study, but this time they align with the characteristics of data meshes, ensuring security and ownership through the integration of Blockchain and NFT technology, thereby paving the way for the development of data markets.…”

Section: Introductionmentioning

confidence: 78%

See 1 more Smart Citation

Security and Ownership in User-Defined Data Meshes

Pingos,

Christodoulou,

Andreou

2024

Algorithms

Self Cite

View full text Add to dashboard Cite

Data meshes are an approach to data architecture and organization that treats data as a product and focuses on decentralizing data ownership and access. It has recently emerged as a field that presents quite a few challenges related to data ownership, governance, security, monitoring, and observability. To address these challenges, this paper introduces an innovative algorithmic framework leveraging data blueprints to enable the dynamic creation of data meshes and data products in response to user requests, ensuring that stakeholders have access to specific portions of the data mesh as needed. Ownership and governance concerns are addressed through a unique mechanism involving Blockchain and Non-Fungible Tokens (NFTs). This facilitates the secure and transparent transfer of data ownership, with the ability to mint time-based NFTs. By combining these advancements with the fundamental tenets of data meshes, this research offers a comprehensive solution to the challenges surrounding data ownership and governance. It empowers stakeholders to navigate the complexities of data management within a decentralized architecture, ensuring a secure, efficient, and user-centric approach to data utilization. The proposed framework is demonstrated using real-world data from a poultry meat production factory.

show abstract

Section: Introductionmentioning

confidence: 78%

“…An enhanced DL metadata framework called DLMetaChain was introduced in [12], which can manage data from diverse sources like IoT data using Blockchain. The paper discussed the changing IoT ecosystem, where a variety of sources produce large amounts of data that are then converted into useful information.…”

Section: Related Researchmentioning

confidence: 99%

Security and Ownership in User-Defined Data Meshes

Pingos,

Christodoulou,

Andreou

2024

Algorithms

Self Cite

View full text Add to dashboard Cite

show abstract

“…The Blockchain and NFT technologies are considered as a viable option for tackling security and privacy concerns, as well as for developing trust between entities, where trust has either been under-developed or nonexistent. This enhanced approach focused on creating an architecture that guarantees the DL data will not be changed or altered [24].…”

Section: Related Workmentioning

confidence: 99%

“…This section summarized the main notions in [15] and [24] that aimed to improve the architecture of DLs and address related challenges, such as security and privacy. In addition, it depicted the use of an appropriate metadata mechanism that contributes to the efficient storing of the data in a DL and its processing to provide insights.…”

Section: Related Workmentioning

confidence: 99%

Discovering Data Domains and Products in Data Meshes Using Semantic Blueprints

Pingos,

Andreou

2024

Preprint

View full text Add to dashboard Cite

Nowadays, one of the greatest challenges in Data Meshes revolves around detecting and creating Data Domains and Data Products for providing the ability to adapt easily and quickly to changing business needs. This requires a disciplined approach to identify, differentiate and prioritize distinct data sources according to their content and diversity. The current paper tackles this highly com-plicated issue and suggests a standardized approach that integrates the concept of Data Blueprints with Data Meshes. In essence, a novel standardization framework is proposed that creates Data Products using a metadata semantic enrichment mechanism, the latter also offering Data Domain readiness and alignment. The approach is demonstrated using real-world data produced by mul-tiple sources in a poultry meat production factory. A set of functional attributes is used to compare qualitatively the proposed approach against existing data structures utilized in storage architectures with quite promising results. Finally, experimentation with different scenarios varying in data product complexity and granularity suggests successful performance.

show abstract

“…A DL often contains proprietary or sensitive information that requires appropriate security measures. Security measures in a DL can be implemented to only allow individuals to have partial access to selected information and anonymize or encrypt data for users in a variety of roles who do not have access to the original data [34]. The lake should also allow one to have ubiquitous access to data from anywhere at any time.…”

Section: Related Workmentioning

confidence: 99%

Data Lake Architecture for Storing and Transforming Web Server Access Log Files

Zagan

Danubianu

2023

IEEE Access

View full text Add to dashboard Cite

Web server access log files are text files containing important data about server activities, client requests addressed to a server, server responses, etc. Large-scale analysis of these data can contribute to various improvements in different areas of interest. The main problem lies in storing these files in their raw form, over long time, to allow analysis processes to be run at any time enabling information to be extracted as foundation for high quality decisions. Our research focuses on offering an economical, secure, and highperformance solution for the storage of large amount of raw log files. Proposed system implements a Data Lake (DL) architecture in cloud using Azure Data Lake Storage Gen2 (ADLS Gen2) for extract-loadtransform (ELT) pipelines. This architecture allows large volumes of data to be stored in their raw form. Afterwards they can be subjected to transformation and advanced analysis processes without the need of a structured writing scheme. The main contribution of this paper is to provide a solution that is affordable and more accessible to perform web server access log data ingestion, storage and transformation over the newest technology, Data Lake. As derivative contribution, we proposed the use of Azure Blob Trigger Function to implement the algorithm of transforming log files into parquet files leading to 90% reduction in storage space compared to their original size. That means much lower storage costs than if they had been stored as log files.A hierarchical data storage model has also been proposed for shared access to data over different layers in the DL architecture, on top of which Data Lifecycle Management (DLM) rules have been proposed for storage cost efficiency. We proposed ingesting log files into a Data Lake deployed in cloud due to ease of deployment and low storage costs. The aim is to maintain this data in the long term, to be used in future advanced analytics processes by cross-referencing with other organizational or external data. That could bring important benefits. While the proposed solution is explicitly based on ADLS Gen2, it represents an important benchmark in approaching a cloud DL solution offered by any other vendor.

show abstract

DLMetaChain: An IoT Data Lake Architecture Based on the Blockchain

Cited by 6 publications

References 17 publications

Security and Ownership in User-Defined Data Meshes

Security and Ownership in User-Defined Data Meshes

Discovering Data Domains and Products in Data Meshes Using Semantic Blueprints

Data Lake Architecture for Storing and Transforming Web Server Access Log Files

Contact Info

Product

Resources

About