Modeling Data Lakes with Data Vault: Practical Experiences, Assessment, and Lessons Learned

Giebler, Corinna; Gröger, Christoph; Hoos, Eva; Schwarz, Holger; Mitschang, Bernhard

doi:10.1007/978-3-030-33223-5_7

Cited by 20 publications

(14 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Meanwhile, DL-related research problems are raising massive attention associated with the implementation of DL prototypes. A large range of challenges are discussed such as metadata management [56], data quality [41], data provenance [120], metadata enrichment [10,59], data preparation [82], dataset organization [6,131], modeling [108,98,52], data integration [60,57] and related dataset discovery [43,16,131,14,130]. Such data lake proposals targeting specific research challenges, are also our main focus in this survey, which will be addressed in Sec.…”

Section: -Present: Prosperity and Diversitymentioning

confidence: 99%

“…The Hadoop Distributed File System (HDFS) is one of the most frequently mentioned data storage systems for data lakes [21,119,13]. HDFS supports a wide range of DATAMARAN [48] Skluma [118] Metadata modeling Generic metadata model (GEMMS [108], Constance [59]) Data vault [98,52] Graph-based metadata model (Diamantiniet al [32,33], Aurum [43], Sawadogoet et al [113])…”

Section: File-based Storage Systemsmentioning

confidence: 99%

See 1 more Smart Citation

Data lake concept and systems: a survey

Hai¹,

Koutras²,

Quix³

et al. 2021

Preprint

View full text Add to dashboard Cite

Although big data has been discussed for some years, it still has many research challenges, especially the variety of data. It poses a huge difficulty to efficiently integrate, access, and query the large volume of diverse data in information silos with the traditional 'schema-on-write' approaches such as data warehouses. Data lakes have been proposed as a solution to this problem. They are repositories storing raw data in its original formats and providing a common access interface. This survey reviews the development, definition, and architectures of data lakes. We provide a comprehensive overview of research questions for designing and building data lakes. We classify the existing data lake systems based on their provided functions, which makes this survey a useful technical reference for designing, implementing and applying data lakes. We hope that the thorough comparison of existing solutions and the discussion of open research challenges in this survey would motivate the future development of data lake research and practice.

show abstract

Section: -Present: Prosperity and Diversitymentioning

confidence: 99%

Section: File-based Storage Systemsmentioning

confidence: 99%

Data lake concept and systems: a survey

Hai¹,

Koutras²,

Quix³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…One may view this as modelling for all possibilities of interactions between data, even those not required by the current business specification document. Proponents of data vaults, Giebler et al (2019) highlight the fact that data are stored in its raw format which improves the auditability of results and flexibility of the system for future adaptability of requirements. The popularity of data vault modelling is increasing and clear principles for the implementation of BISs using data vaults have been developed by industry role players (Stroo & Broekmans, 2018).…”

Section: Theoretical Perspectivesmentioning

confidence: 99%

Sustainable business intelligence systems: Modelling for the future

Goede

2021

Syst Res Behav Sci

View full text Add to dashboard Cite

Business intelligence systems are used in support of strategic decision‐making in organisations. Strategic decisions in support of the sustainable development goals require accurate and accessible data. Of equal importance is the sustainability of the business information systems themselves. Not only should they be sensitive to resource consumption but they should also be designed for the future needs of the organisation. We argue that sustainable business intelligence systems should represent as many perspectives in the organisation as possible supported by the available data. In order to achieve this, a critical (Kantian) systems perspective is suggested that foster appreciation of different perspectives in the organisation. Prevailing belief in the business intelligence system industry is that a key business problem should drive development of the system. Our position is more future oriented in that we argue that inclusion of all available perspectives on business problems will prolong the life of the system, thereby making it more sustainable. This is not an idealistic requirement when a data vault is used for the data model of the system. Contrary to the more common dimensional modelling, data vault modelling does not require restrictive decisions to be made on the scope of the system. In fact, the worst‐case scenario is addressed in the initial model. The paper provides a conceptual discussion of how a business intelligence system should be designed and developed to ensure that it is sustainable. Since the right system should be developed in the right way, we incorporate known success factors into our discussion. We argue that the critical (Kantian) systems perspective can lead to the right system and that data vault modelling in context of lessons from identified success factors is the right way to develop the business intelligence system.

show abstract

“…While it has been acknowledged that metadata management is an important aspect in DLs, there are only few works on the modeling of data and metadata in a DL. Data vault is a dimensional modeling technique frequently applied in DW projects; in Giebler et al (2019), this modeling technique is applied to DLs and compared with other techniques. Because of the fragmentation of the data in many different tables, querying is expensive due to many join operations.…”

Section: Data Models and Semantics In Data Lakesmentioning

confidence: 99%

Data Lake

Quix¹,

Geisler²,

Hai³

2020

Encyclopedia of Big Data

View full text Add to dashboard Cite

Data lakes (DL) have been proposed as a new concept for centralized data repositories. In contrast to data warehouses (DW), which usually require a complex and fine-tuned Extract-Transform-Load (ETL) process, DLs use a simpler model which just aims at loading the complete source data in its raw format into the DL. While a more complex ETL process with data transformation and aggregation increases the data quality, it might also come with some information loss as irregular or unstructured data not fitting into the integrated DW schema will not be loaded into the DW. Moreover, some data silos might not get connected to integrated data repositories at all due to the complexity of the data integration process. DLs address these problems: they should provide access to the source data in its original format without requiring an elaborated ETL process to ingest the data into the lake.

show abstract

Modeling Data Lakes with Data Vault: Practical Experiences, Assessment, and Lessons Learned

Cited by 20 publications

References 10 publications

Data lake concept and systems: a survey

Data lake concept and systems: a survey

Sustainable business intelligence systems: Modelling for the future

Data Lake

Contact Info

Product

Resources

About