Metadata Systems for Data Lakes: Models and Features

Abstract

2 BIAL-X https://www.bial-x.com/ Abstract. Over the past decade, the data lake concept has emerged as an alternative to data warehouses for storing and analyzing big data. A data lake allows storing data without any predefined schema. Therefore, data querying and analysis depend on a metadata system that must be efficient and comprehensive. However, metadata management in data lakes remains a current issue and the criteria for evaluating its effectiveness are more or less nonexistent. In this paper, we introduce MEDAL, a generic, graph-based model for metadata management in data lakes. We also propose evaluation criteria for data lake metadata systems through a list of expected features. Eventually, we show that our approach is more comprehensive than existing metadata systems.

Spatial Data Lake for Smart Cities: From Design to Implementation

“…In summary, the implementation of a data warehouse goes a long way towards normalization of data. Data Lake definitions have been introduced in [11], a detailed comparison with data warehouses is proposed on [27] and revisited in [36]. Data lake is a recent solution that has emerged to meet the needs related to the management and use of big data on which the data warehouse have shown their weaknesses.…”

“…A data lake is a storage structure allowing to store massive data, from different sources in their native format without the need to perform processing beforehand [35,16]. According to [36], a data lake is a scalable storage system and analysis of data of all types, stored in their native format and essentially intended for data specialists who are among others statisticians, analysts and data scientists. The main characteristics associated with data lakes are:…”

“…Finally, the Inter-Metadata section (see Figure 4) records relationships between datasets, and enables users to have a view of data related to their initial query. Four types of relationships are proposed, based on [36]: parenthood, containment, similarity and thematic grouping.…”

“…Moreover, this approach, like the majority, has been set up for a very specific case study, and does not allow, or hardly takes into account, the case of complex data such as spatial data (satellite data). In terms of completeness of the state-of-the-art, regarding the features mentioned above, approaches such as [4,18,17,34,36,41] respectively cover more than the half of them. Unfortunately, these solutions are difficult to apply, since they are not very understandable, in the implementation ways, or have been proposed but not yet implemented.…”

“…In this paper, we particularly address this topic, starting from large heterogeneous data with a strong spatial dimension and elaborating on how to design and implement this specific case study. Inspired from previous works on spatial data normalization [19] and data lake design [40,27,36], we develop a specific methodology and the associated implementation code shared with the community 4 . We demonstrate that data lake infrastructures are not only expert dedicated but could be end-users oriented by offering a suitable query interface.…”