Towards Information Profiling: Data Lake Content Metadata Management

Alserafi, Ayman; Abelló, Alberto; Romero, Óscar; Calders, Toon

doi:10.1109/icdmw.2016.0033

Cited by 34 publications

(42 citation statements)

References 18 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…So that it is mandatory to set up a metadata management system for DL. In fact, the importance of metadata has been emphasized in many papers [1,7,31]. The first research problem that needs to be solved is the content of the metadata.…”

Section: Metadatamentioning

confidence: 99%

See 1 more Smart Citation

Data Lakes: Trends and Perspectives

Ravat

Yan

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

As a relatively new concept, data lake has neither a standard definition nor an acknowledged architecture. Thus, we study the existing work and propose a complete definition and a generic and extensible architecture of data lake. What's more, we introduce three future research axes in connection with our health-care Information Technology (IT) activities. They are related to (i) metadata management that consists of intra-and inter-metadata, (ii) a unified ecosystem for companies' data warehouses and data lakes and (iii) data lake governance.

show abstract

Section: Metadatamentioning

confidence: 99%

“…We only have partial solutions in the literature. Some works concentrate on the detection of relationships between different datasets [1,9,27]. Some other work focus on the extraction of metadata for unstructured data (mostly textual data) [27,29].…”

Section: Metadatamentioning

confidence: 99%

Data Lakes: Trends and Perspectives

Ravat

Yan

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Initial Accepted Scopus 108 53 papers: [1]- [3], [5], [9], [10], [13]- [19], [23]- [29], [31]- [33], [37], [40], [45], [49], [50], [57], [60]- [66], [68], [70], [71], [73], [76]- [78], [81]- [84], [88], [90], [91], [93]- [95] Springer 222 20 papers: [4], [6], [12], [21], [30], [36], [38], [39], [41]- [43], [47], [51], [53], [69], [74], [79], [85], [86], [92] Google Scholar 197 6 papers:...…”

Section: Sourcementioning

confidence: 99%

A Mapping Study about Data Lakes: An Improved Definition and Possible Architectures

Couto¹,

Borges²,

Ruiz³

et al. 2019

International Conferences on Software Engineering and Knowledge Engineering

View full text Add to dashboard Cite

In the past few years, data lakes emerged as a trending topic in big data technologies. Although literature presents different points of view related to its functionalities, it serves mainly to store a variety of data in a big data context. In this paper, we aim to identify and analyze data lake definitions and possible architectures. Our methodology was composed of a systematic literature mapping based on PRISMA, software engineering best practices to perform reviews, and Kappa method to assess results' quality. We performed the search in eight different electronic databases to achieve a wide variety of publishers in Computer Science. We first identified 662 papers matching our search criteria; after filtering, we selected 87 papers for review. We found that the term data lakes was first defined by James Dixon in 2010. We also found that the term is often related to raw data repositories. From the identified definitions, we propose a new one as a means to better state what data lakes refer to and improve how the community use them. Moreover, we foind that Hadoop and its ecosystem compose the most used toolset to create data lakes, revealing that this is the mainstream in architectures for data lakes as of today's available technologies.

show abstract

“…The continuous challenge for healthcare professionals is the retrieval and extraction of medical data and this is due to the scarcity of technologies integration as well as important data which are time consuming and even involves manual workflows [1]. The data inaccessibility renders its inconsequential even though the data exists.…”

Section: Introductionmentioning

confidence: 99%

Techniques for Data Extraction from Heterogeneous Sources with Data Security

Kumari¹,

mrunalini²

2019

IJRTE

View full text Add to dashboard Cite

Data Extraction is the process of mining or fetching relevant information from unstructured data or the heterogeneous sources of data. This paper aims at mining data from three different sources such as online website, flat files and database and the extracted data are even analyzed in terms of precisions, recall and accuracy. In the environment of heterogeneous sources of data, data extraction is one of the crucial issue and therefore considering the present scenario, we can observe that the heterogeneity is expanding widespread. So this paper focus on the different sources for the data extraction and provides a single framework to perform the required tasks. In this paper, healthcare data are considered in order to show the processing starting from data extraction using three different sources to dividing them in to two clusters based on the thresholds value which has been calculated using cosine similarity and finally calculations of parameters like precisions, recall and accuracy for analyzation purpose. Fetching data online is the task in which we cannot fetch simple string from any website. The backend of each page is html and hence this paper focus on extracting that html of the page while mining data from any web server. The webpage contains a lot of html tags and all of these cannot be removed because they are complex tags which cannot be removed by regular expressions. But still 60% filtered data can be attained as demonstrated in this paper as most of the waste html will be removed. While filtration of the data, we should also note that the content containing Google APIs cannot be removed. So filtered data will contain the contents and tags which does not contain Google APIs. In order to provide data security while extraction, the connection string is being used to avoid tampering of data. This paper also focuses on one of the arguable concepts present in the generation of big data which is Data Lake.In originality, the origin about the idea of Data Lake appears from the field of business. An architectural approach which is specially designed in order to store all the data which are potentially relevant in a repository located centrally is referred to as Data Lake. The data which are stored in the central based repository are fetched from the sources belonging to public as well as enterprises and these data are further used for the purpose of organization, discovery of hidden facts, understanding of new concepts, analyzation of stored information etc. Many challenges and concerns related to privacy are faced during the adoption of Data Lake as it is a new concept which brings revolutionization. This paper also highlights some of the issues imposed by Data Lake.

show abstract

Towards Information Profiling: Data Lake Content Metadata Management

Cited by 34 publications

References 18 publications

Data Lakes: Trends and Perspectives

Data Lakes: Trends and Perspectives

A Mapping Study about Data Lakes: An Improved Definition and Possible Architectures

Techniques for Data Extraction from Heterogeneous Sources with Data Security

Contact Info

Product

Resources

About