An Introduction to Data Profiling

Abedjan, Ziawasch

doi:10.1007/978-3-319-96655-7_1

Cited by 4 publications

(1 citation statement)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the SLR about big data profiling, we disregard the papers about literature reviews [34][35][36] to compare only the papers that present solutions (frameworks, algorithms, or software) for big data profiling. We also remove the challenge lack of research from the Table, since all published papers help to address this challenge.…”

Section: Related Workmentioning

confidence: 99%

Untitled

2022

IJDKP

View full text Add to dashboard Cite

When we work in a data lake, data integration is not easy, mainly because the data is usually stored in raw format. Manually performing data integration is a time-consuming task that requires the supervision of a specialist, which can make mistakes or not be able to see the optimal point for data integration among two or more datasets. This paper presents a model to perform heterogeneous in-memory data integration in a Hadoop-based data lake based on a top-k set similarity approach. Our main contribution is the process of ingesting, storing, processing, integrating, and visualizing the data integration points. The algorithm for data integration is based on the Overlap coefficient since it presented better results when compared with the set similarity metrics Jaccard, Sørensen-Dice, and the Tversky index. We tested our model applying it on eight bioinformatics-domain datasets. Our model presents better results when compared to an analysis of a specialist, and we expect our model can be reused for other domains of datasets.

show abstract

Section: Related Workmentioning

confidence: 99%