Information intelligence

Managing Of Meta Data in Data Lake for Data Profiling

“…Data profiling and annotation are currently a hot issue for research and are crucial for understanding DL architectures [3], [12], and [13]. Several methods and strategies have been studied in the past, although the majority of them are concentrated on relational content metadata [7,10], free-text metadata [13], or data provenance metadata [1], [14]. The majority of recent research initiatives point to the necessity of a regulated metadata management approach for merging several BD kinds [8, [13], [15].…”

“…The content metadata is then extracted after the dataset has been digested. Starting with DIG01, which uses straightforward statistical methods and profiling algorithms comparable to [7], the data profile and schema profile are created. This process takes O(n) time.…”

“…Information ProfilingAnalyzing raw data to find structural patterns and statistical distributions is a traditional step in schema extraction and data profiling [7]. Higher-level profiling is currently required, which entails analysing data on the approximate schema and examples of relationships between several datasets rather than just single datasets [8].…”

“…A schema profile provides information on the attributes of a dataset, including their number, data type, and names [10]. The data profiles under consideration describe the single-attribute statistics of values, or the values in the dataset [7]. The third category of content metadata, information profiles, takes advantage of data profiles and data schemas' patterns [3].…”