A novel BIG DATA architecture in support of ADS-B data analytic

Boci, Erton; Thistlethwaite, Susan Brooks

doi:10.1109/icnsurv.2015.7121281

Cited by 6 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is an economical and online data store with powerful data process ability. After processing data in several steps, it will be stored into DW, where it can be analysed and reused for business needs [8,12]. The emergence of DL in companies that have enterprise data warehouses has led to an inevitable question for CIOs: how can DW and DL work together?…”

Section: Complementarity Overviewmentioning

confidence: 99%

“…To achieve the end goal of delivering accurate and consistent business insights repeatedly, the aid of DW and data-driven management will be needed [7]. With structural data algorithms delivered by the ETL(Extract, Transform, Load) data management is now facilitated with fast parsing, discovery, catalog, and preparation of data [12]. Combined with the speed and flexibility of a metadata-based approach to data management, this intelligent DL enables transforming raw Big Data for a variety of consumers without risk [4,5].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

From Single Architectural Design to a Reference Conceptual Meta-Model: An Intelligent Data Lake for New Data Insights

Kachaoui¹

2020

IJETER

View full text Add to dashboard Cite

Recently, a new concept has appeared in the world of new technologies dealing with Big Data. This concept is called Data Lake (DL) and it is becoming the most suitable way to administer and put up Big Data new generation systems. Today's Big Data storage systems suffer from many problems related to data structure, accessibility and data quality. In order to resolve these issues, DL systems offer referential without schema for unprocessed data with a common access interface. Consequently, storing data in a DL without any metadata governance will only generate a "data swamp". This paper describes a new architecture implementation for DL systems with optimal management of metadata. This process treats data from heterogeneous data sources and with a combination of Data Warehouse (DW) for better management of structured data. The proposal system discovers, extracts and classifies structural metadata from various data sources using ontologies. A new generation solution that makes available to all users sources of large amounts of information. This new approach offers companies an answer to data management problems and information availability.

show abstract

Section: Complementarity Overviewmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

From Single Architectural Design to a Reference Conceptual Meta-Model: An Intelligent Data Lake for New Data Insights

Kachaoui¹

2020

IJETER

View full text Add to dashboard Cite

show abstract

“…First data lake systems are described in [4], [10] and [11]. Google's Goods approach [10] is similar to our approach as it also focuses on the collection of metadata for the datasets which are available inside the organization.…”

Section: Data Lakesmentioning

confidence: 99%

“…However, both systems do not aim at extracting different metadata information automatically from various data sources, including semi-structured files. Boci et al [11] present a technical architecture for a data lake system based on the Hadoop eco-system, which is applied in the area of flight tracking.…”

Section: Data Lakesmentioning

confidence: 99%

Metadata Extraction and Management in Data LakesWith GEMMS

Quix

Hai

Vatov

2016

CSIMQ

View full text Add to dashboard Cite

In addition to volume and velocity, Big data is also characterized by its variety. Variety in structure and semantics requires new integration approaches which can resolve the integration challenges also for large volumes of data. Data lakes should reduce the upfront integration costs and provide a more flexible way for data integration and analysis, as source data is loaded in its original structure to the data lake repository. Some syntactic transformation might be applied to enable access to the data in one common repository; however, a deep semantic integration is done only after the initial loading of the data into the data lake. Thereby, data is easily made available and can be restructured, aggregated, and transformed as required by later applications. Metadata management is a crucial component in a data lake, as the source data needs to be described by metadata to capture its semantics. We developed a Generic and Extensible Metadata Management System for data lakes (called GEMMS) that aims at the automatic extraction of metadata from a wide variety of data sources. Furthermore, the metadata is managed in an extensible metamodel that distinguishes structural and semantical metadata. The use case applied for evaluation is from the life science domain where the data is often stored only in files which hinders data access and efficient querying. The GEMMS framework has been proven to be useful in this domain. Especially, the extensibility and flexibility of the framework are important, as data and metadata structures in scientific experiments cannot be defined a priori.

show abstract

References

2020

Data Lakes

View full text Add to dashboard Cite

A novel BIG DATA architecture in support of ADS-B data analytic

Cited by 6 publications

References 0 publications

From Single Architectural Design to a Reference Conceptual Meta-Model: An Intelligent Data Lake for New Data Insights

From Single Architectural Design to a Reference Conceptual Meta-Model: An Intelligent Data Lake for New Data Insights

Metadata Extraction and Management in Data LakesWith GEMMS

References

Contact Info

Product

Resources

About