DOD-ETL: distributed on-demand ETL for near real-time business intelligence

Machado, Gustavo V.; Cunha, Ítalo; Pereira, Adriano C. M.; Oliveira, Leonardo B.

doi:10.1186/s13174-019-0121-z

Cited by 25 publications

(11 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Generating materialized and virtualized RDF is constrained with respect to execution time [30], computing resources [18,30], bandwidth [31], performance [32], and query execution [31] because producers do not know which generation approach is the most suitable given its own resources and the RDF use. Since the producer needs to provide most resources for generating RDF from heterogeneous data sources and answering consumers' queries, guidelines for selecting the right approach are needed to minimize its effort.…”

Section: Problem Statement and Contributionsmentioning

confidence: 99%

Balancing RDF Generation from Heterogeneous Data Sources

Assche

2022

The Semantic Web: ESWC 2022 Satellite Events

View full text Add to dashboard Cite

Knowledge graphs in RDF are often generated from heterogeneous data sources to power services. However, knowledge graph generation is an unbalanced effort for producers compared to consumers of a knowledge graph. In this paper, I present my research about (i) investigating current RDF knowledge graph production and consumption approaches, and (ii) how to involve the consumer into a hybrid RDF generation approach to reduce the necessary resources for generating RDF for producers & consumers. I discuss the shortcomings of existing approaches for RDF generation from heterogeneous data sources (i.e., materialization and virtualization) and how I will address these: a Systematic Literature Review; an analysis and a set of guidelines for producers to select the right approach for an use case; and a combined hybrid approach to balance the producer's and consumer's effort in RDF generation. I already performed a Systematic Literature Review to get an overview of the existing approaches for RDF production from heterogeneous data sources. These results will be used to establish a set of producer guidelines, a benchmark to compare the current materialization and virtualization approaches, and evaluate the proposed hybrid approach. Thanks to my research, knowledge graph production and consumption will be more balanced and accessible to smaller companies and individuals. This way, they can focus on providing better services on top of a knowledge graph instead of being limited by the lack of computing resources to harvest enormous amounts of data from the Web and integrate it into a knowledge graph.

show abstract

Section: Problem Statement and Contributionsmentioning

confidence: 99%

Balancing RDF Generation from Heterogeneous Data Sources

Assche

2022

The Semantic Web: ESWC 2022 Satellite Events

View full text Add to dashboard Cite

show abstract

“…1) Requirements/challenges for real-time stream processing for real-time DWH Following requirements and challenges for implementation of real-time stream processing for real-time DWH were identified after exploring various studies [4], [14]- [18], [20]- [31], [34]- [36], [39]- [41], [43]- [50], [52], [59], [63]- [70], [78]:…”

Section: Assessment Of Rq3: Which Approaches/tools Have Been Repormentioning

confidence: 99%

“…• stream-disk join for structured data • stream-stream join • sql query decomposition • multi-join query processing in cloud DWHs • survey of design approaches from distributed systems, social media and real-time ETL tools • architecture/framework for supporting distributed streaming ETL and data integration in real-time DWH • development of stream ETL engine • distributed on demand ETL framework • code-based real-time ETL tools Other emerging concept related to near real-time ETL has been addressed recently in [78]. They identified and proposed a solution for distributed on demand ETL, and developed a stream processing framework based on Kafka, Beam and Spark Streaming.…”

Section: Relationalmentioning

confidence: 99%

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Mehmood

Anees

2020

IEEE Access

View full text Add to dashboard Cite

Contribution: Recently, real-time data warehousing (DWH) and big data streaming have become ubiquitous due to the fact that a number of business organizations are gearing up to gain competitive advantage. The capability of organizing big data in efficient manner to reach a business decision empowers data warehousing in terms of real-time stream processing. A systematic literature review for real-time stream processing systems is presented in this paper which rigorously look at the recent developments and challenges of real-time stream processing systems and can serve as a guide for the implementation of real-time stream processing framework for all shapes of data streams. Background: Published surveys and reviews either cover papers focusing on stream analysis in applications other than real-time DWH or focusing on extraction, transformation, loading (ETL) challenges for traditional DWH. This systematic review attempts to answer four specific research questions. Research Questions: 1)Which are the relevant publication channels for real-time stream processing research? 2) Which challenges have been faced during implementation of real-time stream processing? 3) Which approaches/tools have been reported to address challenges introduced at ETL stage while processing real-time stream for real-time DWH? 4) What evidence have been reported while addressing different challenges for processing real-time stream? Methodology: A systematic literature was conducted to compile studies related to publication channels targeting real-time stream processing/joins challenges and developments. Following a formal protocol, semi-automatic and manual searches were performed for work from 2011 to 2020 excluding research in traditional data warehousing. Of 679,547 papers selected for data extraction, 74 were retained after quality assessment. Findings: This systematic literature highlights implementation challenges along with developed approaches for real-time DWH and big data stream processing systems and provides their comparisons. This study found that there exists various algorithms for implementing real-time join processing at ETL stage for structured data whereas less work for un-structured data is found in this subject matter. INDEX TERMS Real-time stream processing, big data streaming, structured/un-structured data, ETL, systematic literature review Publication source Channel References No. %age IEEE Transactions on Knowledge and Data Engineering

show abstract

“…To improve the performance of data warehousing a conceptual model is presented, which defines different dimensionality and stereotypes. Similarly, a Data on Demand ETL (DOD-ETL) model is proposed in Machado et al Dec. ( 2019 ), which combines on demand data stream and pipelines in distributed and parallel memory caches to support effective portioning of data. An event-driven architecture is presented in Rieke et al ( 2018 ), to manage spatial data obtained from different capturing devices to maintain the geospatial information according to the real-time data (Bouali et al 2019 ).…”

Section: Introductionmentioning

confidence: 99%

Dynamic multi-variant relational scheme-based intelligent ETL framework for healthcare management

Manickam

Minu

2022

Soft Comput

View full text Add to dashboard Cite

The growth of information technology has opened the gate for the organizations to maintain their data in various forms and at various volumes. This increases the volume and dimension of data being maintained. However, they store their data in their data servers or in cloud environment. Such data have been used to generate various intelligence to support various problems. To support such analysis process, different data have been used and the big data comes to play in this part. Optimizing techniques to help improve the process of ETL could greatly help in real-time analysis of data. ETL optimization could be achieved through several factors simplest being increasing the frequency of the process. Other ways to achieve optimization are through the use of various architectures, programming models, intelligence in transformation and security. To improve the performance of ETL, an efficient dynamic multi-variant relational intelligent ETL framework has been presented in this article. The distributed approach maintains various ontology’s and data dictionaries which have been dynamically updated by different threads of ETL process. Initially, the process is start by applying the extraction process which extracts the data from different sources and finds set of dimensions and their characteristics. Such data extracted have been verified over the data dictionary. Further, the relational score has been measured for each data source with the existing one. Similarly, the method computes the value of multi-variant relational similarity (MVRS) for the data obtained from a single source. This will be performed by different threads of ETL process. According to the value of MVRS, the method performs map reduce and merging of data. According to the value of MVRS the method selects the data node and merges the data to store in the data warehouse. The threads of ETL are capable of reading the changes in data dictionaries and ontology’s to iterate the process of transformation and loading. The method improves the performance of ETL with least time complexity and higher performance.

show abstract

DOD-ETL: distributed on-demand ETL for near real-time business intelligence

Cited by 25 publications

References 21 publications

Balancing RDF Generation from Heterogeneous Data Sources

Balancing RDF Generation from Heterogeneous Data Sources

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Dynamic multi-variant relational scheme-based intelligent ETL framework for healthcare management

Contact Info

Product

Resources

About