Real Time Data Warehouse Updates Through Extraction-Transformation-Loading Process Using Change Data Capture Method

Thulasiram, Sunaadh; Ramaiah, Nagaraja

doi:10.1007/978-3-030-37051-0_62

Cited by 8 publications

(5 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Of the 8 articles, 7 described various methods for incremental updates, particularly focusing on change data capture (CDC). All describe different categories of CDC, like timestamp-based, audit column-based, trigger-based, log-based, application programming interface-based, and data-based snapshots [16,18,[20][21][22][23][24]: (1) Lynch et al [16] and (2) Lenert et al [18] focused on triggers; (3) Wen [20] focused on timestamps and triggers; (4) Thulasiram and Ramaiah [21] and (5) Sun [22] focused on timestamps; (6) Hu and Dessloch [23] focused on timestamps, audit columns, logs, triggers, and snapshots; and (7) Wei Du and Zou [24] focused on snapshots and MapReduce.…”

Section: Literature Reviewmentioning

confidence: 99%

An Extract-Transform-Load Process Design for the Incremental Loading of German Real-World Data Based on FHIR and OMOP CDM: Algorithm Development and Validation

Henke¹,

Peng²,

Reinecke³

et al. 2023

JMIR Med Inform

View full text Add to dashboard Cite

Background In the Medical Informatics in Research and Care in University Medicine (MIRACUM) consortium, an IT-based clinical trial recruitment support system was developed based on the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM). Currently, OMOP CDM is populated with German Fast Healthcare Interoperability Resources (FHIR) using an Extract-Transform-Load (ETL) process, which was designed as a bulk load. However, the computational effort that comes with an everyday full load is not efficient for daily recruitment. Objective The aim of this study is to extend our existing ETL process with the option of incremental loading to efficiently support daily updated data. Methods Based on our existing bulk ETL process, we performed an analysis to determine the requirements of incremental loading. Furthermore, a literature review was conducted to identify adaptable approaches. Based on this, we implemented three methods to integrate incremental loading into our ETL process. Lastly, a test suite was defined to evaluate the incremental loading for data correctness and performance compared to bulk loading. Results The resulting ETL process supports bulk and incremental loading. Performance tests show that the incremental load took 87.5% less execution time than the bulk load (2.12 min compared to 17.07 min) related to changes of 1 day, while no data differences occurred in OMOP CDM. Conclusions Since incremental loading is more efficient than a daily bulk load and both loading options result in the same amount of data, we recommend using bulk load for an initial load and switching to incremental load for daily updates. The resulting incremental ETL logic can be applied internationally since it is not restricted to German FHIR profiles.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

An Extract-Transform-Load Process Design for the Incremental Loading of German Real-World Data Based on FHIR and OMOP CDM: Algorithm Development and Validation

Henke¹,

Peng²,

Reinecke³

et al. 2023

JMIR Med Inform

View full text Add to dashboard Cite

show abstract

“… Load: Load services need support before and after loadings, such as the regeneration of indexes and physical sections of the table. The specificity and structure of each goal when loading is also considered [50].  Refresh: The last step of ETL where updates over time are transferred from data sources to repositories [51].…”

Section: Data Mart Areamentioning

confidence: 99%

Implementing data-driven decision support system based on independent educational data mart

Hamoud

Hussein

Alhilfi

et al. 2021

IJECE

View full text Add to dashboard Cite

<span>Decision makers in the educational field always seek new technologies and tools, which provide solid, fast answers that can support decision-making process. They need a platform that utilize the students’ academic data and turn them into knowledge to make the right strategic decisions. In this paper, a roadmap for implementing a data driven decision support system (DSS) is presented based on an educational data mart. The independent data mart is implemented on the students’ degrees in 8 subjects in a private school (Al-Iskandaria Primary School in Basrah province, Iraq). The DSS implementation roadmap is started from pre-processing paper-based data source and ended with providing three categories of online analytical processing (OLAP) queries (multidimensional OLAP, desktop OLAP and web OLAP). Key performance indicator (KPI) is implemented as an essential part of educational DSS to measure school performance. The static evaluation method shows that the proposed DSS follows the privacy, security and performance aspects with no errors after inspecting the DSS knowledge base. The evaluation shows that the data driven DSS based on independent data mart with KPI, OLAP is one of the best platforms to support short-to-long term academic decisions.</span>

show abstract

“…An indicator is required for the interior intention of a user. Indicators are required systems for studying, analyzing, and presenting the enterprise data in a way that permits senior management to produce decisions [8]. The choice of a DW adopted by an organization is greatly influenced by the manner in which the organization operates as well as the type of decision support system (DSS) required by the organization [9].…”

Section: Introductionmentioning

confidence: 99%

OLAP Mining with Educational Data Mart to Predict Students’ Performance

Najm¹,

Dahr²,

Hamoud³

et al. 2022

IJCAI

View full text Add to dashboard Cite

Academic institutions always try to use a solid platform for supporting their short-to-long term decisions related to academic performance. These platforms utilize historical data and turn them into strategic decisions. The hidden patterns in the data need tools and approaches to be discovered. This paper aims to present a short roadmap for implementing educational data mart based on a data set from Alexandria Private Elementary School, located in the Basrah province of Iraq in the 2017-2018 academic year. The educational data mart is implemented, then the cube is constructed to perform OLAP operations and present OLAP reports. Next, OLAP mining is performed on the educational cube using nine algorithms, namely: decision tree with score method (entropy) and split method (complete)), decision tree with score method (entropy) and split method (complete)), decision tree with score method (entropy) and split method (both)), Logistic, Naïve Bayes, Neural Network, clustering with expectation maximization, clustering with K-means clustering, and association rules mining. According to a comparison of all algorithms, clustering with expectation-maximization proved the highest accuracy with 96.76% for predicting the students' performance and 96.12% for predicting students' grades amongst all other algorithms.

show abstract

Real Time Data Warehouse Updates Through Extraction-Transformation-Loading Process Using Change Data Capture Method

Cited by 8 publications

References 10 publications

An Extract-Transform-Load Process Design for the Incremental Loading of German Real-World Data Based on FHIR and OMOP CDM: Algorithm Development and Validation

An Extract-Transform-Load Process Design for the Incremental Loading of German Real-World Data Based on FHIR and OMOP CDM: Algorithm Development and Validation

Implementing data-driven decision support system based on independent educational data mart

OLAP Mining with Educational Data Mart to Predict Students’ Performance

Contact Info

Product

Resources

About