Frequent patterns in ETL workflows: An empirical approach

Theodorou, Vasileios; Abelló, Alberto; Thiele, Maik; Lehner, Wolfgang

doi:10.1016/j.datak.2017.08.004

Cited by 20 publications

(8 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As already mentioned in the previous section, we mainly use secondary data in the context of AI. In AI, the process of operationalization is often replaced by the ETL process: 'Extract, Transform, Load' (Theodorou et al 2017). Relevant measurements are to be extracted from the data lake(s), then transformed and finally loaded into the (automated) analysis procedures.…”

Section: Statistics For the Assessment Of Data Qualitymentioning

confidence: 99%

Is there a role for statistics in artificial intelligence?

Friedrich

Antes

Behr

et al. 2021

Adv Data Anal Classif

View full text Add to dashboard Cite

The research on and application of artificial intelligence (AI) has triggered a comprehensive scientific, economic, social and political discussion. Here we argue that statistics, as an interdisciplinary scientific field, plays a substantial role both for the theoretical and practical understanding of AI and for its future development. Statistics might even be considered a core element of AI. With its specialist knowledge of data evaluation, starting with the precise formulation of the research question and passing through a study design stage on to analysis and interpretation of the results, statistics is a natural partner for other disciplines in teaching, research and practice. This paper aims at highlighting the relevance of statistical methodology in the context of AI development. In particular, we discuss contributions of statistics to the field of artificial intelligence concerning methodological development, planning and design of studies, assessment of data quality and data collection, differentiation of causality and associations and assessment of uncertainty in results. Moreover, the paper also discusses the equally necessary and meaningful extensions of curricula in schools and universities to integrate statistical aspects into AI teaching.

show abstract

Section: Statistics For the Assessment Of Data Qualitymentioning

confidence: 99%

Is there a role for statistics in artificial intelligence?

Friedrich

Antes

Behr

et al. 2021

Adv Data Anal Classif

View full text Add to dashboard Cite

show abstract

“…Based on literature review, the study of BI system in general has 2 main different aspects; technical approach and managerial approach. The technical approach focuses on techniques and tools to develop data-driven decision support system such as creating the set of tools to support data integration process or designing and developing the system with advanced technologies [21][22][23][24][25][26]. On the other hand, the managerial aspect concentrates on process to see how data is integrated and analyzed to generate the useful and valuable information to decision makers in order to guarantee the successful of system implementation [11][12][13][14][15][16][17][18][19][20][21].…”

Section: Business Intelligence (Bi)mentioning

confidence: 99%

“…The Quality Objective Matrix (QOX) is one of the quality measures of data and information that can be used to examine the performance of relevant data and information to ensure the effectiveness of ETL workflow [1,4,[7][8][24][25]. Previous studies identified four dimensions of QOX: accuracy, completenes, scalability, and efficiency [7][8][30][31][32][33] as presented in Table 1.…”

Section: Qualification Objective Matrix (Qox)mentioning

confidence: 99%

The Design and Implementation of Business Intelligence System for Lecturer Qualification Analysis based on ETL’s QOX Approach

Saebao¹,

Matayong²,

Trakulmaykee³

2021

Adv. sci. technol. eng. syst. j.

View full text Add to dashboard Cite

The study aims to design and evaluate the Business Intelligence (BI) system for lecturer qualification analysis. The Quality Objective Matrix (QOX) was the tool to evaluate the proposed ETL's qualities. This research focused on four key performance indicators of QOX: accuracy, completeness, scalability, and efficiency. In implementation, the developed BI system was evaluated by satisfaction questionnaire which was designed based on the Technology Acceptance Model (TAM). The results showed the proposed ETLs were good qualities in terms of accuracy, completeness, scalability, and efficiency. For satisfaction evaluation, the results also showed the very good level in overall. It was indicated that the users were satisfied to use the proposed BI dashboard for lecturers' qualifications analysis which was developed based on the Standard Criteria for Graduate Studies Programs (SCGSP). They perceived that the Lecturer Qualification Analysis System (LQAS) was easy to use and can help them to reduce time in the data preparing process. Additionally, the results highlighted that the information quality of BI dashboard also was sufficient and accurate for lecturer qualification analysis.

show abstract

“…In ETL, there are also a variety of approaches that seek to reduce the amount of manual labour required. For example, this includes the provision of language features [4,32] or patterns [31,33] that support recurring data preparation behaviours, techniques for managing evolution of ETL programs [8], and development of ETL processes that abstract over more concrete implementation details [3,22]. However, although such work focuses on raising the abstraction levels at which data engineers engage in data preparation tasks, we are not aware of prior results that use feedback on data products to make changes across complete data preparation processes.…”

Section: Related Workmentioning

confidence: 99%

Feedback driven improvement of data preparation pipelines

Κωνσταντίνου

Paton

2020

Information Systems

View full text Add to dashboard Cite

Data preparation, whether for populating enterprise data warehouses or as a precursor to more exploratory analyses, is recognised as being laborious, and as a result is a barrier to costeffective data analysis. Several steps that recur within data preparation pipelines are amenable to automation, but it seems important that automated decisions can be refined in the light of user feedback on data products. There has been significant work on how individual data preparation steps can be refined in the light of feedback. This paper goes further, by proposing an approach in which feedback on the correctness of values in a data product can be used to revise the results of diverse data preparation components. Furthermore, the approach takes into account all the available feedback in determining which actions should be applied to refine the data preparation process. The approach has been implemented to refine the results of of matching, mapping and data repair components in the VADA data preparation system, and is evaluated using deep web and open government data sets from the real estate domain. The experiments have shown how the approach enables feedback to be assimilated effectively for use with individual data preparation components, and furthermore that synergies result from applying the feedback to several data preparation components.

show abstract

Frequent patterns in ETL workflows: An empirical approach

Cited by 20 publications

References 21 publications

Is there a role for statistics in artificial intelligence?

Is there a role for statistics in artificial intelligence?

The Design and Implementation of Business Intelligence System for Lecturer Qualification Analysis based on ETL’s QOX Approach

Feedback driven improvement of data preparation pipelines

Contact Info

Product

Resources

About