Building Data Warehouses Using Automation

Abstract

Software development is a complex endeavor. While significant benefits can be achieved, the process is often laborious, time consuming and error prone requiring multiple iterations in order to achieve the desired result. Issues arise for numerous reasons – coding defects, unclear requirements, migration challenges, lack of convention, and inadequate testing to name a few. When convention and automation are introduced into the software development lifecycle there are significantly fewer opportunities for failure. Automation also allows for shorter development windows. Generally there are fewer errors throughout testing, with the bulk of those being found in unit and functional testing, far before the users get involved in systems acceptance testing. A data warehouse consists of multiple subject areas in which many tasks are common and should be automated for the sake of efficiency and enforcing convention. This article discusses a set of tools that can be used to automate writing data warehouse objects. The article also provides statistics of time saved using automation.

SQL Scorecard for Improved Stability and Performance of Data Warehouses

“…This scorecard process is also destined to support data warehouse governance board in reviewing and evaluating efficiency of ad-hoc queries, database views, stored procedures and macros. In a data warehouse it is imperative to use automation tools as much as possible as it saves time for the tasks performed repetitively (Akahane et al, 2015;Rahman & Rutz, 2015;Mizuno & Liang, 2015;Sah & Vaidya, 2014).…”

Big Data Business Intelligence in Bank Risk Analysis

“…Large volume of historical data also causes performance degradation. Hadoop infrastructure is scalable, fault tolerant, and cost efficient (Rahman & Rutz, 2015). 2.…”

“…To process real-time streaming data, make business intelligence decisions, use fraud detection, or do security checks, data mining tools and models could be applied. To store huge volumes of historical data -that is structured data -HDFS could be leveraged as a repository which is less costly compared to conventional data storage (e.g., databases) systems (Rahman & Rutz, 2015).…”

“…The core of data analytics in banking data is to provide business intelligence in terms of fraud detection and quantification of risks banking transactions. Big data technologies could be used to store historical and archive data that used to be stored in conventional databases and data warehouses (Rahman & Rutz, 2015). Given Hadoop storage is much cheaper than that of storage of relational database systems.…”

Virtual Gas Analyzer in the Maritime Sector

“…In the sector literature, there are several studies about intelligent applications in a logistics platform (Custodio and Machado, 2020). Some authors focus on optimization procedures, as resource allocation or goods picking, to improve the management performance of the logistics platform (Rahman and Rutz, 2015;Petrucci et al, 2010;Manzini et al, 2007). Researches are related to intelligent application on the freight and vehicles management (Oleari et al, 2014;Vivaldini et al, 2010;Amato et al, 2005).…”