Efficient data streams processing in the real time data warehouse

Majeed, Fiaz; Mahmood, Muhammad Sohaib; Iqbal, Mujahid

doi:10.1109/iccsit.2010.5563859

Cited by 9 publications

(6 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This paper proposes real-time data warehouse architecture, which use real-time data storage method and capture business system changed data continuously, use real-time data extraction/integration method to load data into real-time data storage [3]. Use consistent dimension and consistent fact table of dimension modelling theory to make data structure model in real-time data stored and data structure model in historical data warehouse are the same.…”

Section: Real-time Data Warehouse Architecturementioning

confidence: 99%

“…Literature [1] proposes active data warehouse base on ODS and data warehouse concept, which can provide both strategic and tactical decision-making for enterprise. Other representative research results [2][3][4] on active data warehouse are also introduced. Literature [5][6][7][8] describes the challenge of data capture-efficient ETL and proposes their solutions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Research on Real Time Data Warehouse Architecture

Jia

Peng

2013

Communications in Computer and Information Science

View full text Add to dashboard Cite

Abstract. Real time data warehouse is the research hotspots of data warehouse. It expands the application scope of data warehouse and provides real-time decision-making system for business users. This paper describes the concepts of real time data warehouse and proposes a real time data warehouse architecture which is based on real-time cache storage. The architecture consists of three main components: real-time data capture and integration, business event management component and view materialization decision. There are two key technologies: real-time data extraction and materialized view decision-making. This paper describes existing solutions and their shortcomings, then proposes feasible technical solutions: real-time data extraction based on transaction log analysis and materialized view estimation model with time factor.

show abstract

Section: Real-time Data Warehouse Architecturementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Research on Real Time Data Warehouse Architecture

Jia

Peng

2013

Communications in Computer and Information Science

View full text Add to dashboard Cite

show abstract

“…Today, data warehouse (DWH), analytics and business intelligence (BI) stand for some of the most important information initiatives for companies [1], [2]. The continuous evolution of DWH implementation [3], the foundation for decision support systems [4]- [7], with new concepts such as data lakes [8], [9], big data [10]- [15], NoSQL technologies [16]- [19], and real-time streaming [20]- [23], is happening in an era characterized by persistently faster release cycles [24], [25] and constant product enhancements [26], [27]. DWH projects are mostly noted as large [28], time consuming [29], expensive [30]- [32], and change-sensitive [33] enterprise projects.…”

Section: Introductionmentioning

confidence: 99%

Project Management in Data Warehouse Implementations: A Literature Review

Bogojevic

2020

IEEE Access

View full text Add to dashboard Cite

“…The potential issue in data stream handling is memory management [2]. Grid technology for example [3] was used to cope over the storage requirement in the presence of data streams.…”

Section: Introductionmentioning

confidence: 99%

A framework for Internet data real-time processing: A machine-learning approach

Mauro

Sarno

2014

2014 International Carnahan Conference on Security Technology (ICCST)

View full text Add to dashboard Cite

Nowadays, the Internet Service Providers have to keep track of and in some cases to analyze for legal issues, a great amount of Internet data. Real-time big data processing and analysis introduce new challenges that must be addressed by system engineers. This is because: 1) traditional technologies exploiting databases are not designed to process a huge amount of data in real-time 2) classic machine learning algorithms implemented by widely adopted tools as Weka or R are not designed to perform "on the fly" analysis on streamed data. In this paper the authors propose an architecture that makes the real-time big data processing and analysis possible. The proposed architecture is based on two main components: a stream processing engine called Apache Storm and a framework called Yahoo SAMOA allowing to perform data analysis through distributed streaming machine learning algorithms. Our architecture is tested for Skype traffic recognition within network traffic generated by several Personal Computers in a streamed way. Experimental results have shown the effectiveness of proposed solution.

show abstract

Efficient data streams processing in the real time data warehouse

Cited by 9 publications

References 3 publications

Research on Real Time Data Warehouse Architecture

Research on Real Time Data Warehouse Architecture

Project Management in Data Warehouse Implementations: A Literature Review

A framework for Internet data real-time processing: A machine-learning approach

Contact Info

Product

Resources

About