Business Intelligence Platform for Big Data based on Scalable Distributed Two-Layer Data Store

Abstract

Abstract-Data mining is one of the main business intelligence technique. The volume of Big Data expands in such a way that the classical Business Intelligence methods need to be redefined. First, the organization of large data sets requires new distributed database architectures. Second, it is necessary to develop distributed data processing models that provide a high degree of scalability.In this paper we introduce fully scalable BI platform that is suitable for the most common data processing issues. The platform is based on our scalable distributed two-layer data store, which is competitive to existing NoSQL distributed data base systems. We show examples and experimental results showing advantages of our approach.

Highly Scalable Distributed Architecture for NoSQL Datastore Supporting Strong Consistency

“…We have already applied the SD2DS concept into content search datastore [61], [62], [63] and scalable IoT system [64]. The datastore is continuously integrated with many Big Data analysis mechanisms [65], [66]. But all mentioned above applications of SD2DS did not support the data consistency.…”

Content-aware data distribution over cluster nodes

“…The processing model for SD2DS [23] was developed for high dimensional Big Data sets. Such a form of data items is still a big challenge in many data exploration techniques [15].…”

“…SD2DS allows executing distributed processing jobs using mechanism similar to MapReduce paradigm [23]. It divides the jobs into block function, which is executed on each data items partition and aggregate function which gathers the intermediate and final results.…”