Query Optimization: Fund Data Generation Applying NonClustered Indexing and MapReduced Data Cube Numerosity Reduction Method

Abstract

Nowadays, data aggregation is becoming the concern of the financial institution because this is very useful to answer the series of data over the years and at present. Big data is described as a large set of data at a different level that becomes a concern to which it is difficult to manage. It is a problem for most businesses to achieve better query performance while generating and analyzing large data by way of, data aggregation and such an exponential increase of data size makes a query to take a large amount of time and space. The data cube is a widely used tool to provide an efficient way to compute the data into a small data set. In this paper, the query optimization technique is to address the prolonged execution of the query by applying one of the data reduction strategies called numerosity reduction methods; slice and dice data cube operation is to reduce and efficiently aggregate yet maintains the accuracy of the data. The nonclustered index is to quickly retrieve the data without scanning the whole fact table and very useful for some repeated values. MapReduce based approach is for handling large scale data, in which it is of great help to enhance the data cube computation and achieve optimal time over large data set. The technique improves the response time by an average of 94%, and the availability of the memory space becomes 91%. With this, a timely increase in query performance could mean better use of data in operation and timely decision making for management.

Distributed Transcoding System Integrating HDFS and MapReduce

“…A set of basic services for collecting and providing external catalogs of information about files or data. The database node is an underlying entity responsible for managing the actual database storage and retrieval, processing data read and write requests, and periodically reporting data block information to the name node [6][7]. When executing a task, the name node is responsible for providing information such as the location and capacity of the data node where the data source of the task is located.…”