More on Pipelined Dynamic Scheduling of Big Data Streams

Abstract

An important as well as challenging task in modern applications is the management and processing with very short delays of large data volumes. It is quite often, that the capabilities of individual machines are exceeded when trying to manage such large data volumes. In this regard, it is important to develop efficient task scheduling algorithms, which reduce the stream processing costs. What makes the situation more difficult is the fact that the applications as well as the processing systems are prone to changes during runtime: processing nodes may be down, temporarily or permanently, more resources may be needed by an application, and so on. Therefore, it is necessary to develop dynamic schedulers, which can effectively deal with these changes during runtime. In this work, we provide a fast and fair task migration policy while maintaining load balancing and low latency times. The experimental results have shown that our scheme offers better load balancing and reduces the overall latency compared to the state of the art strategies, due to the stepwise communication and the pipeline based processing it employs.

Dynamic Task Scheduling Scheme for Processing Real-Time Stream Data in Storm Environments

“…In this case, response delay and task loss occur during real-time stream processing of data, which causes failure in task completion within the deadline. For resolving these issues, various scheduling schemes have been examined in which the loads on the worker nodes in a real-time stream environment are considered [10][11][12][13][14][15][16][17][18][19][20][21].…”

“…A study [20] proposed a dynamic scheduler that can redistribute the migrated tasks in a fair and fast way based on their previous work [21]. They perform the dynamic scheduling by estimating the load of the nodes to handle task migrations when the system parameters change such as the number of tasks and configuration of executors or nodes.…”

“…Studies [12,[14][15][16][17][18][19][20][21] proposed task scheduling schemes based on various resources of worker nodes. T-Storm performs scheduling only based on traffic information, and thus, cannot solve the problem of load imbalance in worker nodes; it also entails delays in certain tasks if dynamic scheduling is not performed, as in [12].…”

“…Various studies have been conducted for resolving this problem [10][11][12][13][14][15][16][17][18][19][20][21]. In some study, simply calculating the traffic cannot solve the problem of load imbalance on worker nodes.…”

The Impact of Information Systems Implementation in the Greek Manufacturing Enterprises

“…For example, big data is a very prominent IT technology for a variety of fields, such as computing, economics, business, agriculture, accounting, etc. [4][5][6][7][8][9][10][11]. The global competition that already exists among manufacturing firms will continue to be intensive into the future.…”