Improved heuristic job scheduling method to enhance throughput for big data analytics

Abstract

Data-parallel computing platforms, such as Hadoop and Spark, are deployed in computing clusters for big data analytics. There is a general tendency that multiple users share the same computing cluster. The schedule of multiple jobs becomes a serious challenge. Over a long period in the past, the Shortest-Job-First (SJF) method has been considered as the optimal solution to minimize the average job completion time. However, the SJF method leads to a low system throughput in the case where a small number of short jobs consume a large amount of resources. This factor prolongs the average job completion time. We propose an improved heuristic job scheduling method, called the Densest-Job-Set-First (DJSF) method. The DJSF method schedules jobs by maximizing the number of completed jobs per unit time, aiming to decrease the average Job Completion Time (JCT) and improve the system throughput. We perform extensive simulations based on Google cluster data. Compared with the SJF method, the DJSF method decreases the average JCT by 23.19% and enhances the system throughput by 42.19%.Compared with Tetris, the job packing method improves the job completion efficiency by 55.4%, so that the computing platforms complete more jobs in a short time span.

Correlation between Collateral Compensation and Homocysteine Levels in Patients with Acute Cerebral Infarction after Intravenous Thrombolysis Based on Medical Big Data

“…In addition, some predictive work should be done in combination with clinical practice to give full play to the advantages of medical big data. The establishment of the medical big data life cycle is about the preparation of medical big data and the application of medical big data results [ 13 ].…”

Models in the Construction of Accounting Informatization Transformation Based on Digital Twin

“…is paper mainly analyzes the clustering interpolation method. e clustering algorithm mainly classifies the data according to the similarity of the data samples, and then divides the categories [25]. As shown in Figure 5, it is the clustering algorithm model.…”

A survey of intelligent load monitoring in IoT-enabled distributed smart grids

“…It is a big data distributed computing paradigm, in which resources are abstracted and virtualized, and these virtualized resources can be dynamically delivered to users for on-demand use through the network. This makes cloud computing the preferred platform for integrating resources and providing services Hu et al (2022).This is very different from traditional model, especially in terms of scalability. Therefore, cloud computing can be divided into three categories according to types of services it provides, and can be divided into three types, namely Iaas (Infrastructure as a Service), PaaS (Platform as a Service), SaaS (Software as a Service).…”