HFSP: Bringing Size-Based Scheduling To Hadoop

Pastorelli, Mario; Carra, Damiano; Dell’Amico, Matteo; Michiardi, Pietro

doi:10.1109/tcc.2015.2396056

Cited by 31 publications

(14 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Hadoop Fair Sojourn Protocol Scheduler (HFSPS) dan Fair Scheduler merupakan job scheduler yang sudah ada dan berkembang pada sistem Hadoop, dan memiliki keunikan masing-masing dalam menangani karakteristik job yang diproses. HFSPS menerapkan metode scheduling berdasarkan ukuran penjadwalan dengan membangun sebuah knowledge (pengetahuan) mengenai informasi dari ukuran penjadwalan dari sebuah job selama eksekusi sehingga dapat mengoptimalkan completion time [4]. Fair Scheduler menerapkan sistem eksekusi map dan reduces secara bersamaan [5].…”

Section: Pendahuluanunclassified

“…Penelitian [4] dan [7] berkaitan erat dengan penelitian ini. Pada penelitian tersebut telah dilakukan pengujian algoritma Fair Scheduler dan HFSPS, namun terdapat perbedaan dari segi metode pengujian, pada penelitian tersebut kedua algoritma digunakan untuk mengolah data dump.…”

Section: Studi Terkaitunclassified

“…Gambar 3. Pembagian pool pada Fair Scheduler [11] Pada HFSPS semua resource dapat dipakai untuk mengerjakan suatu jobs [4].…”

Section: Algoritma Hfspsunclassified

See 2 more Smart Citations

Studi Perbandingan Performa Algoritma Penjadwalan untuk Real Time Data Twitter pada Hadoop

Prabowo¹,

Abdurohman²

2020

Komputika

View full text Add to dashboard Cite

Hadoop merupakan sebuah framework software yang bersifat open source dan berbasis java. Hadoop terdiri atas dua komponen utama, yaitu MapReduce dan Hadoop Distributed File System (HDFS). MapReduce terdiri atas Map dan Reduce yang digunakan untuk pemrosesan data, sementara HDFS adalah tempat atau direktori dimana data hadoop dapat disimpan. Dalam menjalankan job yang tidak jarang terdapat keragaman karakteristik eksekusinya, diperlukan job scheduler yang tepat. Terdapat banyak job scheduler yang dapat di pilih supaya sesuai dengan karakteristik job. Fair Scheduler menggunakan salah satu scheduler dimana prisnsipnya memastikan suatu jobs akan mendapatkan resource yang sama dengan jobs yang lain, dengan tujuan meningkatkan performa dari segi Average Completion Time. Hadoop Fair Sojourn Protocol Scheduler adalah sebuah algoritma scheduling dalam Hadoop yang dapat melakukan scheduling berdasarkan ukuran jobs yang diberikan. Penelitian ini bertujuan untuk melihat perbandingan performa kedua scheduler tersebut untuk karakteristik data twitter. Hasil pengujian menunjukan Hadoop Fair Sojourn Protocol Scheduler memiliki performansi lebih baik dibandingkan Fair Scheduler baik dari penanganan average completion time sebesar 9,31% dan job throughput sebesar 23,46%. Kemudian untuk Fair Scheduler unggul dalam parameter task fail rate sebesar 23,98%.

show abstract

Section: Pendahuluanunclassified

Section: Studi Terkaitunclassified

See 1 more Smart Citation

Studi Perbandingan Performa Algoritma Penjadwalan untuk Real Time Data Twitter pada Hadoop

Prabowo¹,

Abdurohman²

2020

Komputika

View full text Add to dashboard Cite

show abstract

“…The tests showed that the system running HFSP reduced the average response time significantly, and the algorithm showed good robustness performance. 25 One of the newest research results in size-based scheduling is LARS. The LARS is a network forwarding scheduling algorithm which is derived from LAS, 26 and it improves the LAS ''service hunger'' faults and makes forwarding resource allocation more reasonable.…”

Section: Research On Scheduling Algorithmmentioning

confidence: 99%

CBSBS: A classification-based size-based scheduling and its application in industrial network

2017

International Journal of Distributed Sensor Networks

View full text Add to dashboard Cite

Integration of industrial control network and normal office network makes Ethernet to be widely used in industrial control. The improvement of industrial Ethernet service quality has become a hot research topic because the traditional Ethernet does not optimize the performance for industrial use. The main ideas related to research results are improving the link layer protocols or the scheduling algorithms. The method of improving scheduling algorithm is more suitable for integration of industrial and office network because it does not need to modify the link layer protocol, and it can easily realize the compatibility between office network and industrial network. The size-based scheduling proves to be effective for improving the forwarding performance of small packets (such as industrial packets). One of the newest research results of size-based scheduling is least attained recent service. Least attained recent service optimizes the queuing algorithm by applying historical accumulation factor and decay cycle factor. Least attained recent service improves the forwarding performance of small data flows significantly in data link access. But the application of the two factors in least attained recent service also results in the problem of high memory occupancy rate and unstable performance. We analyzed the cause of least attained recent service problems and designed a classification-based size-based scheduling algorithm according to the quality-of-service requirements of office and industrial networks integration. The tests prove that classification-based size-based scheduling algorithm, which can keep basic characteristics of size-based scheduling algorithm, reduce the memory consumption, and provide more stable forwarding performance than traditional least attained recent service, is an effective algorithm that can provide quality of service in the industrial network and office network integration applications.

show abstract

“…gSched 's goal is to minimize task execution times by assigning tasks to appropriate computing nodes. Pastorelli et al also developed a Hadoop job scheduler HFSP , which aims at reducing overall system response times and guaranteeing fairness by building job size information. By dynamically tuning resources sharing among users and customizing scheduling policies for users, Yao et al built a Hadoop scheduler called LsPS leveraging the knowledge of various heterogeneous MapReduce jobs, thereby reducing average job response times.…”

Section: Related Workmentioning

confidence: 99%

Thermal‐aware task assignments in high performance computing clusters

Taneja

Kulkarni

et al. 2017

Concurrency and Computation

View full text Add to dashboard Cite

Summary Cluster‐level thermal management has gained much attention over the past decade due to rising cooling costs associated with data centers. In this research, we propose and implement a static scheduler called SSched and a dynamic one named DSched. These 2 algorithms schedule jobs based on CPU and disk temperatures of a Hadoop cluster's nodes. Our schedulers rely on a monitoring mechanism to keep track of CPU and disk utilization, maintaining CPU and disk temperatures below a threshold through thermal‐aware scheduling decisions. To facilitate the design of SSched and DSched, we classify jobs into the CPU‐intensive and disk‐intensive categories. When a job arrives, SSched retrieves the utilization stats from a profiled log, estimates the thermal behavior, and places the job on NodeManager to minimize thermal impacts. Unlike SSched, DSched improves thermal efficiency of Hadoop clusters through dynamic load balancing. DSched keeps track of the coolest and hottest nodes in the cluster; tasks are migrated from hot nodes into cool ones if any hot spot is detected. To evaluate the effectiveness of our schedulers, we keep track of average CPU and disk temperatures in a node, managing an optimal outlet temperature across a cluster. We demonstrate that compared with the traditional Hadoop scheduler, SSched and DSched achieve approximately 15% savings in terms of cooling cost with little performance overhead.

show abstract

HFSP: Bringing Size-Based Scheduling To Hadoop

Cited by 31 publications

References 38 publications

Studi Perbandingan Performa Algoritma Penjadwalan untuk Real Time Data Twitter pada Hadoop

Studi Perbandingan Performa Algoritma Penjadwalan untuk Real Time Data Twitter pada Hadoop

CBSBS: A classification-based size-based scheduling and its application in industrial network

Thermal‐aware task assignments in high performance computing clusters

Contact Info

Product

Resources

About