Karasu: A Collaborative Approach to Efficient Cluster Configuration for Big Data Analytics

Scheinert, Dominik; Wiesner, Philipp; Wittkopp, Thorsten; Thamsen, Lauritz; Will, Jonathan; Kao, Odej

doi:10.1109/ipccc59175.2023.10253884

2023 IEEE International Performance, Computing, and Communications Conference (IPCCC) 2023

DOI: 10.1109/ipccc59175.2023.10253884

|View full text |Cite

Karasu: A Collaborative Approach to Efficient Cluster Configuration for Big Data Analytics

Dominik Scheinert,

Philipp Wiesner,

Thorsten Wittkopp

et al.

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2023

2024

Publication Types

Select...

Other2

Relationship

Self Cite2

Independent0

Authors

Journals

Cited by 2 publications

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Towards a Peer-to-Peer Data Distribution Layer for Efficient and Collaborative Resource Optimization of Distributed Dataflow Applications

Scheinert,

Becker,

Will

et al. 2023

2023 IEEE International Conference on Big Data (BigData)

Self Cite

View full text Add to dashboard Cite

Performance modeling can help to improve the resource efficiency of clusters and distributed dataflow applications, yet the available modeling data is often limited. Collaborative approaches to performance modeling, characterized by the sharing of performance data or models, have been shown to improve resource efficiency, but there has been little focus on actual data sharing strategies and implementation in production environments. This missing building block holds back the realization of proposed collaborative solutions.In this paper, we envision, design, and evaluate a peerto-peer performance data sharing approach for collaborative performance modeling of distributed dataflow applications. Our proposed data distribution layer enables access to performance data in a decentralized manner, thereby facilitating collaborative modeling approaches and allowing for improved prediction capabilities and hence increased resource efficiency. In our evaluation, we assess our approach with regard to deployment, data replication, and data validation, through experiments with a prototype implementation and simulation, demonstrating feasibility and allowing discussion of potential limitations and next steps.

show abstract

Towards a Peer-to-Peer Data Distribution Layer for Efficient and Collaborative Resource Optimization of Distributed Dataflow Applications

Scheinert,

Becker,

Will

et al. 2023

2023 IEEE International Conference on Big Data (BigData)

Self Cite

View full text Add to dashboard Cite

show abstract

Privacy-Preserving Sharing of Data Analytics Runtime Metrics for Performance Modeling

Will,

Scheinert,

Zunzer

et al. 2024

Companion of the 15th ACM/SPEC International Conference on Performance Engineering

Self Cite

View full text Add to dashboard Cite

Performance modeling for large-scale data analytics workloads can improve the efficiency of cluster resource allocations and job scheduling. However, the performance of these workloads is influenced by numerous factors, such as job inputs and the assigned cluster resources. As a result, performance models require significant amounts of training data. This data can be obtained by exchanging runtime metrics between collaborating organizations. Yet, not all organizations may be inclined to publicly disclose such metadata.We present a privacy-preserving approach for sharing runtime metrics based on differential privacy and data synthesis. Our evaluation on performance data from 736 Spark job executions indicates that fully anonymized training data largely maintains performance prediction accuracy, particularly when there is minimal original data available. With 30 or fewer available original data samples, the use of synthetic training data resulted only in a one percent reduction in performance model accuracy on average. CCS CONCEPTS• Computing methodologies → Distributed computing methodologies; • Information systems; • Security and privacy;

show abstract

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

hi@scite.ai

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

Karasu: A Collaborative Approach to Efficient Cluster Configuration for Big Data Analytics

Cited by 2 publications

References 27 publications

Towards a Peer-to-Peer Data Distribution Layer for Efficient and Collaborative Resource Optimization of Distributed Dataflow Applications

Towards a Peer-to-Peer Data Distribution Layer for Efficient and Collaborative Resource Optimization of Distributed Dataflow Applications

Privacy-Preserving Sharing of Data Analytics Runtime Metrics for Performance Modeling

Contact Info

Product

Resources

About