AI Benchmarking for Science: Efforts from the MLCommons Science Working Group

Thiyagalingam, Jeyan; Laszewski, Gregor von; Yin, Junqi; Emani, Murali; Papay, Juri; Barrett, Gregg; Łuszczek, Piotr; Tsaris, Aristeidis; Kirkpatrick, Christine L.; Wang, Feiyi; Gibbs, Tom; Vishwanath, Venkatram; Shankar, Mallikarjun; Fox, Geoffrey; Hey, Tony

doi:10.1007/978-3-031-23220-6_4

Cited by 7 publications

(7 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition to these studies, we have tested the performance of machine learning applications executed in the CoCos.ai environment. We used the SciML-bench tool [47], which provides the most versatile set of features compared to the other scientific machine learning benchmarking approaches [48]. The secure virtual machine we used for testing on a server, mentioned at the beginning of Section 4, had allocated 16 cores and 16 The encrypted data and code files are stored in a predetermined location in the file system.…”

Section: Performance Evaluationmentioning

confidence: 99%

Secure Multiparty Computation Using Secure Virtual Machines

Miladinović,

Milaković,

Vukasović

et al. 2024

Electronics

View full text Add to dashboard Cite

The development of new processor capabilities which enable hardware-based memory encryption, capable of isolating and encrypting application code and data in memory, have led to the rise of confidential computing techniques that protect data when processed on untrusted computing resources (e.g., cloud). Before confidential computing technologies, applications that needed data-in-use protection, like outsourced or secure multiparty computation, used purely cryptographic techniques, which had a large negative impact on the processing performance. Processing data in trusted enclaves protected by confidential computing technologies promises to protect data-in-use while possessing a negligible performance penalty. In this paper, we have analyzed the state-of-the-art in the field of confidential computing and present a Confidential Computing System for Artificial Intelligence (CoCoS.ai), a system for secure multiparty computation, which uses virtual machine-based trusted execution environments (in this case, AMD Secure Encrypted Virtualization (SEV)). The security of the proposed solution, as well as its performance, have been formally analyzed and measured. The paper reveals many gaps not reported previously that still exist in the current confidential computing solutions for the secure multiparty computation use case, especially in the processes of creating new secure virtual machines and their attestation, which are tailored for single-user use cases.

show abstract

Section: Performance Evaluationmentioning

confidence: 99%

Secure Multiparty Computation Using Secure Virtual Machines

Miladinović,

Milaković,

Vukasović

et al. 2024

Electronics

View full text Add to dashboard Cite

show abstract

“…The Science Working Group is concerned with improving the science beyond just a static benchmark (Thiyagalingam et al, 2022). The work reported here has been conducted as part of the MLCommons Science Working Group goals.…”

Section: Figurementioning

confidence: 99%

“…As today's academic institutions provide machine learning (ML), deep learning (DL), and high-performance computing (HPC) educational efforts, we attempt to identify if it is possible to leverage existing large-scale efforts from the MLCommons community (Thiyagalingam et al, 2022;MLCommons, 2023). We focus solely on challenges and opportunities cast by the MLCommons efforts to achieve this goal.…”

Section: Introductionmentioning

confidence: 99%

“…To provide a manageable entry point into answering this question, we summarize numerous insights that we obtained while improving and conducting earthquake benchmarks within the MLCommons TM Science Working Group, porting it to HPC big data systems. This includes insights into the usability and capability of HPC big data systems, the usage of the MLCommons benchmarking science applications (Thiyagalingam et al, 2022) and insights from improving the applicability in educational efforts.…”

Section: Introductionmentioning

confidence: 99%

“…Although MLCommons has many applications, we decided to use an application from the MLCommons Science Working Group as we most closely work as part of this group. It has four major benchmarks as documented in Thiyagalingam et al (2022) in von .…”

Section: Course Overview and Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Opportunities for enhancing MLCommons efforts while leveraging insights from educational MLCommons earthquake benchmarks efforts

von Laszewski,

Fleischer,

Knuuti

et al. 2023

Front. High Perform. Comput.

View full text Add to dashboard Cite

MLCommons is an effort to develop and improve the artificial intelligence (AI) ecosystem through benchmarks, public data sets, and research. It consists of members from start-ups, leading companies, academics, and non-profits from around the world. The goal is to make machine learning better for everyone. In order to increase participation by others, educational institutions provide valuable opportunities for engagement. In this article, we identify numerous insights obtained from different viewpoints as part of efforts to utilize high-performance computing (HPC) big data systems in existing education while developing and conducting science benchmarks for earthquake prediction. As this activity was conducted across multiple educational efforts, we project if and how it is possible to make such efforts available on a wider scale. This includes the integration of sophisticated benchmarks into courses and research activities at universities, exposing the students and researchers to topics that are otherwise typically not sufficiently covered in current course curricula as we witnessed from our practical experience across multiple organizations. As such, we have outlined the many lessons we learned throughout these efforts, culminating in the need for benchmark carpentry for scientists using advanced computational resources. The article also presents the analysis of an earthquake prediction code benchmark while focusing on the accuracy of the results and not only on the runtime; notedly, this benchmark was created as a result of our lessons learned. Energy traces were produced throughout these benchmarks, which are vital to analyzing the power expenditure within HPC environments. Additionally, one of the insights is that in the short time of the project with limited student availability, the activity was only possible by utilizing a benchmark runtime pipeline while developing and using software to generate jobs from the permutation of hyperparameters automatically. It integrates a templated job management framework for executing tasks and experiments based on hyperparameters while leveraging hybrid compute resources available at different institutions. The software is part of a collection called cloudmesh with its newly developed components, cloudmesh-ee (experiment executor) and cloudmesh-cc (compute coordinator).

show abstract

FAIR for AI: An interdisciplinary and international community building perspective

et al. 2023

Self Cite

View full text Add to dashboard Cite

A foundational set of findable, accessible, interoperable, and reusable (FAIR) principles were proposed in 2016 as prerequisites for proper data management and stewardship, with the goal of enabling the reusability of scholarly data. The principles were also meant to apply to other digital assets, at a high level, and over time, the FAIR guiding principles have been re-interpreted or extended to include the software, tools, algorithms, and workflows that produce data. FAIR principles are now being adapted in the context of AI models and datasets. Here, we present the perspectives, vision, and experiences of researchers from different countries, disciplines, and backgrounds who are leading the definition and adoption of FAIR principles in their communities of practice, and discuss outcomes that may result from pursuing and incentivizing FAIR AI research. The material for this report builds on the FAIR for AI Workshop held at Argonne National Laboratory on June 7, 2022.

show abstract

AI Benchmarking for Science: Efforts from the MLCommons Science Working Group

Cited by 7 publications

References 14 publications

Secure Multiparty Computation Using Secure Virtual Machines

Secure Multiparty Computation Using Secure Virtual Machines

Opportunities for enhancing MLCommons efforts while leveraging insights from educational MLCommons earthquake benchmarks efforts

FAIR for AI: An interdisciplinary and international community building perspective

Contact Info

Product

Resources

About