Programming with neural surrogates of programs

Abstract

Surrogates, models that mimic the behavior of programs, form the basis of a variety of development workflows. We study three surrogate-based design patterns, evaluating each in case studies on a large-scale CPU simulator.With surrogate compilation, programmers develop a surrogate that mimics the behavior of a program to deploy to end-users in place of the original program. Surrogate compilation accelerates the CPU simulator under study by 1.6×. With surrogate adaptation, programmers develop a surrogate of a program then retrain that surrogate on a different task. Surrogate adaptation decreases the simulator's error by up to 50%. With surrogate optimization, programmers develop a surrogate of a program, optimize input parameters of the surrogate, then plug the optimized input parameters back into the original program. Surrogate optimization finds simulation parameters that decrease the simulator's error by 5% compared to the error induced by expert-set parameters.In this paper we formalize this taxonomy of surrogatebased design patterns. We further describe the programming methodology common to all three design patterns. Our work builds a foundation for the emerging class of workflows based on programming with surrogates of programs.CCS Concepts: • Software and its engineering → Automatic programming; Software evolution; • Computing methodologies → Machine learning.

Turaco: Complexity-Guided Data Sampling for Training Neural Surrogates of Programs

“…For example, consider Table 3. This table shows the results of a hyperparameter search to choose the fastest-to-execute neural network that meets an error threshold of 10% 3 (this table is a replication of Table 4 in Appendix A of Renda et al [2021], with an added column of "Error -5%"). Renda et al chose the network with size 64, which has a 1.57× speedup; however, a decrease in error of 5% would result instead in choosing the network with size 32, which has a 2.01× speedup, a 28% improvement in application performance.…”

“…Beyond these examples, surrogate programming has developed into a diverse set of techniques and applications across many areas of computing and science. Renda et al [2021] organize the uses of neural network based surrogates of programs into three categories.…”

“…Surrogate programming -the act of replacing a program with a surrogate model of its behaviorhas been increasingly leveraged to solve a variety of software development challenges [Renda et al 2021]. For example, Esmaeilzadeh et al [2012] train neural networks to mimic existing numerical programs (a fast Fourier transform, a triangle intersection detection algorithm, a JPEG encoder, etc.)…”

“…The validation error and speedup of BERT models over a range of candidate embedding widths. This table is a replication of Table4in Appendix A ofRenda et al [2021], with an added column of "Error -5%".…”