Multi-objectivizing software configuration tuning

Abstract

Automatically tuning software configuration for optimizing a single performance attribute (e.g., minimizing latency) is not trivial, due to the nature of the configuration systems (e.g., complex landscape and expensive measurement). To deal with the problem, existing work has been focusing on developing various effective optimizers. However, a prominent issue that all these optimizers need to take care of is how to avoid the search being trapped in local optima -a hard nut to crack for software configuration tuning due to its rugged and sparse landscape, and neighboring configurations tending to behave very differently. Overcoming such in an expensive measurement setting is even more challenging. In this paper, we take a different perspective to tackle this issue. Instead of focusing on improving the optimizer, we work on the level of optimization model. We do this by proposing a meta multi-objectivization model (MMO) that considers an auxiliary performance objective (e.g., throughput in addition to latency). What makes this model unique is that we do not optimize the auxiliary performance objective, but rather use it to make similarly-performing while different configurations less comparable (i.e. Pareto nondominated to each other), thus preventing the search from being trapped in local optima.Experiments on eight real-world software systems/environments with diverse performance attributes reveal that our MMO model is statistically more effective than state-of-the-art single-objective counterparts in overcoming local optima (up to 42% gain), while using as low as 24% of their measurements to achieve the same (or better) performance result. CCS CONCEPTS• Software and its engineering → Software performance; Software configuration management and version control systems.

The Weights can be Harmful: Pareto Search versus Weighted Search in Multi-Objective Search-Based Software Engineering

“…3.1.1 Problem. Many software systems are highly configurable and adaptable (at design time or runtime) [11,12,14,16,18,20,22,77,85], which raises a search problem and opportunity for one to tune their configuration options for multiple performance concerns, such as latency, throughput, and memory consumption. According to the literature, SCT has been widely studied in SBSE, e.g., [8, 12, 20-22, 48, 58, 59, 77, 85].…”

“…Albeit work exists on performance modeling for configurable and adaptable systems [13, 15? ], there are no well-defined objective functions for 𝑓 1 , 𝑓 2 , and 𝑓 3 in SCT; thereby to guarantee accuracy, every evaluation needs to be done by profiling the software under a benchmark [8,12,21,22,48,77].…”

Artifact for paper "Lifelong Dynamic Optimization for Self-Adaptive Systems: Fact or Fiction?" published at IEEE SANER 2022

“…2) More importantly, since f t (x) and f a (x) often come with rather different scales for SASs [9], [11], normalization is required. Chen and Li [27] has shown that under such a case, the w can become a highly-sensitive parameter to tune before using MMO in the search. This is clearly difficult for SASs where pre-tuning is infeasible.…”

“…Refining MMO by finding an appropriate replacement of the auxiliary objective f a (x) is non-trivial, as we need to ensure that (1) f a (x) is an easy-to-measure objective; and (2) it should be within the same scale as that of f t (x) (hence the w can be removed by setting w = 1) while (3) similarly performing adaptation plans on f t (x) can have very different values on f a (x), which is an important requirement for multiobjectivization with MMO as shown by Chen and Li [27].…”

“…Further, in RQ3, we examine the trajectory of planning, which also indicates what would happen if a smaller k is used. The other parameter settings follow what has been pragmatically used from the literature [9], [40], [14], [27]. However, we acknowledge that examining alternative parameters (and larger k) can be an interesting topic and we leave this for future work.…”