Validating Quality of Service for Reusable Software Via Model-Integrated Distributed Continuous Quality Assurance

Abstract

Abstract. Quality assurance (QA) tasks, such as testing, profiling, and performance evaluation, have historically been done in-house on developer-generated workloads and regression suites. Performanceintensive systems software, such as that found in the scientific computing grid and distributed real-time and embedded (DRE) domains, increasingly run on heterogeneous combinations of OS, compiler, and hardware platforms. Such software has stringent quality of service (QoS) requirements and often provides a variety of configuration options to optimize QoS. As a result, QA performed solely in-house is inadequate since it is hard to manage software variability, i.e., ensuring software quality on all supported target platforms across all desired configuration options. This paper describes how the Skoll project is addressing these issues by developing advanced QA processes and tools that leverage the extensive computing resources of user communities in a distributed, continuous manner to improve key software quality attributes.

A model-based distributed continuous quality assurance process to enhance the quality of service of evolving performance-intensive software systems

“…We are also integrating our DCQA technologies into the DRE software repository maintained by the ESCHER Institute (www.escherinstitute.org), which is a non-profit organization 4 established to preserve, maintain, and promote the technology transfer of governmentsponsored R&D tools and frameworks in the DRE computing domain.…”

“…In this context, model-based techniques involve visual representations for defining entities and their interactions in an application domain using domain-specific building blocks. These improvements are embodied in BGML [4], which is a model-based benchmarking toolsuite designed to evaluate the QoS of implementations of the CORBA Component Model (CCM), which is shown in Figure 2 and described in Sidebar 1. 2 BGML allows CCM users to:…”

“…However, both ForeSight and SoftArch tools lack DCQA environments to accurately capture QoS variations on a range of varied hardware, OS and compiler platforms. Rather than using a generic mathematical models to predict performance, the BGML tools use a feedback-driven approach [4], wherein the DCQA environment is used to empirically evaluate the QoS characteristics offline. This information can then be used to provide the modeler with accurate system information.…”

“…At the heart of the Skoll work presented in this paper is BGML [4], which is Model-based toolsuite 1 that applies generative model-based software techniques [5] to measure and optimize the QoS of reusable performance-intensive software configurations.…”

Main effects screening: a distributed continuous quality assurance process for monitoring performance degradation in evolving software systems

“…To make it easier to implement DCQA processes, we also integrated model-based software development tools with Skoll, which help developers capture the variant and invariant parts of DCQA processes and the software systems they are applied to within highlevel models that can be processed to automatically generate configuration files and other supporting code artifacts [7]. Some model-based tools integrated with Skoll include the Options Configuration Modeling language (OCML) [14] that models configuration options and inter-option constraints and the Benchmark Generation Modeling Language (BGML) [8] that composes benchmarking experiments to observe QoS behavior under different configurations and workloads.…”