Improving component selection and monitoring with controlled experimentation and automated measurements

Abstract

a b s t r a c tContext: A number of approaches have been proposed for the general problem of software component evaluation and selection. Most approaches come from the field of Component-Based Software Development (CBSD), tackle the problem of Commercial-off-the-shelf component selection and use goal-oriented requirements modelling and multi-criteria decision making techniques. Evaluation of the suitability of components is carried out largely manually and partly relies on subjective judgement. However, in dynamic, distributed environments with high demands for transparent selection processes leading to trustworthy, auditable decisions, subjective judgements and vendor claims are not considered sufficient. Furthermore, continuous monitoring and re-evaluation of components after integration is sometimes needed. Objective: This paper describes how an evidence-based approach to component evaluation can improve repeatability and reproducibility of component selection under the following conditions: (1) Functional homogeneity of candidate components and (2) High number of components and selection problem instances. Method: Our evaluation and selection method and tool empirically evaluate candidate components in controlled experiments by applying automated measurements. By analysing the differences to system-development-oriented scenarios, the paper shows how the process of utility analysis can be tailored to fit the problem space, and describes a method geared towards automated evaluation in an empirical setting. We describe tool support and a framework for automated measurements. We further present a taxonomy of decision criteria for the described scenario and discuss the data collection means needed for each category of criteria. Results: To evaluate our approach, we discuss a series of case studies in the area of digital preservation. We analyse the criteria defined in these case studies, classify them according to the taxonomy, and discuss the quantitative coverage of automated measurements. Conclusion: The results of the analysis show that an automated measurement, evaluation and selection framework is necessary and feasible to ensure trusted and repeatable decisions.

Decision criteria in digital preservation: What to measure and how

“…The key result of such a preservation planning activity is a preservation plan . On the other hand, the selection problem can be seen as a domain‐specific instance of the general problem of component evaluation and selection, which has a long history in the areas of software engineering and information systems design (Becker & Rauber, 2010).…”

“…The core part of our method for creating such plans is a component evaluation and selection procedure that relies on a variation of utility analysis to support this multi‐objective decision‐making process. Our evidence‐based approach to component evaluation can improve repeatability and reproducibility of component selection under the following conditions: (a) functional homogeneity of candidate components and (b) a high number of components and selection problem instances (Becker & Rauber, 2010). Its implementation in preservation planning results in five phases, as shown in Figure 1.…”

“…The tool implements the planning method and creates well‐documented, machine‐readable preservation plans. The method has been applied to a wide range of digital preservation scenarios (Guttenbrunner, Becker, & Rauber, 2010; Kulovits et al, 2009; Zierau, Kejser, & Kulovits, 2010), and the tool Plato has experienced significant uptake in the digital preservation community.…”

Third Case Study for the Dynamic Multilevel Component Selection

“…The problems of identification and selection the right software components out of a range of choices to satisfy a set of requirements have received considerable attention in the field of component-based software engineering during the last two decades [9,10].…”

COTS products selection using fuzzy chance-constrained multiobjective programming

“…Kaur and Mann [16] presented an approach for defining evaluation criteria for reusable software components. Becker and Rauber [2] described how an evidence-based approach to component evaluation can improve the repeatability and reproducibility of component selection considering the functional homogeneity of candidate components and high number of component and selection problem instances. Tang et al [32] proposed an optimization model to solve the component selection problem by simultaneously considering reusability and compatibility.…”