FOSSES: Framework for open‐source software evaluation and selection

Abstract

A plethora of approaches exists for the evaluation and selection of open-source software (OSS) in the literature. However, these approaches are hardly ever used in practice for the following reasons: first, the lack of a situational-based procedure to define the evaluation criteria for OSS given its varied and dynamic nature; second, the inability of existing evaluation techniques, such as the analytic hierarchy process, to cope well with uncertainty factors, thus producing misleading results that affect the quality of decisions made; and third, a significant number of existing approaches require the prototyping of alternatives being considered in order to facilitate evaluation and decision-making. This study addresses the aforementioned challenges by evolving a process framework for evaluating and selecting OSS. The proposed framework is validated by applying it to a case study. In addition, expert opinion was elicited via questionnaires from 10 experts, and overall feedback suggests that 80% of them are willing to adopt the approach. KEYWORDS analytic hierarchy process, evidential reasoning, MCDM, open-source software, process framework, weighted scoring model 780The MCDM problem can be referred to as the process of choosing software among available alternatives based on a number of conflicting attributes. 8 MCDM is aimed at 9 (1) helping decision makers choose the best alternative, (2) sorting out the alternatives that seem good among the set of available alternatives, and (3) ranking the alternatives in decreasing order of their performance. A large number of well-known MCDM methods exist, such as the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS), data envelopment analysis, the analytic hierarchy process (AHP), the analytic network process, VIKOR, decision-making trial and evaluation laboratory, the Preference Ranking Organization Method for Enrichment Evaluations, and the Elimination and Choice Expressing Reality method, to mention a few. 10 The AHP is the most popular in terms of its application in software evaluation and selection. 7,11,12 However, the AHP has some known limitations, which can easily be surmounted by using the evidential reasoning (ER) approach in place of the AHP. 13 The ER approach is different from most conventional MCDM methods in that it employs a belief structure to represent an assessment as a distribution. For instance, the distributed assessment of a given software's quality may be {(Excellent, 70%), (Good, 30%), (Average, 0%), (Poor, 0%), (Worst, 0%)}, which means that the quality of the software is assessed to be Excellent with 70% of belief degree and Good with 30% of belief degree. Some of the advantages of using ER over the AHP 13 include the following:i. ability to handle very large multi-attribute decision-making problems compared to the AHP; ii. ability to assess newly added alternatives independently, whereas the AHP would have to repeat an assessment procedure to incorporate new alternatives; iii. ability to produce consistent ranking after n...

Threat intelligence platform for the energy sector

“…This feature could be particularly beneficial with the advent of new, alternative TI platforms, for instance, dedicated to other economic sectors. However, since the main characteristics considered in the assessments enabled by the framework are security, availability, and cost, to achieve a more comprehensive view, further software metrics and selection approaches would need to be investigated. For instance, the system selection situation can be represented as a multicriteria decision‐making problem, for which several solutions are available, including the analytic hierarchy process, the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) or data envelopment analysis.…”

“…For instance, the system selection situation can be represented as a multicriteria decision‐making problem, for which several solutions are available, including the analytic hierarchy process, the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) or data envelopment analysis. This observation was utilized in the development of the Framework for Open‐Source Software Evaluation and Selection (FOSSES) …”

Using composition trees to matching software requirements—An external agency's approach to support software acquisition

“…They listed requirements analysis and especially gaps analysis between system features and requirements as one of the key factors. However, they did not address how those high‐tech processes could be implemented in small companies that do not have the essential knowledge and resources.Similarly, in one other study, Adewumi et al 45 proposed an approach to choose a system from a set of available systems. They defined key factors such as functional suitability and usability for selecting a system and then defined metrics for each factor.…”

Entropy based Software Reliability Growth Modelling for Open Source Software Evolution

“…In a study [32] it was reported that a majority of OSS quality assurance models primarily concentrate on datadominant software evaluations. A study proposed a framework of process to address the challenges for the evaluation and selection of OSS [33]. The evolution of OSS is characterized by the number of issues that are reported or requests made for their enhancements.…”