Towards a Method for the Evaluation of Reference Architectures: Experiences from a Case

Abstract

Abstract. Reference architectures provide major guidelines for the structure of a class of information systems. Because of their fundamental role, reference architectures have to be of high quality. Before accepting a reference architecture, it has to go through a rigorous evaluation process. A number of methods exist for the evaluation of software architectures. In this paper, we analyze the main differences between concrete software architectures and reference architectures. We discuss the effects of these differences on the evaluation of reference architectures and show that existing methods cannot be directly applied for the evaluation of reference architectures. For the evaluation of a reference architecture for e-contracting systems, we used the Architecture Tradeoff Analysis Method with a number of adaptations and extensions. We present our approach and share our experiences from this evaluation process. Based on the analysis and our experiences gained, we present our vision for a method for the evaluation of reference architectures.

A Process Framework for Designing Software Reference Architectures for Providing Tools as a Service

“…Different considerations for evaluating a reference architecture have been proposed [3,7,16]. Avgeriou et al [3] have proposed to evaluate a SRA using scenarios and prototyping.…”

“…The prototyping helps analyze the suitability of the implementation decisions such as platform choices and programming languages for the design decisions incorporated in a SRA. Angelov et al [7,16] have argued that straightforward adoption of architecture evaluation methods such as Architecture Tradeoff Analysis Methods (ATAM) [17] and Software Architecture Analysis Methods (SAAM) [18] is not feasible because: gathering all the stakeholders and generating scenarios for a SRA evaluation may not be possible, there can be a significantly large diversity of stakeholders and the levels of abstractions in the designed components can be quite high. Hence, it is important to identify the most relevant architecture requirements by involving domain experts or domain models and then preparing scenarios by involving a SRA's potential users [7,16].…”

“…One of the most comprehensive and detailed guidelines have been reported by Angelov et al [1,2,7,16]. Their work provides a classification technique of the reference architectures based upon the domain-specific maturity and how the reference architectures are designed.…”

“…addition of new components or only selecting a subset of a SRA), which requires a mechanism that can be used to analyze quality and completeness of the instantiated architectures. Although there have been attempts to provide a systematic approaches for reference architecture design, documentation and evaluation [2,3,7], to the best of our knowledge, there has been a little work done on providing a process framework for SRA design given the specific needs of SRA design and evaluation of the TSPACE. Our work reported in this paper aims to address the following research objectives:…”

A classification of software reference architectures: Analyzing their success and effectiveness

“…Reference architectures (RA) can be used as an inspiration in the design of concrete architectures or as a standardization tool that guarantees the interoperability between systems and between components of systems [21]. A reference architecture is used for the design of concrete architectures in multiple contexts, affecting different stakeholders in each context [4]. Nowadays, the increasing complexity of software, the need for efficient and effective software design processes and for high levels of system interoperability lead to an increasing role of reference architectures in the software design process.…”