Towards a benchmark for traceability

Abstract

Rigorously evaluating and comparing traceability link generation techniques is a challenging task. In fact, traceability is still expensive to implement and it is therefore difficult to find a complete case study that includes both a rich set of artifacts and traceability links among them. Consequently, researchers usually have to create their own case studies by taking a number of existing artifacts and creating traceability links for them. There are two major issues related to the creation of one's own example. First, creating a meaningful case study is time consuming. Second, the created case usually covers a limited set of artifacts and has a limited applicability (e.g., a case with traces from high-level requirements to low-level requirements cannot be used to evaluate traceability techniques that are meant to generate links from documentation to source code). We propose a benchmark for traceability that includes all artifacts that are typically produced during the development of a software system and with end-to-end traceability linking. The benchmark is based on an irrigation system that was elaborated in a book about software design. The main task considered by the benchmark is the generation of traceability links among different types of software artifacts. Such a traceability benchmark will help advance research in this field because it facilitates the evaluation and comparison of traceability techniques and makes the replication of experiments an easy task. As a proof of concept we used the benchmark to evaluate the precision and recall of a link generation technique based on the vector space model. Our results are comparable to those obtained by other researchers using the same technique. ABSTRACTRigorously evaluating and comparing traceability link generation techniques is a challenging task. In fact, traceability is still expensive to implement and it is therefore difficult to find a complete case study that includes both a rich set of artifacts and traceability links among them. Consequently, researchers usually have to create their own case studies by taking a number of existing artifacts and creating traceability links for them. There are two major issues related to the creation of one's own example. First, creating a meaningful case study is time consuming. Second, the created case usually covers a limited set of artifacts and has a limited applicability (e.g., a case with traces from high-level requirements to low-level requirements cannot be used to evaluate traceability techniques that are meant to generate links from documentation to source code). We propose a benchmark for traceability that includes all artifacts that are typically produced during the development of a software system and with end-to-end traceability linking. The benchmark is based on an irrigation system that was elaborated in a book about software design. The main task considered by the benchmark is the generation of traceability links among different types of software artifacts. Such a traceability benchmark w...

A Framework for Traceability Tool Comparison

“…Charrada et al [2] developed a benchmark for traceability which is based on the following three criteria, which are identical to those proposed by Dekhtyar et al [5]: precision, recall, and time. The evaluation of their benchmark was accomplished by analyzing one traceability mining tool and gauging its performance according to the proposed criteria.…”

“…RR2: Integration with other tools [14]. RR3: Performance [5,2]. RR4: Usability and data visualization [14].…”

Development of Robust Traceability Benchmarks

“…This is due to its high cost, complexity, time-consuming nature, and errorproneness [10,16,17]. This results in there being almost no publicly available projects containing very robust benchmarks [5]. Second, we can evaluate a new recovery technique with benchmarks developed by other researchers and/or applied by them to other software projects.…”

“…Third, we can establish our own benchmarks to meet our specific recovery technique evaluation needs. Due to the well-known difficulty of obtaining or using the above two types of benchmarks, researchers usually create their own benchmarks to conduct evaluations of their new recovery techniques [5,8,9,11]. Nevertheless, a major issue arises here: how do we actually go about building affordable and robust traceability benchmarks?…”

“…Nevertheless, a major issue arises here: how do we actually go about building affordable and robust traceability benchmarks? Most research to date has focused on the theory, basic principles, or the establishment of all-inclusive traceability benchmarks [5,9]. Unfortunately, there are no generally agreed or applied approaches or guidelines that have been proposed to assist researchers in manually building robust benchmarks for identifying and verifying links in a system.…”