Make the Most of Your Time: How Should the Analyst Work with Automated Traceability Tools?

Dekhtyar, Alex; Hayes, Jane Huffman; Larsen, Jody

doi:10.1109/promise.2007.8

Cited by 23 publications

(21 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[9] showed improvement in the performance of a traceability environment after several rounds of an expert reviewing results automatically inferred in a prior round. Based on this paper, we would speculate that policies (a) and (b) would significantly decrease the time required by an expert to achieve performance plateaus.…”

Section: Discussionmentioning

confidence: 99%

Implications of ceiling effects in defect predictors

Menzies

Turhan

Bener

et al. 2008

Proceedings of the 4th International Workshop on Predictor Models in Software Engineering

149

113

View full text Add to dashboard Cite

Context: There are many methods that input static code features and output a predictor for faulty code modules. These data mining methods have hit a "performance ceiling"; i.e., some inherent upper bound on the amount of information offered by, say, static code features when identifying modules which contain faults.Objective: We seek an explanation for this ceiling effect. Perhaps static code features have "limited information content"; i.e. their information can be quickly and completely discovered by even simple learners.Method: An initial literature review documents the ceiling effect in other work. Next, using three sub-sampling techniques (under-, over-, and micro-sampling), we look for the lower useful bound on the number of training instances.Results: Using micro-sampling, we find that as few as 50 instances yield as much information as larger training sets.Conclusions: We have found much evidence for the limited information hypothesis. Further progress in learning defect predictors may not come from better algorithms. Rather, we need to be improving the information content of the training data, perhaps with case-based reasoning methods.

show abstract

Section: Discussionmentioning

confidence: 99%

Implications of ceiling effects in defect predictors

Menzies

Turhan

Bener

et al. 2008

Proceedings of the 4th International Workshop on Predictor Models in Software Engineering

149

113

View full text Add to dashboard Cite

show abstract

“…As of today, no large-scale study involving human experts has been conducted in TR [D. Cuddeback(2010)]. Different studies show the importance of experts' opinion in TR [D. Cuddeback(2010), Dekhtyar et al(2007)Dekhtyar, Hayes, and Larsen, Eder et al(1999)Eder, Panagos, and Rabinovich, Soloway and Ehrlich(1989]. Ghazarian et al [Ghazarian(2009)] showed that developers cause 82% of the problems of missed implementation in some software systems.…”

Section: Experts' Opinionmentioning

confidence: 99%

Factors Impacting the Inputs of Traceability Recovery Approaches

Ali

Guéhéneuc

Antoniol

2011

Software and Systems Traceability

View full text Add to dashboard Cite

In requirement engineering, researchers have proposed various tractability recovery approaches. To the best of our knowledge, all traceability recovery approaches have low precision and recall. Our main claim in this chapter is that there exist factors that impact the traceability approaches' inputs, in particular source document, target document, and experts' opinion, that cause low precision and recall. In this chapter, we pursue four objectives: first, to identify and document factors that impact traceability recovery approaches' inputs; second, to identify metrics/tools to measure/improve the quality of the inputs with respect to the identified factors, third, to provide precautions to control these factors, and, fourth, to empirically prove and quantify the effect of one of these factors-expert's programming knowledge-on the traceability recovery approaches' inputs. To achieve the first two objectives, we perform an incremental literature review of traceability recovery approaches and identify and document three key inputs and the seven factors impacting these inputs, out of 12 identified factors. We analyse the reported results in literature for the identified factors to address our third objective. We conduct an empirical study to assess the impact of expert's programming knowledge, to address our fourth objective. We use the effort, number of correct answers, and time to measure the effect of expert's programming knowledge on traceability recovery. We conclude that, in the literature, seven factors impacting the inputs of traceability recovery approaches have been identified, documented, and reported along with related metrics/tools and precautions. We suggest that practitioners should be wary of these seven factors and researchers should focus on the five others to improve traceability recovery approaches.

show abstract

“…The three constants α, β, and χ are provided so that a level of importance can be specified by the user for the initial query, the relevant documents and the irrelevant documents. According to [9], placing emphasis on the relevant documents may improve the recall (new relevant documents may be found) while emphasizing the irrelevant documents may affect precision (false positives may be removed). Joachims recommends weighting the positive information four times higher than the negative [12].…”

Section: Rf With Rocchiomentioning

confidence: 99%