On human analyst performance in assisted requirements tracing: Statistical analysis

Dekhtyar, Alex; Dekhtyar, Olga; Holden, Jeff; Hayes, Jane Huffman; Cuddeback, David; Kong, Wei-Keat

doi:10.1109/re.2011.6051649

Cited by 37 publications

(71 citation statements)

References 21 publications

(59 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Kong et al [9] analyzed the process of the traceability validation tasks. Dekhtyar et al [10] confirmed the observation by statistical analysis. The analysis indicates that developers cannot utilize accurate traceability links.…”

Section: Related Workmentioning

confidence: 68%

“…As a consequence, a feature location technique with higher recall is a more promising direction. In traceability recovery experiments [8], [9], [10], no significant difference was found between precision and recall. One possible reason for this result is that subjects can understand features in detail during their tasks by reading relevant methods in initial lists.…”

Section: Discussionmentioning

confidence: 99%

“…The table shows that subjects given a less-accurate list can significantly improve the list. In contrast, subjects given an accurate list tend to decrease recall but keep the overall accuracy of the list, differently from the result of a traceability research [10]. The effect of validation is further discussed in Section V-D.…”

Section: A Rq1mentioning

confidence: 95%

“…However, a question remains as to whether the accuracy of automated feature location techniques actually contributes or not to the performance of developers. This question arises from an observation reported by Cuddeback et al [8], Kong et al [9], and Dekhtyar et al [10]. They asked analysts to do manual validation of a traceability matrix between requirements and system tests.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

On the effectiveness of accuracy of automated feature location technique

Ishio

Hayashi

Kawamura³

et al. 2013

2013 20th Working Conference on Reverse Engineering (WCRE)

View full text Add to dashboard Cite

Abstract-Automated feature location techniques have been proposed to extract program elements that are likely to be relevant to a given feature. A more accurate result is expected to enable developers to perform more accurate feature location. However, several experiments assessing traceability recovery have shown that analysts cannot utilize an accurate traceability matrix for their tasks. Because feature location deals with a certain type of traceability links, it is an important question whether the same phenomena are visible in feature location or not. To answer that question, we have conducted a controlled experiment. We have asked 20 subjects to locate features using lists of methods of which the accuracy is controlled artificially. The result differs from the traceability recovery experiments. Subjects given an accurate list would be able to locate a feature more accurately. However, subjects could not locate the complete implementation of features in 83% of tasks. Results show that the accuracy of automated feature location techniques is effective, but it might be insufficient for perfect feature location.

show abstract

Section: Related Workmentioning

confidence: 68%

Section: Discussionmentioning

confidence: 99%

Section: A Rq1mentioning

confidence: 95%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

On the effectiveness of accuracy of automated feature location technique

Ishio

Hayashi

Kawamura³

et al. 2013

2013 20th Working Conference on Reverse Engineering (WCRE)

View full text Add to dashboard Cite

show abstract

“…Some of the prior research by one of the authors, concentrated on studying what human analysts do with the results of automated methods and whether they tend to recognize good advice provided by the automated methods [3], [2], [4]. This work indicated that not all trace links are equally easily recognizable by human analysts: some links present in the experimental studies were almost uniformally recovered and recognized by humans, while other links were extremely difficult for them.…”

Section: Introductionmentioning

confidence: 99%

Human recoverability index: A TraceLab experiment

Dekhtyar

Hilton

2013

2013 7th International Workshop on Traceability in Emerging Forms of Software Engineering (TEFSE)

Self Cite

View full text Add to dashboard Cite

Abstract-It has been generally accepted that not all trace links in a given requirements traceability matrix are equal -both human analysts and automated methods are good at spotting some links, but have blind spots for some other. One way to choose automated techniques for inclusion in assisted tracing processes (i.e., the tracing processes that combine the expertise of a human analyst and special-purpose tracing software) is to select the techniques that tend to discover more links that are hard for human analysts to observe and establish on their own. This paper proposes a new measure of performance of a tracing method: human recoverability index-based recall. In the presence of knowledge about the difficulty of link recovery by human analysts, this measure rewards methods that are able to recover such links over methods that tend to recover the same links as the human analysts. We describe a TraceLab experiment we designed to evaluate automated trace recovery methods based on this measure and provide a case study of the use of this experiment to profile and evaluate different automated tracing techniques.

show abstract

Error leakage and wasted time: sensitivity and effort analysis of a requirements consistency checking process

Hayes

Antoniol

et al. 2016

J Software Evolu Process

Self Cite

View full text Add to dashboard Cite

Several techniques are used by requirements engineering practitioners to address difficult problems such as specifying precise requirements while using inherently ambiguous natural language text and ensuring the consistency of requirements. Often, these problems are addressed by building processes/tools that combine multiple techniques where the output from 1 technique becomes the input to the next. While powerful, these techniques are not without problems.Inherent errors in each technique may leak into the subsequent step of the process. We model and study 1 such process, for checking the consistency of temporal requirements, and assess error leakage and wasted time. We perform an analysis of the input factors of our model to determine the effect that sources of uncertainty may have on the final accuracy of the consistency checking process. Convinced that error leakage exists and negatively impacts the results of the overall consistency checking process, we perform a second simulation to assess its impact on the analysts' efforts to check requirements consistency. We show that analyst's effort varies depending on the precision and recall of the subprocesses and that the number and capability of analysts affect their effort. We share insights gained and discuss applicability to other processes built of piped techniques.

show abstract

On human analyst performance in assisted requirements tracing: Statistical analysis

Cited by 37 publications

References 21 publications

On the effectiveness of accuracy of automated feature location technique

On the effectiveness of accuracy of automated feature location technique

Human recoverability index: A TraceLab experiment

Error leakage and wasted time: sensitivity and effort analysis of a requirements consistency checking process

Contact Info

Product

Resources

About