Using benchmarking to advance research: a challenge to software engineering

Sim, Susan Elliott; Easterbrook, Steve; Holt, Richard C.

doi:10.1109/icse.2003.1201189

Cited by 158 publications

(176 citation statements)

References 13 publications

Supporting

Mentioning

174

Contrasting

Order By: Relevance

“…As noted by Sim et al, benchmarks also need to evolve [27]; this our case as well. Our benchmark is still relatively small, so we need to integrate the histories of other programs.…”

Section: Discussionmentioning

confidence: 99%

Replaying IDE interactions to evaluate and improve change prediction approaches

Robbes

Pollet

Lanza

2010

2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010)

View full text Add to dashboard Cite

Abstract-Change prediction helps developers by recommending program entities that will have to be changed alongside the entities currently being changed. To evaluate their accuracy, current change prediction approaches use data from versioning systems such as CVS or SVN. These data sources provide a coarse-grained view of the development history that flattens the sequence of changes in a single commit. They are thus not a valid basis for evaluation in the case of developmentstyle prediction, where the order of the predictions has to match the order of the changes a developer makes.We propose a benchmark for the evaluation of change prediction approaches based on fine-grained change data recorded from IDE usage. Moreover, the change prediction approaches themselves can use the more accurate data to fine-tune their prediction. We present an evaluation procedure and use it on several change prediction approaches, both novel and from the literature, and report on the results.

show abstract

“…As noted by Sim et al, benchmarks also need to evolve [27]; this our case as well. Our benchmark is still relatively small, so we need to integrate the histories of other programs.…”

Section: Discussionmentioning

confidence: 99%

Replaying IDE interactions to evaluate and improve change prediction approaches

Robbes

Pollet

Lanza

2010

2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010)

View full text Add to dashboard Cite

show abstract

“…Sim et al challenged the software engineering community to create benchmarks, or a set of tests used to compare the performance of alternative techniques [37]. As such, benchmarks in the form of ontology-based frameworks 1 have been published in software engineering [16,21,24,43].…”

Section: Introductionmentioning

confidence: 99%

Toward a framework for evaluating extreme programming

Williams¹,

Krebs²,

Layman³

et al. 2004

"8th Internation Conference on Empirical Assessment in Software Engineering (EASE 2004)" Workshop - 26th International Conferen

View full text Add to dashboard Cite

show abstract

“…In essence, a benchmark consists of a collection of subject data that can be consumed by different techniques, and can be used to draw coherent and valid conclusions about the respective performance of these techniques. The case for benchmarking is discussed comprehensively by Sim et al (2003 Perhaps one aspect that stands out with respect to competitions is the fact that, by the nature of a competition, it encourages wide-spread participation in the way that a conventional benchmark might not. Competitions offer an incentive to participate (the offer of a prize, and the potential to top the hall of fame).…”

Section: Benchmarking In Software Engineering Researchmentioning

confidence: 99%

STAMINA: a competition to encourage the development and assessment of software model inference techniques

et al. 2012

View full text Add to dashboard Cite

Models play a crucial role in the development and maintenance of software systems, but are often neglected during the development process due to the considerable manual effort required to produce them. In response to this problem, numerous techniques have been developed that seek to automate the model generation task with the aid of increasingly accurate algorithms from the domain of Machine Learning. From an empirical perspective, these are extremely challenging to compare; there are many factors that are difficult to control (e.g. the richness of the input and the complexity of subject systems), and numerous practical issues that are just as troublesome (e.g. tool availability). This paper describes the StaMinA (State Machine Inference Approaches) competiton, that was designed to address these This work is supported by the EPSRC STAMINA project (EP/H002456/1), the EPSRC REGI project (EP/F065825/1), the Regional Government of Wallonia (GISELE project, RW Conv. 616425) (2013) 18:791-824 problems. The competition attracted numerous submissions, many of which were improved or adapted versions of techniques that had not been subjected to extensive empirical evaluations, and had not been evaluated with respect to their ability to infer models of software systems. This paper shows how many of these techniques substantially improve on the state of the art, providing insights into some of the factors that could underpin the success of the best techniques. In a more general sense it demonstrates the potential for competitions to act as a useful basis for empirical software engineering by (a) spurring the development of new techniques and (b) facilitating their comparative evaluation to an extent that would usually be prohibitively challenging without the active participation of the developers.

show abstract

Using benchmarking to advance research: a challenge to software engineering

Cited by 158 publications

References 13 publications

Replaying IDE interactions to evaluate and improve change prediction approaches

Replaying IDE interactions to evaluate and improve change prediction approaches

Toward a framework for evaluating extreme programming

STAMINA: a competition to encourage the development and assessment of software model inference techniques

Contact Info

Product

Resources

About