Transformed Vargha-Delaney Effect Size

Neumann, Geoffrey; Harman, Mark; Poulding, Simon

doi:10.1007/978-3-319-22183-0_29

Cited by 37 publications

(29 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While these larger schemas see reduced overheads with the virtual technique, the 5 smaller schemas do not benefit from the decrease in database interactions afforded by the presented method, thus leading to the negative values of the percentage of mean time saved seen in Figure 4. Yet, even in these cases in which a small schema and a fast DBMS should outperform virtual mutation analysis, we found that the difference in execution time was always less than 100 milliseconds, a negligible amount that experts agree is not perceivable by users of a software tool [12].…”

Section: Resultsmentioning

confidence: 62%

“…Values of Â12 close to the 0.5 value are viewed as showing no effect. When discussing effect sizes for the execution times of the two methods, we follow Neumann et al [12] and say that a value of Â12 closer to zero indicates that virtual is the preferred technique while a value near one shows that standard is faster.…”

Section: Rq1 (Efficiency): How Does the Time Overhead Of Virtual Mutamentioning

confidence: 99%

“…To control for these threats we ran 30 trials and used box and whisker plots and statistical tests to analyze the results. Adhering to the advice of Neumann et al [12], we disregarded all timing differences of 100 ms -as they would not be perceived by users of our toolto ensure that we did not misapply the Vargha-Delaney effect size. Another threat to the validity of our results is potential defects in the data generator or one of the mutation analysis methods.…”

Section: Rq1 (Efficiency): How Does the Time Overhead Of Virtual Mutamentioning

confidence: 99%

See 2 more Smart Citations

Virtual mutation analysis of relational database schemas

McMinn

Kapfhammer

Wright

2016

Proceedings of the 11th International Workshop on Automation of Software Test

View full text Add to dashboard Cite

Relational databases are a vital component of many modern software applications. Key to the definition of the database schemawhich specifies what types of data will be stored in the database and the structure in which the data is to be organized -are integrity constraints. Integrity constraints are conditions that protect and preserve the consistency and validity of data in the database, preventing data values that violate their rules from being admitted into database tables. They encode logic about the application concerned, and like any other component of a software application, need to be properly tested. Mutation analysis is a technique that has been successfully applied to integrity constraint testing, seeding database schema faults of both omission and commission. Yet, as for traditional mutation analysis for program testing, it is costly to perform, since the test suite under analysis needs to be run against each individual mutant to establish whether or not it exposes the fault. One overhead incurred by database schema mutation is the cost of communicating with the database management system (DBMS). In this paper, we seek to eliminate this cost by performing mutation analysis virtually on a local model of the DBMS, rather than on an actual, running instance hosting a real database. We present an empirical evaluation of our virtual technique revealing that, across all of the studied DBMSs and schemas, the virtual method yields an average time saving of 51% over the baseline.

show abstract

Section: Resultsmentioning

confidence: 62%

Section: Rq1 (Efficiency): How Does the Time Overhead Of Virtual Mutamentioning

confidence: 99%

Section: Rq1 (Efficiency): How Does the Time Overhead Of Virtual Mutamentioning

confidence: 99%

See 1 more Smart Citation

Virtual mutation analysis of relational database schemas

McMinn

Kapfhammer

Wright

2016

Proceedings of the 11th International Workshop on Automation of Software Test

View full text Add to dashboard Cite

show abstract

“…The completion of Tom's analysis is further delayed when his colleague suggests that he read a recent article, by Neumann et al [4], suggesting that Vargha-Delaney effect sizes be "transformed" to ensure that the correct conclusions are reached. Since Tom is computing effect sizes for the execution timings of an ATDG algorithm, he realises that, following Neumann et al's advice, he must transform his large data set.…”

Section: Hitchhikers' Dilemmasmentioning

confidence: 99%

“…When new analysis ideas emerge (e.g., the transformations proposed by Neumann et al [4]), then the developers of these new methods can fork sbst-analysis (as shown by the in label), add new functions or modify the existing ones, and then submit a merge request back to the maintainers of sbst-analysis; once approved the new code will be available to the community, thereby speeding the uptake of this new analysis idea. Now, let's assume that sbst-analysis exists and Tom decides to use it to help him calculate effect sizes.…”

Section: Shared Repositoriesmentioning

confidence: 99%

Hitchhikers need free vehicles!

Kapfhammer

McMinn

Wright

2016

Proceedings of the 9th International Workshop on Search-Based Software Testing

View full text Add to dashboard Cite

As a means for improving the maturity of the data analysis methods used in the search-based software testing field, this paper presents the need for shared repositories of well-documented statistical analysis code and replication data. In addition to explaining the benefits associated with using these repositories, the paper gives suggestions (e.g., the testing of analysis code) for improving the study of data arising from experiments with randomized algorithms.

show abstract

An empirical comparison of validation methods for software prediction models

Ali

Gravino

2021

J Software Evolu Process

View full text Add to dashboard Cite

Model validation methods (e.g., k-fold cross-validation) use historical data to predict how well an estimation technique (e.g., random forest) performs on the current (or future) data. Studies in the contexts of software development effort estimation (SDEE) and software fault prediction (SFP) have used and investigated different model validation methods. However, no conclusive indications to suggest which model validation method has a major impact on the prediction accuracy and stability of estimation techniques. Some studies have investigated model validation methods specific to data about either SDEE or SFP. To the best of our knowledge, there is no study in the literature, which has employed different validation methods both with SDEE and SFP data. The aim of this paper is to consider different methods (10) from the family of cross-validation (CV) and bootstrap validation methods to identify which one contributes to obtaining a better prediction accuracy for both types of data. We also evaluate which model validation methods allow the estimation techniques to provide stable performances (i.e., with lower variance). To this aim, we present an empirical study involving six datasets from the domain of SDEE and six datasets from the SFP domain. The results reveal that repeated 10-fold CV with SDEE and optimistic boot with SFP data are the model validation methods that provide a better prediction accuracy in a greater number of experiments than the other model validation methods. Furthermore, a model validation method can improve the prediction accuracy up to 60% with SDEE data and up to 36% when employing SFP data. The analysis also reveals that repeated fivefold CV produces more stable performances when the experiments are repeated on the same data.

show abstract

Transformed Vargha-Delaney Effect Size

Cited by 37 publications

References 16 publications

Virtual mutation analysis of relational database schemas

Virtual mutation analysis of relational database schemas

Hitchhikers need free vehicles!

An empirical comparison of validation methods for software prediction models

Contact Info

Product

Resources

About