Improving test suites via operational abstraction

Harder, Michael; Mellen, Jeff; Ernst, M.

doi:10.1109/icse.2003.1201188

Cited by 112 publications

(124 citation statements)

References 39 publications

(30 reference statements)

Supporting

Mentioning

120

Contrasting

Order By: Relevance

“…This is different from the problems that traditional test selection techniques address [6,21]. In those problems, there are test oracles for unselected test inputs.…”

Section: Unit-test Generation and Selectionmentioning

confidence: 99%

“…An operational abstraction is a collection of logical statements that abstract the program's runtime behavior [21]. It is syntactically identical to a formal specification.…”

Section: Operational Abstraction Generationmentioning

confidence: 99%

“…An operational abstraction describes the actual behavior during program execution of an existing unit test suite [21]. We use the generated operational abstractions to guide test generation tools, so that the tools can more effectively generate test inputs to violate these operational abstractions.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Tool-assisted unit-test generation and selection based on operational abstractions

Xie

Notkin

2006

Autom Software Eng

View full text Add to dashboard Cite

Abstract. Unit testing, a common step in software development, presents a challenge. When produced manually, unit test suites are often insufficient to identify defects. The main alternative is to use one of a variety of automatic unit-test generation tools: these are able to produce and execute a large number of test inputs that extensively exercise the unit under test. However, without a priori specifications, programmers need to manually verify the outputs of these test executions, which is generally impractical. To reduce this cost, unit-test selection techniques may be used to help select a subset of automatically generated test inputs. Then programmers can verify their outputs, equip them with test oracles, and put them into the existing test suite. In this paper, we present the operational violation approach for unit-test generation and selection, a black-box approach without requiring a priori specifications. The approach dynamically generates operational abstractions from executions of the existing unit test suite. These operational abstractions guide test generation tools to generate tests to violate them. The approach selects those generated tests violating operational abstractions for inspection. These selected tests exercise some new behavior that has not been exercised by the existing tests. We implemented this approach by integrating the use of Daikon (a dynamic invariant detection tool) and Parasoft Jtest (a commercial Java unit testing tool), and conducted several experiments to assess the approach.

show abstract

“…This is different from the problems that traditional test selection techniques address [6,21]. In those problems, there are test oracles for unselected test inputs.…”

Section: Unit-test Generation and Selectionmentioning

confidence: 99%

“…An operational abstraction is a collection of logical statements that abstract the program's runtime behavior [21]. It is syntactically identical to a formal specification.…”

Section: Operational Abstraction Generationmentioning

confidence: 99%

See 1 more Smart Citation

Tool-assisted unit-test generation and selection based on operational abstractions

Xie

Notkin

2006

Autom Software Eng

View full text Add to dashboard Cite

show abstract

“…In the case of its absence, we propose that field data could be exploited in the form of count spectra to capture the frequency at which entities are exercised by real users. Harder et al (2003) proposed a specification-based test case selection technique called "operational difference." Their technique dynamically generates a collection of logical statements (called "operational abstractions") that abstract the program's operation based on the execution of test cases.…”

Section: Related Workmentioning

confidence: 99%

“…Besides that, although the same term of operational coverage is defined in Harder et al, the meaning is different from the one we used here. In (Harder et al 2003), operational coverage is defined in terms of precision and recall and the term is used to evaluate the quality of an operational abstraction when compared with an oracle or goal specification. In our work, operational coverage is a coverage criterion that can be used for both test adequacy and selection in the context of operational profile-based testing.…”

Section: Related Workmentioning

confidence: 99%

An assessment of operational coverage as both an adequacy and a selection criterion for operational profile based testing

Miranda

Bertolino²

2017

Software Qual J

View full text Add to dashboard Cite

While the relation between code coverage measures and fault detection is actively studied, only few works have investigated the correlation between measures of coverage and of reliability. In this work, we introduce a novel approach to measuring code coverage, called the operational coverage, that takes into account how much the program's entities are exercised so to reflect the profile of usage into the measure of coverage. Operational coverage is proposed as (i) an adequacy criterion, i.e., to assess the thoroughness of a black box test suite derived from the operational profile, and as (ii) a selection criterion, i.e., to select test cases for operational profile-based testing. Our empirical evaluation showed that operational coverage is better correlated than traditional coverage with the probability that the next test case derived according to the user's profile will not fail. This result suggests that our approach could provide a good stopping rule for operational profile-based testing. With respect to test case selection, our investigations revealed that operational coverage outperformed the traditional one in terms of test suite size and fault detection capability when we look at the average results.

show abstract

Ongoing verification of a multiphysics community code: FLASH

et al. 2013

View full text Add to dashboard Cite

When developing a complex, multi-authored code, daily testing on multiple platforms and under a variety of conditions is essential. It is therefore necessary to have a regression test suite that is easily administered and configured, as well as a way to easily view and interpret the test suite results. We describe the methodology for verification of FLASH, a highly capable multiphysics scientific application code with a wide user base. The methodology uses a combination of unit and regression tests and an in-house testing software that is optimized for operation under limited resources. Although our practical implementations do not always comply with theoretical regression-testing research, our methodology provides a comprehensive verification of a large scientific code under resource constraints.

show abstract

Improving test suites via operational abstraction

Cited by 112 publications

References 39 publications

Tool-assisted unit-test generation and selection based on operational abstractions

Tool-assisted unit-test generation and selection based on operational abstractions

An assessment of operational coverage as both an adequacy and a selection criterion for operational profile based testing

Ongoing verification of a multiphysics community code: FLASH

Contact Info

Product

Resources

About