A Critical Analysis of Empirical Research in Software Testing

Abrahamsson, Pekka; Moser, Raimund; Pedrycz, Witold; Sillitti, Alberto; Succi, Giancarlo

doi:10.1109/esem.2007.40

Cited by 53 publications

(52 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…1, the identification of categories and choices (or their equivalents) is also needed in domain testing (Beizer 1990), equivalence partitioning (Myers 2004), and in-parameter-order (Lei and Tai 1998;Tai and Lei 2002). Second, in line with the thoughts of the software community (Briand 2007;Carver et al 2008;Porter and Johnson 1997;Tichy 1998), observation of human performance is an essential element of software engineering. In this regard, our results should play a part in the contributions to software engineering research.…”

Section: Discussionmentioning

confidence: 99%

Contributions of tester experience and a checklist guideline to the identification of categories and choices for software testing

et al. 2010

View full text Add to dashboard Cite

An early step for most black-box testing methods is to identify a set of categories and choices (or their equivalents) from the specification. The identification is often performed in an ad hoc manner, thus the quality of categories and choices is in doubt. Poorly identified categories and choices will affect the comprehensiveness of test cases. In this paper, we describe several comparative studies using three commercial specifications and discuss the major results. The objectives of our studies are (a) to investigate the differences in the types and amounts of mistakes made between inexperienced and experienced software testers in an ad hoc identification approach and (b) to determine the extent of mistake reduction after discussing the mistakes with the software testers and providing them with an identification checklist.

show abstract

Section: Discussionmentioning

confidence: 99%

Contributions of tester experience and a checklist guideline to the identification of categories and choices for software testing

et al. 2010

View full text Add to dashboard Cite

show abstract

“…Unfortunately, Software testing has been described as difficult, time consuming and inadequate [1], [2]. Several testing techniques are available now but there is no consensus on the absolute or relative applicability or usability of these techniques.…”

Section: A Software Testingmentioning

confidence: 99%

“…A testing technique can be viewed as a coverage criterion and an oracle [1]. The general belief is that the choice of a test oracle is generally orthogonal to that of test criterion.…”

Section: G Test Oraclesmentioning

confidence: 99%

“…Moreover, it is expected that software testing will remain in the future as one of the best available tools that we have to ensure software reliability [1].…”

Section: A Software Testingmentioning

confidence: 99%

“…As we mentioned before, empirical studies are the best candidate to evaluate the effectiveness of testing techniques [1], [3], [5], [13]. Empirical studies in testing are characterized by being very difficult and expensive in terms of time consumption and labor work [5], [17].…”

Section: The Lack Of Well-formed Empirical Studiesmentioning

confidence: 99%

See 2 more Smart Citations

Open Problems in Software Test Coverage

Alrmuny¹

2014

LNSE

View full text Add to dashboard Cite

Abstract-Software testing is concerned with the production and maintenance of high quality systems. Test coverage is one major component in software testing. It gives a meaningful description of how much testing is required. The level of test coverage is an indicator of testing thoroughness, which in turn helps testers decide when to stop testing. Several studies addressed coverage criteria effectiveness and adequacy. However, there are several interesting open problems. We identify nine open problems in test coverage and discuss what issues need to be addressed. The objective of this paper is to direct the attention of researchers and testers toward the areas that need to be investigated.

show abstract

A replicated empirical study to evaluate software testing methods

2017

View full text Add to dashboard Cite

Many empirical studies have been performed to evaluate software testing methods in past decades. However, we are still not able to generalize the results as most studies are not complete and differ significantly from one another. To contribute to the existing knowledge base of software testing methods, we performed an empirical study to evaluate 3 testing methods: (1) code reading by stepwise abstraction, (2) functional testing using equivalence partitioning and boundary value analysis, and (3) structural testing using 100% branch, multiple-condition, loop, and relational-operator coverage using a well-defined and standard schema. A controlled experiment is performed with 18 subjects who applied the 3 defect detection techniques to 3 C programs in a fractional factorial experimental design to observe failures and isolate faults. The experimental results show that (1) the techniques are equally effective in terms of observing failures and finding faults, (2) the effectiveness of the techniques depends on the nature of the program, (3) all the testing techniques are equally efficient in case of failure observation, (4) the techniques are different in their efficiency in terms of fault isolation where code reading performed better than that of structural and functional testing, and (5) with respect to the fault types, all the techniques were equally effective in observing failures and isolating faults except in case of cosmetic faults where functional testing performed better than the other 2 techniques The effectiveness and efficiency of testing techniques were significantly influenced by the type of program. The results presented in this paper contribute to an empirical knowledge base of testing methods and may be helpful for the software engineers to decide the appropriate techniques in improving the software testing process.

show abstract

A Critical Analysis of Empirical Research in Software Testing

Cited by 53 publications

References 19 publications

Contributions of tester experience and a checklist guideline to the identification of categories and choices for software testing

Contributions of tester experience and a checklist guideline to the identification of categories and choices for software testing

Open Problems in Software Test Coverage

A replicated empirical study to evaluate software testing methods

Contact Info

Product

Resources

About