Extended firewall for regression testing: an experience report

Abstract

Testing firewalls have proven to be a useful approach for regression testing in both functional and objectoriented software. They involve only the modules that are closely related to the changed modules. They lead to substantially reduced regression tests but still are very effective in detecting regression faults. This paper investigates situations when data-flow paths are longer, and the testing of modules and components only one level away from the changed elements may not detect certain regression faults; an extended firewall considers these longer data paths. We report empirical studies that show the degree to which an extended firewall detected more faults, and how much more testing was required to achieve this increased detection. L. WHITE ET AL.based on a repeated software change, and the goal of the regression testing is to find residual problems in the unchanged parts of the software.A primitive regression testing strategy would re-run all tests that validate functionality of all software components that were not impacted by the change. However, the regression tests need to be repeated frequently, making the re-run of the complete regression test too expensive; therefore, several heuristics were developed as a response. Selective testing means that although an initial test suite has been developed and is available, the programmers select only a subset pertinent to testing the specific changes made in the software [1,2]. Another approach emphasizes prioritization, where the priority is established among the tests and the tests are run within the available time, based on their priority [3].Selective and prioritization testing assumes that the regression test suite exists and has the desirable quality. However, we have found that often in industrial practice, the initial test suites are inconsistent and poorly constructed. For this situation, we have developed an approach to regression testing called firewall testing that emphasizes an on-demand test creation rather than selection from a given testing suite; of course, existing tests can always be reused, if they satisfy the firewall criterion. The firewall testing considers only components that are directly interacting with the changed components (one level away) and thoroughly tests them. Firewall testing is based on the observation that most residual bugs are caused by a failure of the programmer to propagate the change to some directly interacting program components. The firewall is an imaginary boundary between these components and the rest of the software system.As a method of regression testing for industrial situations, the testing firewall (TFW) has proven to be a useful approach for both procedure-oriented and object-oriented software. It has turned out to be particularly useful in the situations where the test suites are incomplete, which is a common situation in industrial software development; the firewall allows the identification of missing tests and provides a simple procedure to develop these missing tests. TFW has been developed for vario...

Migration of information systems in the Italian industry: A state of the practice survey

“…Also, companies indicated problems related to testing the migrated application and to project planning, confirming something we knew from discussions with industry managers: in many companies there is no consolidated practice for testingand in particular for regression testing. Despite the great advances academia has done in this context [11][12][13][14][15], such an activity is often performed in an ad-hoc manner. Similarly, effort estimation and project planning could benefit more of the advances research made during recent (and past) years [16][17][18][19].…”

Variable granularity for improving precision of impact analysis

“…We will investigate whether the inspected sets, built during the IA process, can be used as regression testing firewalls; firewalls are the sets of program components that have to be retested to ensure that the existing functionality is not broken when new features are introduced [30].…”

Recent Advances in Regression Testing Techniques

“…The approach sets a firewall around the modified modules and the modules related to them, and selects test cases within the firewall. This approach has been applied to a different language paradigm (object-oriented language [21,22]) and wider application domains (eg, GUI, COTS applications, and distributed software [23][24][25]). • Graph walking approach: This approach was presented by Rothermel and Harrold [10,13,15], which was already discussed.…”