Component Testing Is Not Enough - A Study of Software Faults in Telecom Middleware

Abstract

Abstract. The interrelationship between software faults and failures is quite intricate and obtaining a meaningful characterization of it would definitely help the testing community in deciding on efficient and effective test strategies. Towards this objective, we have investigated and classified failures observed in a large complex telecommunication industry middleware system during [2003][2004][2005][2006]. In this paper, we describe the process used in our study for tracking faults from failures along with the details of failure data. We present the distribution and frequency of the failures along with some interesting findings unravelled while analyzing the origins of these failures. Firstly, though "simple" faults happen, together they account for only less than 10%. The majority of faults come from either missing code or path, or superfluous code, which are all faults that manifest themselves for the first time at integration/system level; not at component level. These faults are more frequent in the early versions of the software, and could very well be attributed to the difficulties in comprehending and specifying the context (and adjacent code) and its dependencies well enough, in a large complex system with time to market pressures. This exposes the limitations of component testing in such complex systems and underlines the need for allocating more resources for higher level integration and system testing.

A Case Study of Source Code Evolution

“…However, based on the results obtained from a pilot study undertaken as preparation for this current study, we custom defined our fault classification scheme to tailor the needs of the case under study. This approach is consistent with the view expressed by Eldh et al [7] that different software domains have different sets of faults and fault distributions. In our classification, we used the following 15 classes of faults:…”

“…The fault taxonomy used in our study is partly derived from the ones used in other studies of faults such as [7] and [5]. However, based on the results obtained from a pilot study undertaken as preparation for this current study, we custom defined our fault classification scheme to tailor the needs of the case under study.…”

“…Eldh et al [7] studied and classified the failures in a large complex telecommunication middleware system. They concluded that (a) faults of omission are the dominating class of software faults (b) faults of omission (38.3% of software faults), together with spurious faults (8.3% of all software faults) indicate that faults related to unclear specifications dominate among the real software faults in the telecom system studied.…”

Towards a Test Automation Improvement Model (TAIM)

“…We have looked at the automatic build-and regression test management which is performed after a system fix [10], thereby combining several KA in our model. We have worked with triaging and fault localization [8] as well as fault classifications [12] and have attempted to compare requirements of test tools [13].…”

A Test Suite Generation Method for Component Interaction Testing

“…Although unit testing can validate the correctness of local module [1], interaction errors may be still existed during component composition in practice [2], so component interaction should test thoroughly and completely during integration [3].…”