Adequacy of Limited Testing for Knowledge Based Systems

Abstract

Knowledge-based engineering and computational intelligence are expected to become core technologies in the design and manufacturing for the next generation of space exploration missions. The literature is contradictory on how we are to assess such systems. Studies indicate significant disagreement regarding the amount of testing needed for system assessment. The sizes of standard black-box test suites are impractically large since the black-box approach neglects the internal structure of knowledge-based systems. On the contrary, practical results repeatedly indicate that only a few tests are needed to sample the range of behaviors of a knowledge-based program.In this paper, we model testing as a search process over the internal state space of the knowledge-based system. When comparing different test suites, the test suite that examines larger portion of the state space is considered more complete. Our goal is to investigate the trade-off between the completeness criterion and the size of test suites. The results of testing experiment on tens of thousands of mutants of real-world knowledge based systems indicate that a very limited gain in completeness can be achieved through prolonged testing. The use of simple (or random) search strategies for testing appears to be as powerful as testing by more thorough search algorithms.

Saturation effects in testing of formal models

“…Based on our experiments, two average case mathematical analyses [38,41], and a literature review [35], we believe the saturation effect is common. We further speculate that random search will be sufficient in many domains.…”

“…Lines 29-36 prepare for the time i + 1. Nodes found at time i are put into the queue for input to time i + 1 (line 32), and all other nodes are reset to their initial waitValues (lines [33][34][35][36].…”

Testing nondeterminate systems

“…Based on a mathematical analysis of NAYO graph reachability and a literature review [12], we have argued that wide range of nondeterminate systems have easy reachability; i.e. they can be adequately and quickly probed with a small number of randomly selected tests.…”

“…One interesting feature is that for a wide range of NAYO graphs, most of those graphs are reachable using a small number of randomly generated inputs despite NAYO computation being nondeterminate. A recent literature review offers much evidence that this is a commonly observed effect [12]. For example, researchers in AI and requirements engineering explore inconsistent and nondeterminate theories.…”

“…An often repeated observation is that a small number of inputs can often reach significant errors in a program [4,12]. Various researchers have noted that the portions of a program used in normal operation are a small subset of the total program [3,6], and that test coverage suites often do not target the entire program [8,11].…”