Critical success metrics: evaluation at the business level

MENZIES, TIM

doi:10.1006/ijhc.1999.0329

Cited by 6 publications

(2 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, most of the expert systems literature proposes evaluations based on very few tests 1 , as indicated in Figure 2. For example, Menzies defined a minimal test procedure requiring a mere 40 successful tests to check an expert controller for a complex chemical plant (125 kilometers of highly inter-connected piping) [32]. Upon completion of testing, its performance (in terms of the correctness) is assessed via comparing it with human operators.…”

mentioning

confidence: 99%

Adequacy of Limited Testing for Knowledge Based Systems

Čukić

2000

Int. J. Artif. Intell. Tools

Self Cite

View full text Add to dashboard Cite

Knowledge-based engineering and computational intelligence are expected to become core technologies in the design and manufacturing for the next generation of space exploration missions. The literature is contradictory on how we are to assess such systems. Studies indicate significant disagreement regarding the amount of testing needed for system assessment. The sizes of standard black-box test suites are impractically large since the black-box approach neglects the internal structure of knowledge-based systems. On the contrary, practical results repeatedly indicate that only a few tests are needed to sample the range of behaviors of a knowledge-based program.In this paper, we model testing as a search process over the internal state space of the knowledge-based system. When comparing different test suites, the test suite that examines larger portion of the state space is considered more complete. Our goal is to investigate the trade-off between the completeness criterion and the size of test suites. The results of testing experiment on tens of thousands of mutants of real-world knowledge based systems indicate that a very limited gain in completeness can be achieved through prolonged testing. The use of simple (or random) search strategies for testing appears to be as powerful as testing by more thorough search algorithms.

show abstract

mentioning

confidence: 99%

Adequacy of Limited Testing for Knowledge Based Systems

Čukić

2000

Int. J. Artif. Intell. Tools

Self Cite

View full text Add to dashboard Cite

show abstract

“…'s observation. Menzies [2000] tries to answer a hard question: how to assess a system in the absence of an oracle. One method for oracle-less testing is a critical success metric (CSM).…”

mentioning

confidence: 99%

Editorial: Evaluating knowledge engineering techniques

Harmelen¹

1999

International Journal of Human-Computer Studies

Self Cite

View full text Add to dashboard Cite

Conceptualization Maturity Metrics for Expert Systems

Hauge

Britos

García-Martínez

2006

IFIP International Federation for Information Processing

View full text Add to dashboard Cite

Abstract. Metrics used on development of expert systems is not a well investigated problem area. This article suggests some metrics to be used to measure the maturity of the conceptualization process and the complexity of the decision process in the problem domain. We propose some further work to be done with these metrics. Applying those metrics makes new and interesting problems, concerning the structure of knowledge to surface.

show abstract

Critical success metrics: evaluation at the business level

Cited by 6 publications

References 25 publications

Adequacy of Limited Testing for Knowledge Based Systems

Adequacy of Limited Testing for Knowledge Based Systems

Editorial: Evaluating knowledge engineering techniques

Conceptualization Maturity Metrics for Expert Systems

Contact Info

Product

Resources

About