A methodology for certification of modeling and simulation applications

Abstract

Certification of modeling and simulation (M&S) applications poses significant technical challenges for M&S program managers, engineers, and practitioners. Certification is becoming increasingly more important as M&S applications are used more and more for military training, complex system design evaluation, M&S-based acquisition, problem solving, and critical decision making. Certification, a very complex process, involves the measurement and evaluation of hundreds of qualitative and quantitative elements, mandates subject matter expert evaluation, and requires the integration of different evaluations. Planning and managing such measurements and evaluations requires a unifying methodology and should not be performed in an ad hoc manner. This paper presents such a methodology. The methodology consists of the following body of methods, rules, and postulates: (a) employment of subject matter experts, (b) construction of a hierarchy of indicators, (c) relative criticality weighting of indicators using the analytic hierarchy process, (d) using a rule-based expert knowledge base with an object-oriented specification language, (e) assignment of crisp, fuzzy, and nominal scores for the indicators, (f ) aggregation of indicator scores, (g) graphical representation of the indicator scores and weights, (h) hypertext certification report, and (i) interpretation of the results. The methodology can be used for certification of any kind of M&S application either throughout the M&S development life cycle or after the development is completed.

Conceptual modelling for simulation Part I: definition and requirements

“…Assessment criteria for models have been discussed by a number of authors, for instance, Gass and Joel (1981), Ören (1981, 1984), Robinson and Pidd (1998) and Balci (2001). The majority of this work is in the domain of large scale military and public policy models;…”

Independent Verification and Validation of an Industrial Simulation Model

“…Balci [8] points out that this use of terminology is at odds with the International Standards Organization, and suggests the use of the term 'certification' is more appropriate. Tullos-Banks et al [9] identify a clear distinction between IV&V, certification and verification, validation and accreditation (VV&A).…”

“…In a later paper, Balci [8] suggests that seven top level indicators should be used for model certification: These indicators explicitly identify the need to assess not only the product, but also the process (model life-cycle) and project (management issues) in the modelling effort [21]. Balci [22] also identifies the importance of assessing the quality of the model developers.…”

“…Most of these entail subjective judgements based on evidence, although more objective scores are possible for some indicators. In describing his certification methodology, Balci [8] identifies three types of scoring that might be used: crisp (a single value between 0 and 100; fuzzy (an interval between 0 and 100); nominal (a named score which is assigned a predetermined value).…”

“…When these weights are combined with a level of satisfaction score for each criterion (intensity), determined by an independent assessor, an overall numerical rating (between 0 and 1) is calculated. Balci [8] also employs the Analytic Hierarchy Process for weighting indicators to calculate a final score. Pohl and Brade [28] suggest the issue is addressed with fuzzy multi-attribute decision theory.…”