Daniel P. Murphy/Steven M. Mosher Black Diamond Consultants 1079 E. 37th Street Long Beach, CA 90807-4150

Abstract

Roger Smith's recent article addressed many relevant issues with respect to simulating in a Complete Man in the Loop Environment. The comprehensive environment has the potential to yield more robust and powerful results. However, at the same time, we also feel it is imperative that the analyst and end user of the information recognize the impact of the 'complete' MITL environment.With the advent of a comprehensive simulation environment, the experiment's chosen response variable is also more susceptible to variations introduced from ('interdependent') contributors. The greater the number of interconnected modules/physical effects, the greater the number of sources of variation. A simple hypothesis test comparing the performance of System A versus System B may Lulcover ~ a trend, but the trend may only exist at an litsigiiific(7iit confidence level.One of the more obvious sources of variation is the man in the loop. In some tests which we have monitored the human operator was shown to vary in effectiveness with variation on the order of 16:1. We can certainly block this phenomena. But if the goal is to determine the viability of the system, then inclusion of a representative population of participants is desirable. The drawback in increasing the number of factors is reduced power of the test. The solution is implementation of Design of Experiment principals tuned for each test.Another feature of the man in the loop learning. Mr. Smith has indicated that he has used blocking as an effective tool to normalize these effects. Learning also has other implications. In the simulation world, participants who die effectively live to fight another day (trial). They therefore have the ability to teach other participants lessons learned from previous fatal mistakes. The test director must therefore circumvent lessons learned as a result of fatal mistakes.One option available to the designer of the experiment is to extract all participants after they have 'died'. The drawback to this solution lies in the potentially limited number of qualified participants. Alternatively we can place a gag order on all dead participants. This solution is not perfect in that dead participants still leam from fatal mistakes. However, at least their lessons are not distributed to participants. Another aspect to the learning phenomena revolves about innovative technologies. Consider the problem posed when introducing Thrust Vectoring technology to a pilot accustom to today's technology. Even if the sim-pilot is provided a substantial learning period he will in all likeliness be deficient when compared to his year 2000 peer. In this case the analysis should shy away from estimation of top level utility and isolate on meh-ics which quantify the mechanics of new technology.Without a doubt the importance of a more complete and representative Operation Research tool cannot be discounted. However, we feel it is imperative that all those involved recognize the associated limitations therein. We therefore advocate careful configuration and design of die c...