<title>Training-set-based performance measures for data-adaptive decisioning systems</title>

Abstract

Performance measures are derived for data-adaptive hypothesis testing by systems trained on stochastic data. The measures consist of the averaged performance of the system over the ensemble of training sets. The training set-based measures are contrasted with maximum aposteriori probability (MAP) test measures. It is shown that the training set-based and MAP test probabilities are equal if the training set is proportioned according to the prior probabilities of the hypotheses. Applications of training set-based measures are suggested for neural net and training set design.Hypothesis testing by a data-adaptive system, such as a neural net, is fundamentally different from classical hypothesis testing. In the former a representative data set, corresponding to known hypotheses, is used to train the system. System parameters are varied until the system training sethypothesis space mapping best approximates the known map. The assumptions of a sufficiently representative training set and the ability of the system to associate are required to extend the map to arbitrary data [1]. In contrast, classical hypothesis testing derives from an assumed model for the data, often a signal in Gaussian noise, from which optimum tests are defined {2].In this paper performance measures are derived based only on the procedure by which an adaptive system is trained. It is assumed that, if a system is perfectly trained on a representative data set for each hypothesis, an appropriate performance estimate is the averaged performance over the ensemble of training sets. This averaged performance, which is computed in terms of training set size and data distributions, reflects an uncertainty inherent in learning from a finite representation of the data. Of course an exact measure of system performance is obtained by testing the system on an ensemble of independent performance sets. However, in order to predict this performance an exact model of the system mapping must be known. This is difficult for model-based systems in general; but even more difficult for adaptive systems in which the exact mapping is training set-dependent.In the following training set-based performance measures are derived for a data-adaptive system on an arbitrary data-based N-hypothesis test. The nearest neighbor classifier is a typical realization of a training set-based system which has been implemented in artificial neural networks [3,4]. A maximum aposteriori probability (MAP) test is also formulated and represented for a decisioning system with output in [0, 11N• A possible neural net representation of the MAP test contains N output neurons. For a net input x the jth deepest layer neuron literally outputs p(H2Ix)c[0, 1], which is the conditional probability for hypothesis I1, i = 1, . . . , N. This rather stringent condition was obtained in Ref.[5] using a Boltzmann/perceptron net combination to implement the MAP test. It has also been proven that, assuming the training set is sThis work was sponsored by the Department of the Air Force under contract F19628-90-C-0...

Spectral and spatial algorithm architecture for classification of hyperspectral and LIDAR

“…Our motivation for using a back propagation neural net for sensor fusion is its successful performance, as shown by Levine and Khuon in [7], [9], [10], and [11]. In this section we review some of these results, where successful data fusion by Chair and Varshney [5] in terms of probability of detection and probability of false alarms were shown to be achieved similarly by a cascaded neural net by Levine and Khuon [6][7][8][9].…”

“…In this section we review some of these results, where successful data fusion by Chair and Varshney [5] in terms of probability of detection and probability of false alarms were shown to be achieved similarly by a cascaded neural net by Levine and Khuon [6][7][8][9]. The data fusion rule for a binary decision was obtained within the distributed sensor processing architecture.…”

“…We would like to express special gratitude to Brandon Wilson, NGA IBP, for his help in co-registration of LIDAR and HSI data, Thomas Parr, BBN, for his LIDAR/HSI phenomenology analysis, Eric Truslow, MIT Lincoln Laboratory, for his SEM Matlab code and SEM map, Dr. Dimitris Manolakis, MIT Lincoln Laboratory, for his perspectives on neural net sensor fusion algorithms, Dr. Robert Levine, Spectral Science, for his technical contribution [7] in Section 2.3, Alexey Castrodad, NGA IBP, for his technical and structural review of this entire manuscript.…”

Distributed adaptive framework for multispectral/hyperspectral imagery and three-dimensional point cloud fusion

“…Our motivation comes from observing the successful performance of NNSF methods in scenarios relevant to our application; specifically, the successful data fusion by Chair and Varshney 31 was shown to be achieved similarly (in terms of probability of detection and probability of false alarms) by a cascaded neural net by Levine and Khuon. [32][33][34][35][36] Successful performance of neural networks was achieved for nonlinear detection and classification problems for imagery data fusion in this research. A detailed discussion of the FNN for data/ sensor fusion can be found in Refs.…”

Wireless Physical-Layer Identification: Modeling and Validation

“…The performance of the classification/identification can be evaluated by the error probabilities, which can be theoretically calculated through hypothesis testing models. 1) Classification: For N-users classification scenario, we adopt N-hypothesis testing techniques to apply to N genuine users with M N reference fingerprints in the database for each user [23]. The hypothesis H N is that the obtained signal is from the genuine user #N. Then the classification probability can be expressed by P(H i |H j ), i, j = 1, ..., N, i j, which is the probability that the RFF from genuine user # j is classified as the genuine user #i.…”