Empirically Estimable Classification Bounds Based on a Nonparametric Divergence Measure

Abstract

Information divergence functions play a critical role in statistics and information theory. In this paper we show that a non-parametric f-divergence measure can be used to provide improved bounds on the minimum binary classification probability of error for the case when the training and test data are drawn from the same distribution and for the case where there exists some mismatch between training and test distributions. We confirm the theoretical results by designing feature selection algorithms using the criteria from these bounds and by evaluating the algorithms on a series of pathological speech classification tasks.

Ranking the parameters of deep neural networks using the fisher information

“…More recently, Berisha et al have introduced the D αdivergence between distributions [19,16], defined as…”

“…The FR test statistic, C(X p , X q ), is constructed by first generating a Euclidean minimal spanning tree (MST) on the concatenated data set, X p ∪ X q , and then counting the number of edges connecting a data point from p to a data point from q. In [19], the authors define an asymptotically consistent estimator for (4) in terms of the FR test statistic. In other words, the quantity in (4) can be estimated directly from the data sampled from p and q without parametric assumptions on these distributions.…”

Introducing Information Measures via Inference [Lecture Notes]

“…This approach allows the definition of general classes of information measures, including as special cases Shannon's entropy and KL divergence, in an intuitive way that reveals their operational significance. The variational formulations that define the information measures as optimal inference problems can be used to derive learning algorithms, such as in [6], as well as estimates of information measures [11], [5].…”

“…The function g can be used to define the relative importance of errors made in favor of one distribution or the other. We note that the merit function (16) can also be formally related to the error probability of binary hypothesis testing [11].…”

An 18 kW solar array research facility for fault detection experiments

“…Common shading and fault conditions will be safely generated in order to build a comprehensive dataset for designing and evaluating monitoring techniques. The use of statistical signal processing algorithms [1,17,18,21], imaging techniques [19] for forecasting solar irradiance [27], and machine learning [20,24,26,28,29].…”