Statistical Generalization of Performance-Related Heuristics for Knowledge-Lean Applications

Abstract

In this paper, we present new results on the automated generalization of performance-related heuristics learned for knowledge-lean applications. By first applying genetics-based learning to learn new heuristics for some small subsets of test cases in a problem space, we study methods to generalize these heuristics to unlearned subdomains of test cases. Our method uses a new statistical metric called probability of win. By assessing the performance of heuristics in a range-independent and distribution-independent manner, we can compare heuristics across problem subdomains in a consistent manner. To illustrate our approach, we show experimental results on generalizing heuristics learned for sequential circuit testing, VLSI cell placement and routing, branch-and-bound search, and blind equalization. We show that generalization can lead to new and robust heuristics that perform better than the original heuristics across test cases of different characteristics.

Performance Evaluation of Search Engines Using Enhanced Vector Space Model

“…Performances of search engines have been compared based on similarity value computed in Table 2 and 3. The statistical metric probability of win (P win ) (Li and Shang, 2000a;2000b;Shang and Li, 2002;Ieumwananonthachai and Wah, 1996) measure statistically how much better (or worse) sample mean of one hypothesis, µ 1 is as compared to other, µ 2 . In hypothesis testing hypothesis {H: µ 1 >µ 2 } is specified without alternative hypothesis and it is evaluated based on sample values.…”

Optimization design of filter banks in subband image coding

“…We tried different values, 1, 0.5, 0.4, 0.3, 0.2, and 0.1, for weight w in (14). The best result was obtained when w = 0:2.…”

“…The nonlinear optimization problem in (14) can be solved by various nonlinear optimization methods such as random search methods, simulated annealing, and evolutionary algorithms. Simulated annealing was used in our experiments.…”

“…Instead of obtaining the best filter bank for each training image, we want a filter bank that performs well on many images. To achieve this goal, symmetric improvement ratio [14] is used to measure how well a filter bank performs across multiple images. Symmetric improvement ratio is a normalization method that avoids anomalies in inconsistent orderings of two hypotheses due to the choice of the baseline hypothesis.…”

“…with performance values A 1 ; ; A m and B 1 ; ; B m respectively on m test cases, the symmetric improvement ratio S is defined as follows [14]:…”