The Lorenz Curve for Model Assessment in Exponential Order Statistic Models

Osborne, Jason A.; Severini, Thomas A.

doi:10.1080/00949650211423

“…exponential distribution, the subjective criteria are based on less comprehensive evaluations of the distribution statistics and are described below. A separate study is required to re-evaluate these distributions in terms of 'goodness-of-fit' to more complex distributions such as compound exponential, gamma, beta, zeta, Weibull, Pareto, and other distributions (see for example [32,39,40]). …”

Section: Distributionsmentioning

confidence: 99%

On SAT instance classes and a method for reliable performance experiments with SAT solvers

Brglez¹,

Li²,

Stallmann³

2004

Ann Math Artif Intell

View full text Add to dashboard Cite

A recent series of experiments with a group of state-of-the-art SAT solvers and several well-defined classes of problem instances reports statistically significant performance variability for the solvers. A systematic analysis of the observed performance data, all openly archived on the Web, reveals distributions which we classify into three broad categories: (1) readily characterized with a simple χ 2 -test, (2) requiring more in-depth analysis by a statistician, (3) incomplete, due to time-out limit reached by specific solvers. The first category includes two well-known distributions: normal and exponential; we use simple first-order criteria to decide the second category and label the distributions as near-normal, near-exponential and heavy-tail. We expect that good models for some if not most of these may be found with parameters that fit either generalized gamma, Weibull, or Pareto distributions. Our experiments show that most SAT solvers exhibit either normal or exponential distribution of execution time (runtime) on many equivalence classes of problem instances. This finding suggests that the basic mathematical framework for these experiments may well be the same as the one used to test the reliability or lifetime of hardware components such as lightbulbs, A/C units, etc. A batch of N replicated hardware components represents an equivalence class of N problem instances in SAT, a controlled operating environment A represents a SAT solver A, and the survival function R A (x) (where x represents the lifetime) is the complement of the solvability function S A (x) = 1 − R A (x) where x may represent runtime, implications, backtracks, etc. As demonstrated in the paper, a set of unrelated benchmarks or randomly generated SAT instances available today cannot measure the performance of SAT solvers reliably -there is no control on their 'hardness'. However, equivalence class instances as defined in this paper are, in effect, replicated instances of a specific reference instance. The proposed method not only provides a common platform for a systematic study and a reliable improvement of deterministic and stochastic SAT solvers alike but also supports the introduction and validation of new problem instance classes.

show abstract

“…exponential distribution, the subjective criteria are based on less comprehensive evaluations of the distribution statistics and are described below. A separate study is required to re-evaluate these distributions in terms of 'goodness-of-fit' to more complex distributions such as compound exponential, gamma, beta, zeta, Weibull, Pareto, and other distributions (see for example [32,39,40]). …”

Section: Distributionsmentioning

confidence: 99%

On SAT Instance Classes and a Method for Reliable Performance Experiments with SAT Solvers

Brglez

¹

,

Li

²

,

Stallmann

³

2005

Annals of Mathematics and Artificial Intelligence

View full text Add to dashboard Cite

A recent series of experiments with a group of state-of-the-art SAT solvers and several well-defined classes of problem instances reports statistically significant performance variability for the solvers. A systematic analysis of the observed performance data, all openly archived on the Web, reveals distributions which we classify into three broad categories: (1) readily characterized with a simple χ 2 -test, (2) requiring more in-depth analysis by a statistician, (3) incomplete, due to time-out limit reached by specific solvers. The first category includes two well-known distributions: normal and exponential; we use simple first-order criteria to decide the second category and label the distributions as near-normal, near-exponential and heavy-tail. We expect that good models for some if not most of these may be found with parameters that fit either generalized gamma, Weibull, or Pareto distributions. Our experiments show that most SAT solvers exhibit either normal or exponential distribution of execution time (runtime) on many equivalence classes of problem instances. This finding suggests that the basic mathematical framework for these experiments may well be the same as the one used to test the reliability or lifetime of hardware components such as lightbulbs, A/C units, etc. A batch of N replicated hardware components represents an equivalence class of N problem instances in SAT, a controlled operating environment A represents a SAT solver A, and the survival function R A (x) (where x represents the lifetime) is the complement of the solvability function S A (x) = 1 − R A (x) where x may represent runtime, implications, backtracks, etc. As demonstrated in the paper, a set of unrelated benchmarks or randomly generated SAT instances available today cannot measure the performance of SAT solvers reliably -there is no control on their 'hardness'. However, equivalence class instances as defined in this paper are, in effect, replicated instances of a specific reference instance. The proposed method not only provides a common platform for a systematic study and a reliable improvement of deterministic and stochastic SAT solvers alike but also supports the introduction and validation of new problem instance classes.

show abstract