Practical metrics for performance evaluation of estimation algorithms are discussed. A variety of metrics useful for evaluating various aspects of the performance of an estimation algorithm is introduced and justified. They can be classified in two different ways: 1) absolute error measures (without a reference), relative error measures (with a reference), or frequency counts (of some events), and 2) optimistic (i.e., how good the performance is), pessimistic (i.e., how bad the performance is), or balanced (neither optimistic nor pessimistic). Pros and cons of these metrics and the widely-used RMS error are explained. The paper advocates replacing the RMS error in many cases by a measure called average Euclidean error.Manuscript

No abstract

This paper deals with practical measures for performance evaluation of estimators and filters. Several new measures useful for evaluating various aspects of the performance of an estimator or filter are proposed and justified, including measurement error reduction factors, and success and failure rates. Pros and cons of some widely used measures are explained. In particular, the merits of a measure called average Euclidean error (AEE) over the widely used RMS error is presented and it is advocated that RMS error should be replaced by the AEE in many cases.

Many estimators and filters provide assessments (e.g., MSE matrices) of their own estimation errors. They are, however, obtained based on simplifying assumptions that are not necessarily valid Then the questions are: Are these selfassessments trustable? How trustable are they? We referred to these problems as the credibility of the estimatorslfilters. Solid technical answers to thefirst question are provided in two companion papersfor this conference based on statistical hypothesis testing. Complementary to those, we answer the second question in this paper by proposing afamily of metrics, called noncredibility indices (NCI) and inclination indicators (12), that measure how credible various self-assessments are. We show that the NCI and I2 have many desirable properties and are more appropriate than a bunch ofpossible alternatives and by far superior to a heuristic measure currently in use explicitly or implicitly. We also provide simple numerical examples to illustrate the application ofthe metrics proposed

No abstract