A comparison of performance measures via online search

Abstract

Though competitive analysis has been a very useful performance measure for the quality of online algorithms, it is recognized that it sometimes fails to distinguish between algorithms of different quality in practice. A number of alternative measures have been proposed, but, with a few exceptions, these have generally been applied only to the online problem they were developed in connection with. Recently, a systematic study of performance measures for online algorithms was initiated [Boyar, Irani, Larsen: Eleventh International Algorithms and Data Structures Symposium 2009], first focusing on a simple server problem. We continue this work by studying a fundamentally different online problem, online search, and the Reservation Price Policies in particular. The purpose of this line of work is to learn more about the applicability of various performance measures in different situations and the properties that the different measures emphasize. We investigate the following analysis techniques: Competitive, Relative Worst Order, Bijective, Average, Relative Interval, Random Order, and Max/Max. In addition to drawing conclusions on this work, we also investigate the measures' sensitivity to integral vs. real-valued domains, and as a part of this work, generalize some of the known performance measures. Finally, we have established the first optimality proof for Relative Interval Analysis.

Online Bin Covering: Expectations vs. Guarantees

“…Up until that paper, most performance measures were introduced for a specific problem to overcome the limitations of competitive analysis. In [8], comparisons of performance measures different from competitive analysis were initiated, and this line of work has been continued in [6,9,7], among others. Our results supplement results in [11], showing that no deterministic algorithm for the bin covering problem can be better than 1 2 -competitive and giving an asymptotically optimal algorithm for the case of items being uniformly distributed on (0, 1).…”

Relative Worst-Order Analysis: A Survey

“…Further systematic comparisons of performance measures were made in [26] and [27], again comparing algorithms on relatively simple problems. The paper [26] considered competitive analysis, bijective analysis, average analysis, relative interval analysis, random-order analysis, and RWOA. There were differences between the measures, but the most clear conclusions were that bijective analysis found all algorithms incomparable and average analysis preferred an intuitively poorer algorithm.…”

Online algorithms for conversion problems: A survey

“…Further, online conversion is mainly concerned with competitive analysis against the true optimum (OPT ). Hence, in this survey we omit the various alternatives to the competitive ratio proposed in the literature, and refer to [12,13].…”

“…Boyar et al [13] study the applicability as well as the properties of the RPP considering various performance measures. They show that different measures lead to different optimal reservation prices, e.g., average-case analysis indicates that the maximal possible RP is the best, namely q * = M.…”

“…This approach is a hybrid of the Bayesian approach and competitive analysis. An open question is to systematically study the alternatives to competitive analysis in comparison with each other on the same online problem [13].…”