Loss functions for predicted click‐through rates in auctions for online advertising

Abstract

Abstract. We characterize the optimal loss functions for predicted click-through rates in auctions for online advertising. While standard loss functions such as mean squared error or log likelihood severely penalize large mispredictions while imposing little penalty on smaller mistakes, a loss function reflecting the true economic loss from mispredictions imposes significant penalties for small mispredictions and only slightly larger penalties on large mispredictions. We illustrate that when the model is misspecified, using such a loss function can improve economic efficiency, but the efficiency gain is likely to be small.

Offline Evaluation of Response Prediction in Online Advertising Auctions

“…The regular utility on the other hand tries to estimate the bidder profit but may fail to correctly penalize mispredictions. The expected utility, a metric recently proposed in [5], captures best of both words. This paper demonstrated empirically that indeed the expected utility achieves better correlations with online results.…”

“…And it also well-calibrated in the sense that its expected value is maximized under perfect predictions, pi = E(ai | xi) [5,Theorem 4]. This last property is a direct consequence of the truthfulness of a secondprice auction.…”

“…In a closely related work, [5] analyzes the same expected utility as a loss function to train machine learning models. The idea of introducing a corrupting distribution to make a statistic or a loss function more reliable is also known in the machine learning literature as noise injection [2,6].…”

“…Uniform : it is easy to show (see [5,Theorem 2]) that the expected utility is, up to an irrelevant additive term and multiplicative factor, equal to the squared error, weighted by the value: v 2 (a − p) 2 .…”

“…But these metrics are oblivious to how the probabilities will affect the bidder performance. We propose here an empirical study of a metric recently proposed in [5] that is specifically designed for evaluating a bidder.…”