Discounting older data

Abstract

This article describes two general methods for discounting older data in the realtime analysis of a data stream. In the first method, the distribution of a data stream is estimated by a series of orthogonal basis functions, and the coefficients of this estimate are updated as new data arrive by combining windowing and exponential smoothing techniques. The second method involves sequential hypothesis testing. When new data arrive, test significance level is adjusted by alpha-investing, which raises or reduces the significance level of subsequent hypothesis tests on the basis of whether the previous hypothesis test rejects or fails to reject the null hypothesis. Both these methods are nonparametric in nature. 