A Test Detecting the Outliers for Continuous Distributions Based on the Cumulative Distribution Function of the Data Being Tested

Abstract

One of the pillars of experimental science is sampling. Based on the analysis of samples, estimations for populations are made. There is an entire science based on sampling. Distribution of the population, of the sample, and the connection among those two (including sampling distribution) provides rich information for any estimation to be made. Distributions are split into two main groups: continuous and discrete. The present study applies to continuous distributions. One of the challenges of sampling is its accuracy, or, in other words, how representative the sample is of the population from which it was drawn. To answer this question, a series of statistics have been developed to measure the agreement between the theoretical (the population) and observed (the sample) distributions. Another challenge, connected to this, is the presence of outliers - regarded here as observations wrongly collected, that is, not belonging to the population subjected to study. To detect outliers, a series of tests have been proposed, but mainly for normal (Gauss) distributions—the most frequently encountered distribution. The present study proposes a statistic (and a test) intended to be used for any continuous distribution to detect outliers by constructing the confidence interval for the extreme value in the sample, at a certain (preselected) risk of being in error, and depending on the sample size. The proposed statistic is operational for known distributions (with a known probability density function) and is also dependent on the statistical parameters of the population—here it is discussed in connection with estimating those parameters by the maximum likelihood estimation method operating on a uniform U(0,1) continuous symmetrical distribution.

Detecting Extreme Values with Order Statistics in Samples from Continuous Distributions

“…A method of constructing the observed distribution of the g1 statistic, Equation (11), has already been reported elsewhere [15]. A method of constructing the observed distribution of the Anderson-Darling (AD) statistic, Equation (9), has already been reported elsewhere [17]; the method for constructing the observed distribution of any OS via Monte Carlo (MC) simulation, Equations (5)- (12), is described here and it is used for TS, Equation (12).…”

“…In this process of estimation, there is an intrinsic variability that cannot be ascertained by one statistic alone. This is the reason that calculating the risk of being in error from a battery of statistics is necessary, Equation (15).…”

“…In the following case studies, the values of the sample statistics were calculated with Equations (5)-(10) (AD, KS, CM, KV, WU, H1; see also Figure 10), while the risks of being in error-associated with the values of sample statistics (α Statistic for those)-were calculated with the program developed and posted online available at http://l.academicdirect.org/Statistics/tests. The g1 Statistic (Equation (11)) and α g1 were calculated as given in [15], while the TS Statistic (Equation (12)) was calculated with Algorithm 4. For FCS and α FCS , Equation (15) was used.…”

“…Recent uses of those statistics include [11] (CM), [12] (WU), [13] (KS), [14] (AD), and [15] (H1). Any of the above given test statistics are to be used, providing a risk of being in error for the assumption (or a likelihood to observe) that the sample ({x 1 , ..., x n }) was drawn from the population (X).…”

“…The other alternative in assessing the quality of sampling refers to an individual observation in the sample, specifically the less likely one (having associated q 1 or q n with the notations given in Equation (4)). The test statistic is g1 [15], given in Equation (11).…”