Detection of outliers

Hadi, Ali S.; Imon, A. H. M. Rahmatullah; Werner, Mark

doi:10.1002/wics.6

Cited by 115 publications

(73 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The term `outlier' refers to values of the response variables which are unusually large or small [3,7]. Since all of the explanatory variables in our case are factors, any abnormal values are likely to be outliers.…”

Section: Methodsmentioning

confidence: 99%

The Outlier Detection for Ordinal Data Using Scalling Technique of Regression Coefficients

Adnan

Sugiarto

2017

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

ABSTRACT. The aims of this study is to detect the outliers by using coefficients of Ordinal Logistic Regression (OLR) for the case of k category responses where the score from 1 (the best) to 8 (the worst). We detect them by using the sum of moduli of the ordinal regression coefficients calculated by jackknife technique. This technique is improved by scalling the regression coefficients to their means. R language has been used on a set of ordinal data from reference distribution. Furthermore, we compare this approach by using studentised residual plots of jackknife technique for ANOVA (Analysis of Variance) and OLR. This study shows that the jackknifing technique along with the proper scaling may lead us to reveal outliers in ordinal regression reasonably well. IntroductionOn the category scale, the panellists are asked to give a score, for an example 1, 2, …, 8 on his opinion about the taste of a certain food. This scale is a discrete number and arranged as an ordinal scale which usually ranges from 1 (the best) to 8 (the worst), or from 'extremely tender' to 'extremely tough' for tenderness of meat, for example. Joo et al. [8] reviewed the definition of fresh meat quality, meat quality traits and variations of meat quality. They also presented the recent knowledge underlying the relationship between fresh meat quality traits and muscle fiber characteristics. In the research, they proposed several potential factors including breed, genotype, sex, hormone, growth performance, diet, muscle location, exercise and ambient temperature that can be used to manipulate muscle fiber characteristics and subsequently meat quality in animals.This problem is a very popular topic on sensory evaluation. A comprehensive review of sensory and consumer science has been conducted by Naes et al. [10]. They also used a 8-category scale for scoring quality attributes. Some researchers omit a central, neutral category to produce a more even distribution of responses. A line or an unstructured scale is a judge response on a solid line from 'little' to 'much' or from 'none' to extremely'.One of the common used response models for ordinal data analysis is the proportional odds model. This model is an extension of logistic model for binary data where their responses are more than two categories [3]. He discussed loglinear models for ordinal variables which have some advantages compared to the model treating all variable as nominal. Whenever the relationship between response and explanatory variables needs to be distinguished then we use the 'logit'model. The model is based on the success probability on response variables, hence, the procedure has to ensure that fitted probabilities are in the range (0,1). Logit model can be found using logit transformation based on logistic function.

show abstract

Section: Methodsmentioning

confidence: 99%

The Outlier Detection for Ordinal Data Using Scalling Technique of Regression Coefficients

Adnan

Sugiarto

2017

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

show abstract

“…Various methods for identifying outliers have been proposed based on di↵erent methodologies, like statistical reasoning (Hadi et al (2009)), distances (Angiulli and Pizzuti (2002); Knorr et al (2000);and Orair et al (2010)), or densities (Breunig et al (2000)); (De Vries et al (2010) and Keller et al (2012)). But the issue is not completely solved, and in some methodologies, such as causal inference, this issue may become crucial.…”

Section: A Brief Review Of the Literaturementioning

confidence: 99%

Outliers in Semi-Parametric Estimation of Treatment Effects

2017

View full text Add to dashboard Cite

Average treatment e↵ects estimands can present significant bias under the presence of outliers. Moreover, outliers can be particularly hard to detect, creating bias and inconsistency in the semi-parametric ATE estimads. In this paper, we use Monte Carlo simulations to demonstrate that semi-parametric methods, such as matching, are biased in the presence of outliers. Bad and good leverage points outliers are considered. The bias arises because bad leverage points completely change the distribution of the metrics used to define counterfactuals. Whereas good leverage points increase the chance of breaking the common support condition and distort the balance of the covariates and which may push practitioners to misspecify the propensity score. We provide some clues to diagnose the presence of outliers and propose a reweighting estimator that is robust against outliers based on the Stahel-Donoho multivariate estimator of scale and location. An application of this estimator to LaLonde (1986) data allows us to explain the Dehejia and Wahba (2002) and Smith and Todd (2005) debate on the inability of matching estimators to deal with the evaluation problem.JEL: C21, C14, C52, C13

show abstract

“…Here, swamping means a true positive becomes a false negative, while masking means a true negative becomes a false positive. These two terminologies are often used for outlier detection (Ben-Gal 2005;Hadi et al 2009): swamping means some non-outliers are identified as outliers, while masking means some outliers are not identified; outliers mask themselves by swamping some non-outliers. We borrow them here to characterize the two results of (P1) (P2) in MB discovery because of their similar behaviors in "masking" themselves and "swamping" others.…”

Section: Introductionmentioning

confidence: 99%

Swamping and masking in Markov boundary discovery

Liu

2016

Mach Learn

View full text Add to dashboard Cite

This paper considers the problems of swamping and masking in Markov boundary discovery for a target variable. There are two potential reasons for swamping and masking: one is incorrectness of some conditional independence (CI) tests, and the other is violation of local composition. First, we explain why the incorrectness of CI tests may lead to swamping and masking, analyze how to reduce the incorrectness of CI tests, and build an algorithm called LRH under local composition. For convenience, we integrate the two existing algorithms, IAMB and KIAMB, and our LRH into an algorithmic framework called LCMB. Second, since LCMB may prematurely stop searching if local composition is violated, a theoretical improvement on LCMB is made as follows: we analyze how to resume the stopped search of LCMB, construct a corresponding algorithmic framework called WLCMB, and show that its correctness only needs a more relaxed condition than that of LCMB. Finally, we apply LCMB and WLCMB to a number of Bayesian networks. The experimental results reveal that LRH is much more efficient than the existing two LCMB algorithms and that WLCMB can further improve LCMB.

show abstract

Detection of outliers

Cited by 115 publications

References 62 publications

The Outlier Detection for Ordinal Data Using Scalling Technique of Regression Coefficients

The Outlier Detection for Ordinal Data Using Scalling Technique of Regression Coefficients

Outliers in Semi-Parametric Estimation of Treatment Effects

Swamping and masking in Markov boundary discovery

Contact Info

Product

Resources

About