Robust Visual Mining of Data with Error Information

Sun, Jianyong; Kabán, Ata; Raychaudhury, Somak

doi:10.1007/978-3-540-74976-9_60

Cited by 4 publications

(3 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, the experimental results in [5] and [42] favor the tree-structured factorization over the full factorization…”

Section: B Tree-structured Variational Algorithmmentioning

confidence: 99%

“…It has been adopted in various applications, such as clustering [5], [11], [36], [43], visualization [42], [47], robust projectionsrobust probabilistic principal component analysis [52] and robust canonical correlation analysis (CCA) [3] and factor analyzers [4], [12], [30], in signal processing [10], sequential data modeling [9], image processing [8], [46], to name just a few. The t-distribution has heavy tails, hence it gives non-zero probability to observations that are far away from the bulk of the density.…”

Section: A Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A Fast Algorithm for Robust Mixtures in the Presence of Measurement Errors

Sun

Kabán

2010

IEEE Trans. Neural Netw.

Self Cite

View full text Add to dashboard Cite

Abstract-In experimental and observational sciences, detecting atypical, peculiar data from large sets of measurements has the potential of highlighting candidates of interesting new types of objects that deserve more detailed domain-specific followup study. However, measurement data is nearly never free of measurement errors. These errors can generate false outliers that are not truly interesting. Although many approaches exist for finding outliers, they have no means to tell to what extent the peculiarity is not simply due to measurement errors. To address this issue, we have developed a model-based approach to infer genuine outliers from multivariate data sets when measurement error information is available. This is based on a probabilistic mixture of hierarchical density models, in which parameter estimation is made feasible by a tree-structured variational expectationmaximization algorithm. Here, we further develop an algorithmic enhancement to address the scalability of this approach, in order to make it applicable to large data sets, via a K-dimensionaltree based partitioning of the variational posterior assignments. This creates a non-trivial tradeoff between a more detailed noise model to enhance the detection accuracy, and the coarsened posterior representation to obtain computational speedup. Hence, we conduct extensive experimental validation to study the accuracy/speed tradeoffs achievable in a variety of data conditions. We find that, at low-to-moderate error levels, a speedup factor that is at least linear in the number of data points can be achieved without significantly sacrificing the detection accuracy. The benefits of including measurement error information into the modeling is evident in all situations, and the gain roughly recovers the loss incurred by the speedup procedure in large error conditions. We analyze and discuss in detail the characteristics of our algorithm based on results obtained on appropriately designed synthetic data experiments, and we also demonstrate its working in a real application example.Index Terms-K-dimensional (KD)-tree, measurement errors, outlier detection, robust mixture modeling, variational expectation-maximization (EM) algorithm.

show abstract

“…Moreover, the experimental results in [5] and [42] favor the tree-structured factorization over the full factorization…”

Section: B Tree-structured Variational Algorithmmentioning

confidence: 99%

Section: A Related Workmentioning

confidence: 99%

A Fast Algorithm for Robust Mixtures in the Presence of Measurement Errors

Sun

Kabán

2010

IEEE Trans. Neural Netw.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In particular, visualization may be used for outlier detection, which highlights surprises in the data, i.e. data instances that do not comply with the general behavior or model of the data (Sun et al, 2007). In addition, the user is aided in selecting the appropriate data through a visual interface.…”

Section: Data and Information Visualization Data Visualizationmentioning

confidence: 99%

Visual Data Mining from Visualization to Visual Information Mining

Viktor¹,

Paquet²

Encyclopedia of Data Warehousing and Mining, Second Edition

View full text Add to dashboard Cite

INTRODUCTIONThe current explosion of data and information, which are mainly caused by the continuous adoption of data warehouses and the extensive use of the Internet and its related technologies, has increased the urgent need for the development of techniques for intelligent data analysis. Data mining, which concerns the discovery and extraction of knowledge chunks from large data repositories, addresses this need. Data mining automates the discovery of hidden patterns and relationships that may not always be obvious. Data mining tools include classification techniques (such as decision trees, rule induction programs and neural networks) (Kou et al., 2007); clustering algorithms and association rule approaches, amongst others.Data mining has been fruitfully used in many of domains, including marketing, medicine, finance, engineering and bioinformatics. There still are, however, a number of factors that militate against the widespread adoption and use of this new technology. This is mainly due to the fact that the results of many data mining techniques are often difficult to understand. For example, the results of a data mining effort producing 300 pages of rules will be difficult to analyze. The visual representation of the knowledge embedded in such rules will help to heighten the comprehensibility of the results. The visualization of the data itself, as well as the data mining process should go a long way towards increasing the user's understanding of and faith in the data mining process. That is, data and information visualization provide users with the ability to obtain new insights into the knowledge, as discovered from large repositories. This paper describes a number of important visual data mining issues and introduces techniques employed to improve the understandability of the results of data mining. Firstly, the visualization of data prior to, and during, data mining is addressed. Through data visualization, the quality of the data can be assessed throughout the knowledge discovery process, which includes data preprocessing, data mining and reporting. We also discuss information visualization, i.e. how the knowledge, as discovered by a data mining tool, may be visualized throughout the data mining process. This aspect includes visualization of the results of data mining as well as the learning process. In addition, the paper shows how virtual reality and collaborative virtual environments may be used to obtain an immersive perspective of the data and the data mining process as well as how visual data mining can be used to directly mine functionality with specific applications in the emerging field of proteomics. BACKGROUND

show abstract