Model-based exception mining for object-relational data

Abstract

This paper is based on a previous publication [29]. Our work extends exception mining and outlier detection to the case of object-relational data. Object-relational data represent a complex heterogeneous network [12], which comprises objects of different types, links among these objects, also of different types, and attributes of these links. This special structure prohibits a direct vectorial data representation. We follow the well-established Exceptional Model Mining framework, which leverages machine learning models for exception mining: A object is exceptional to the extent that a model learned for the object data differs from a model learned for the general population. Exceptional objects can be viewed as outliers. We apply stateof-the-art probabilistic modelling techniques for object-relational data that construct a graphical model (Bayesian network), which compactly represents probabilistic associations in the data. A new metric, derived from the learned object-relational model, quantifies the extent to which the individual association pattern of a potential outlier deviates from that of the whole population. The metric is based on the likelihood ratio of two parameter vectors: One that represents the population associations, and another that represents the individual associations. Our method is validated on synthetic datasets and on real-world data sets about soccer matches and movies. Compared to baseline methods, our novel transformed likelihood ratio achieved the best detection accuracy on all datasets.

TabReformer: Unsupervised Representation Learning for Erroneous Data Detection

“…Existing research [4][5][6][7] applies data modeling approaches to detect outliers in numerical data, e.g., Gaussian mixture models [4] or histogram modeling [5]. Moreover, recent research has applied machine learning techniques, such as unsupervised learning [6] and active learning [7], to detect outliers in relational databases. For example, Riahi and Schulte [6] propose a technique to learn a model for outlier detection using Bayesian networks.…”

“…Moreover, recent research has applied machine learning techniques, such as unsupervised learning [6] and active learning [7], to detect outliers in relational databases. For example, Riahi and Schulte [6] propose a technique to learn a model for outlier detection using Bayesian networks. The method integrates exception mining with statistical-relational learning to detect outliers in relational data.…”

“…As a result, there have been numerous studies to investigate automating the process of detecting erroneous data. A lot of research is targeted to handle outlier detection [4][5][6][7], rule violations [8,9], and duplicate data detection [10]. Rule-based systems [8,9] count on the identification of a set of data quality rules using integrity constraints [11] to specify functional dependencies or other constraints that may define data quality in the given domain.…”

A Survey of Research Progress and Hot Front of Natural Gas Load Forecasting From Technical Perspective

“…Literature [60] uses a robust extreme learning machine (ELM) algorithm to perform regression analysis on load data. Literature [ 61 ] adopts nested hierarchical Dirichlet process for abnormal data, and proposes an unsupervised learning method based on Bayesian model; Literature [62] proposes a data anomaly detection method based on exceptional model mining (EMM) framework for object-relational data. However, on the whole, the research on data processing of natural gas load forecasting is still deficient, which needs a further research.…”