DistAD: A Distributed Generic Anomaly Detection Framework over Large KGs

Abstract

The last decades have witnessed significant advancements in terms of data generation, management, and maintenance especially in the area of data lakes, and heterogeneous data. This has resulted in vast amounts of data becoming available in a variety of forms and formats including RDF. As RDF data have been created by liberal curation methods (e.g. crowdsourcing and automatic extraction tools with limited restriction and cross-validation on input data), they are prone to various kinds of errors that can be hidden in different dimensions (i.e. subject, predicate, and object level). Detecting those errors not only improves the KGs quality but also makes it possible to detect anomalous events in the data which can be used for subsequent analysis. In this paper, we present DistAD, a generic, scalable, and distributed framework for anomaly detection on large RDF knowledge graphs. DistAD provides a great granularity for the end-users to select from a vast number of different algorithms, methods, and (hyper-)parameters to detect outliers. The proposed framework is fully open-source, well-documented, and fully integrated into the active community project Semantic Analytics Stack (SANSA). The experiments on real-world use cases disclose that the framework is not only able to handle huge RDF data but also able to successfully detect hidden anomalies/outliers in KGs.