Automatic Machine Learning-Based OLAP Measure Detection for Tabular Data

Abstract

zhang2011handling combining naive bayes and em, li2015trip combining dependencies and web information, zhang2019learning combining KNN and Regression, song2020imputing combining KNN and likelihood maximization, chhabra2018missing combining k-means and association rules, aydilek2013hybrid combining fuzzy c-means, support vector regression and genetic algorithm, etc. There are also hybrid approaches which use techniques of the same category like latifi2012evaluation combining knn and random forest and wang2017cosset combining crowdsourcing and knowledge base, etc. Nowadays, it is difficult for companies and organisations without Business Intelligence (BI) experts to carry out data analyses. Existing automatic data warehouse design methods cannot treat with tabular data commonly defined without schema. Dimensions and hierarchies can still be deduced by detecting functional dependencies, but the detection of measures remains a challenge. To solve this issue, we propose a machine learning-based method to detect measures by defining three categories of features for numerical columns. The method is tested on real-world datasets and with various machine learning algorithms, concluding that random forest performs best for measure detection.