Recommendation of Imputing Value for Sensor Data based on Programming by Example

Abstract

Large volumes of data are typically used during analyses. Data preprocessing, which involves detecting outliers, handling missing data, data formatting, integration, and normalization, is essential for achieving accurate results. Many tools and methods are available for reducing preprocessing time. However, most analysts face difficulties when using them. This paper proposes a method for handling outliers and missing data, called Automated PRE-Processing for Sensor Data (APREP-S). For reducing analysis resources, we combine programming by example and machine learning via Bayesian inference, inputting human knowledge to APREP-S as an example and calculating a proper proportion by machine learning via Bayesian inference. We also define k-Shape as the calculation of the rate of similarity of time-series data. In evaluation, we use sensor data of temperature and humidity and compare the sum of the square of the errors of four methods, between original data and outputs of each methods, (1) APREP-S, (2) mean of the entire data, (3) mean of the around-the-target imputation data, and (4) spline interpolation. It is verified that APREP-S is a more suitable method for humidity data than temperature data. preprocessing method. we consider the reason is that humidity data have more changing points.

Dynamic exergy analysis: From industrial data to exergy flows

“…While only one variable was missing half the data that was filled with linear regression, real data might have randomly distributed missing data for several variables. In that case, more sophisticated machine‐learning approaches are excellent for imputing missing data (Nagashima & Kato, 2020; Gruenwald et al., 2007).…”

Flexible Imputation Method for Sensor Data based on Programming by Example: APREP-S

“…We previously proposed "APREP-S," which conducts automatic pre-processing by integrating human knowledge with machine learning via the PBE approach [3], [4]. In this study, we enhance the previous APREP-S and propose a workflow to select site-specific features at each project site, and use clustering as the classification method.…”

“…This led us to previously propose APREP-S (Automated Preprocessing for sensor data) [3], [4] to reduce the resources that need to be devoted to pre-processing. This is a method that integrates machine learning based on Bayesian inference with human 1 Tokyo Woman's Christian University, Suginami, Tokyo 167-8585, Japan a) h18m001@cis.twcu.ac.jp b) yuka@lab.twcu.ac.jp knowledge using programming by example (PBE) approach -details of the proposed APREP-S are provided in Section 3, and PBE approach is explained in Section 1.2.…”