Data Pre-processing Techniques in the Regional Emission's Load Profiles Case

Abstract

Different data sources, data types and platforms are involved in modelling emissions load profiles. In our case, we model emission load profiles at the regional or city level. However, we found missing values, redundancy and inconsistency in the datasets, and in most cases data preprocessing is unavoidable. Data pre-processing converts the data into a clean and tidy dataset for the subsequent modelling steps or statistical analyses. Therefore, some common techniques for data pre-processing such as cleaning, transformation, integration, reduction and some terms in data mining such as filtering and selection have been applied in our case study. We usually do the data pre-processing of moderate problems and a small amount of data using a spreadsheet application, whereas we use the programming language to do the more complex and big data size tasks. As a result, it has been found that understanding the nature of our data collection, the data flow process and the desired output comprehensively is the key for efficiency in data pre-processing. The applied techniques have helped us to provide the proper input for modelling the regional emission load profile efficiently.

Synthesising Residential Electricity Load Profiles at the City Level Using a Weighted Proportion (Wepro) Model

“…In data collection and pre-processing phases, it is important to study the latter comprehensively, as it can improve data quality and the accuracy of the result [52]. Data corruption, missing values and outliers are the commonest problems in data-processing [52,53]. In general, there are four tasks in data pre-processing: cleaning, transformation, integration and reduction [52,54,55].…”

“…Data corruption, missing values and outliers are the commonest problems in data-processing [52,53]. In general, there are four tasks in data pre-processing: cleaning, transformation, integration and reduction [52,54,55]. Table 2 summarises the common problems of data pre-processing tasks and their solutions:…”

“…In this work, the data collected from amsterdam (The Netherlands), are in the form of a panel dataset, which is a cross-sectional data sample at specific point in time [52]. The panel dataset consists of information on age groups, the gender structure of each age group, the labour force, national holidays, solar irradiation and temperature datasets.…”

“…Table 2. Data pre-processing: The tasks perform in data pre-processing include their common problems and solutions of these problems [52,55].…”

A Systematicity Review on Residential Electricity Load-Shifting at the Appliance Level

“…As a consequence, load-shifting forms part of our research on DSM in the residential sector (Figure 1). In our previous works on synthesising residential electricity load profiles [28,[35][36][37], we found that appliances are mainly used to synthesize domestic load profiles. In accordance with what was revealed in [15], there is a need to incorporate the consumption patterns of electrical appliances into DSM models.…”