Prioritization‐based subsampling quality assessment methodology for mobility‐related information systems

Abstract

Mobility‐related information systems, such as on‐street parking information (OSPI) systems have become more popular in the original equipment manufacturer (OEM) industry over the last decade. However, there is a lack of methods to assess their quality at a large scale. This paper introduces a data‐driven methodology to measure the true quality by fleet data prioritization‐based subsampling strategies (PSSs). It is applied to the use case of OSPI using parking events (PE), but is applicable to other mobility‐related information systems utilizing their respective fleet data. PSSs are defined based on neighbourhoods and time periods. Each PSS generates a unique set of spatio‐temporally important areas at different quadkey zoom levels over 168 week‐hours, called slices. The importance weight in each slice depends on the volume of PE within them. The algorithm for each PSS automatically selects important areas and time frames that are vital to be observed. Sample prediction models are used for the benefits assessment of the methodology by comparing it against non‐prioritized randomized selection of ground truth. It is proven that the methodology can lessen the effort of ground truth collection, while maintaining the amount of information necessary to assess the true quality of a prediction model.

Development of a Data-Driven On-Street Parking Information System Using Enhanced Parking Features

“…For this dataset, the sparse data collection strategy (i.e., where, when, and how much data) was beyond the control of the authors. In the validation phase and the final scoring phase, a prioritizationbased quality assessment [43] is used to adjust the scores depending on the amount of parking events that occurred in each spatio-temporal cell. This helps eliminate unimportant hours.…”

“…Hours that have no ground truth data are excluded from evaluation and are a limitation of this study. Nonetheless, these hours are also considered unimportant hours in Munich based on the study of Gomari et al [43]. The selected metric for analysis in this study was the Mean Squared Error (MSE) as described below, which is also called the Brier Loss for cases with binary outcomes: (1) where p is the predicted probability outcome, o is the observation at instance t (0 means there was no available parking spot, 1 means there was at least one available spot), and N is the total number of instances.…”

“…For further insights, additional metric scores are calculated using the Mean Absolute Error (MAE) and the Root Mean Square Error (RMSE) which can be found in the Appendix. Additionally, given the BMW user-centric system in this study, the proposed prioritization-based quality assessment of [43] is implemented. This method essentially adjusts the scores by taking the weighted sum of the scores of each quadkey at zoom level 14, denoated as KPIp.…”

“…The calculated prioritization-based MSE scores [43] are illustrated in Table 3. The worst performing model scores are achieved at ED1 when only temporal features are considered.…”

“…And as seen, the model can predict in line with the availability patterns. If there are discrepancies, these are considered in the prioritizationbased scores, which adjusts in accordance with spatiotemporal importance [43].…”