Data Organisation and Process Design Based on Functional Modularity for a Standard Production Process

Abstract

We propose to use the principles of functional modularity to cope with the essential complexity of statistical production processes. Moving up in the direction of international statistical production standards (GSBPM and GSIM), data organisation and process design under a combination of object-oriented and functional computing paradigms are proposed. The former comprises a standardised key-value pair abstract data model where keys are constructed by means of the structural statistical metadata of the production system. The latter makes extensive use of the principles of functional modularity (modularity, data abstraction, hierarchy, and layering) to design production steps. We provide a proof of concept focusing on an optimisation approach to selective editing applied to real survey data in standard production conditions at the Spanish National Statistics Institute. Several R packages have been prototyped implementing these ideas. We also share diverse aspects arising from the practicalities of the implementation.

An end-to-end statistical process with mobile network data for official statistics

“…Finally, we mention two points. On the one hand, random variables N rt are defined recursively in the time index t, so that once we have computed the probability distribution at time t 0 , then we can use (41) to compute the probability distribution at later times t > t 0 . On the other hand, Monte Carlo techniques should be again used to build these probability distributions.…”

“…We make a proposal for an end-to-end statistical process going from the raw telco data generated at the mobile telecommunication networks to the final target population count estimates. The proposal follows the principles of functional modularity adapted to statistical production [41] focusing on input and output data as well as the throughput of each production step. The proposal so far focuses on a single-MNO scenario.…”

Answering Current Challenges of and Changes in Producing Official Time Use Statistics Using the Data Collection Platform MOTUS

“…In addition, these processes must remain standardized for reasons of comparability, yet flexible and agile enough to meet (country) specific needs and allow statistics to be disseminated quickly. At the same time, these processes must not compromise on the quality and reliability of the collected data (Salgado et al 2018;Stodden 2014).…”