Key Research Challenges of Incorporating Official Statistics with Big Data Analytics and Lean Six Sigma

Abstract

The amount of created and captured data has been increasing rapidly. Since 2018, the amount of data generated and collected surpasses 90% of the data produced in the previous years. To meet the 2030 United Nations Economic Commission for Europe (UNECE) statistical agenda, National Statistics Organizations (NSOs) must meet the required amount of data demanded through the modernization of statistics. Relevant studies showed the implementation of official statistics with lean six sigma, resulting in a 5% operational budget reduction. Moreover, integrating official statistics with big data analytics produced more efficient and timelier statistics. However, in this industry 4.0 era, there is a lack of practices in the integration of big data tools and technique with the official statistics. The reason might be the uncertainty on the challenges and potential solutions. Therefore, this paper systematically reviews and analyses the critical challenges and potential solutions for the integration of big data analytics and lean six sigma into the processes of generating official statistics. A comprehensive and systematic literature review was conducted. Findings showed that the challenges of integrating big data are lack of proper infrastructure, data quality and data privacy. On the other hand, challenges of integrating lean six sigma are lack of management support, lack of common goal and the hierarchical organization structure. This paper will contribute to future work by using the identified challenges of integrating official statistics with big data analytics and lean six sigma to develop and propose a framework which overcomes the critical challenges and maximize the benefits.