Data Integration by Combining Big Data and Survey Sample Data for Finite Population Inference

Abstract

Summary The statistical challenges in using big data for making valid statistical inference in the finite population have been well documented in literature. These challenges are due primarily to statistical bias arising from under‐coverage in the big data source to represent the population of interest and measurement errors in the variables available in the data set. By stratifying the population into a big data stratum and a missing data stratum, we can estimate the missing data stratum by using a fully responding probability sample and hence the population as a whole by using a data integration estimator. By expressing the data integration estimator as a regression estimator, we can handle measurement errors in the variables in big data and also in the probability sample. We also propose a fully nonparametric classification method for identifying the overlapping units and develop a bias‐corrected data integration estimator under misclassification errors. Finally, we develop a two‐step regression data integration estimator to deal with measurement errors in the probability sample. An advantage of the approach advocated in this paper is that we do not have to make unrealistic missing‐at‐random assumptions for the methods to work. The proposed method is applied to the real data example using 2015–2016 Australian Agricultural Census data.

Big Data Integration Solutions in Organizations: A Domain-Specific Analysis

“…These data sets contain the layers of raster and geospatial data. Kim and Tam (2020) have proposed a data integration estimator [17]. This is a classification technique with non-parametric and overlapping units which recognizes and corrects misclassification errors.…”

On Making Valid Inferences by Integrating Data from Surveys and Other Sources

“…Since the sizes N B and N C = N − N B are known, a more efficient post-stratified estimator is given by b Kim and Tam (2018) showed that with a simple random sample A, the poststratified estimator achieves a large reduction in the design variance compared to the design unbiased estimator b Y ¼ ∑ i∈A d i y i based only on the probability sample A. In particular, if the sampling fraction f = n/N is small and the population variance σ 2 y and the variance of the population units not belonging to B, denoted…”

“…where N B , N C and Y B are known. The resulting calibration estimator ∑ i ∈ A w i y i is identical to the post-stratified estimator b Y P (Kim and Tam 2018). However, the main advantage of the calibration approach is that it permits the inclusion of other calibration constraints, if available.…”

“…The case of measurement errors only in the probability sample A is more complex and requires a measurement error model. Kim and Tam (2018) also studied calibration estimation in the case of unit nonresponse in the probability sample A, assuming a general response model allowing the probability of response to depend on the study variable. Kim and Tam (2018) gave an interesting application of the above setup in official statistics.…”

“…Kim and Tam (2018) also studied calibration estimation in the case of unit nonresponse in the probability sample A, assuming a general response model allowing the probability of response to depend on the study variable. Kim and Tam (2018) gave an interesting application of the above setup in official statistics. In this application, the non-probability sample (or big data) is the Australian Agricultural Census with 85% response rate and the probability sample is the Rural Environment and Agricultural Commodities Survey.…”