Assessing Response Quality by Using Multivariate Control Charts for Numerical and Categorical Response Quality Indicators

Abstract

When assessing interview response quality to identify potentially low-quality interviews, both numerical and categorical response quality indicators (mixed indicators) are usually available. However, research on how to use them simultaneously is very rare. In the current article, we extend the application of conventional multivariate control charts to include response quality indicators that are of a mixed type. We analyze data from the eighth round of the European Social Survey in Belgium, characterized by six numerical and two categorical response quality indicators. First, we employ a principal component analysis mix procedure (PCA Mix) to transform the mixed quality indicators into principal components. The principal component scores are subsequently used to construct a Hotelling T2 statistic. To deal with the non-multivariate normal nature of the principal component scores obtained from the PCA Mix, a nonparametric bootstrap method is then applied to calculate the control limit for the T2 statistic. Second, we suggest tools to interpret an identified outlier in terms of finding the responsible original indicator(s). Third, we present a cyclic procedure for determining the “in-control” data, by iteratively removing the outliers until the process is considered as being in control. Lastly, we identify the most important indicators that discriminate the outliers from the in-control data. Our results imply that multivariate control charts based on relevant projection tools such as PCA Mix in combination with the bootstrap technique have great potential for use in evaluating interview response quality and identifying outliers.

Identifying Outliers in Response Quality Assessment by Using Multivariate Control Charts Based on Kernel Density Estimation

“…The last indicator is item nonresponse rate, calculated as the ratio of item nonresponse ("don't know", refusal, and no answer) to the total number of the applicable items. For a more detailed introduction to the calculation of the indicators, we refer to our previous work (Jin and Loosveldt 2020).…”

Multivariate semiparametric control charts for mixed-type data

“…Multivariate CCs for mixed-type data have only been studied by Ning and Tsung, 43 Tuerhong and Kim, 44 Ding et al, 45 Wang et al, 46 Ashan et al, 47,49,48 and Jin and Loosveldt. 50 Specifically, Ning and Tsung 43 suggested a CC for high-dimensional and mixed-type observations that exploits a density-based method, the Local Outlier Factor (LOF). The authors of the LOF methodology assume that the data form relatively well separated clusters (“clustering condition”).…”

“…51 The CL of the latter ( P C A m i x) CC is calculated using Kernel Density Estimation (KDE). Jin and Loosveldt 50 suggested a modification of the previous idea where the test statistic remains the same, but the CL is established via bootstrap. Tuerhong and Kim 44 also introduced a CC for mixed-type data with bootstrap CL whose test statistic is slightly different as the quadratic form of T 2 is calculated by the aid of Gower’s distance.…”

Nonparametric multivariate control chart for numerical and categorical variables

“…Out of its available tools, the control chart is the most important and most widely used. Although initially applied in industrial fields, the control chart has found uses in many non-industrial fields, such as public health (see the review article by Thor et al 2007), finance (e.g., Kovářık and Sarga 2014;Bodnar and Schmid 2011;Bilson et al 2010), and even surveys (Jin and Loosveldt 2020;Jin et al 2019;Sirkis et al 2011). It calculates control limits using statistical equations, and graphically presents the fluctuations of the quality characteristic of a process.…”

“…One problem that arises is that unlike the scores based on standard principal component analysis (PCA), the scores based on PCA Mix do not follow any known family of distribution (Ahsan et al 2018). Among the limited research that has integrated the use of PCA Mix with the T 2 chart, Ahsan et al (2018Ahsan et al ( , 2021 employed kernel density estimation (KDE) whereas Jin and Loosveldt (2020) employed the bootstrap method to estimate a certain percentile (e.g., the 99 th percentile) of the PCA Mix-based T 2 distribution to use as the control limit of their control charts.…”