Measuring Skills in Developing Countries

Abstract

Measures of cognitive, noncognitive, and technical skills are increasingly used in developing country surveys, but have mostly been validated in high-income countries. We use a survey experiment in Western Kenya to test the reliability and validity of commonly used skills measures. Cognitive skills measures are found to be reliable and internally consistent, technical skills are very noisy, and measurement error in noncognitive skills is found to be non-classical. Addressing both random and systematic measurement error using common psychometric practices and repeated measures leads to some improvements and clearer predictions, though concerns remain. These findings hold for a replication in Colombia. JEL codes: O12, O13, O15The data and programs for full replication are available online at https://www.laajaj.com/research 1 Jack Pfeiffer provided invaluable research assistance and programming skills, field support, tenacity and ideas throughout the project and we gratefully acknowledge all his contributions to this project. provided excellent research assistance. Data collection was organized through IPA Kenya and S.E.I. in Colombia and we acknowledge the excellent field teams for key contributions during piloting and translation and for all efforts during the data collection. We are indebted to Chris Soto for many suggestions and insights from the psychometric literature. This open access article is distributed under the terms of the CC-BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0) and is freely available online at: http://jhr.uwpress.org 2 Hence we broadly follow the distinction of Heckman and Kautz (2012) who distinguish between cognitive abilities, personality traits, and other acquired skills. Jones and Kondylis (2018) is a recent example using a measure of acquired agricultural skills, in which measurement error is being discussed as potential reason for lack of impact. 3 Measures of risk aversion and time preferences, which have a longer history of use in developing country surveys, have received more scrutiny. Chuang and Schechter (2015) review the evidence, including their stability over time. 4 While explanatory factor analysis is used elsewhere in the economics literature on skills Heckman, Stixrud and Urzua, 2006), we also build on insights from the psychometrics literature, such as for the corrections for acquiescence bias (the tendency to agree even when statements are contradictory, or "yea-saying"). 5 Following McKenzie (2012), in order to reduce the noise in the outcome variables, the measures of yield and practices are obtained from the average over the four seasons that followed the collection of skills data.

Be Wary of Those Who Ask: A Randomized Experiment on the Size and Determinants of the Enumerator Effect

“…There is also evidence that the enumerator effect varies with the type of question, being more salient for questions concerning gender-related issues, religion, ethnicity, and politics (Schaeffer, 1980;Davis and Silver, 2003;Himelein, 2016;Laajaj and Macour, 2017).…”

“…The main method used to identify the presence of an enumerator effect and its magnitude is to look at how much enumerators themselves contribute to any variation in observed responses. The authors begin by testing for the predictive power of enumerators on respondent answers by examining the R 2 in an enumerator fixed effects regression, similar to Himelein (2016) and Laajaj and Macour (2017). A high R 2 is interpreted as the enumerator effect picking up a large amount of the variation in responses, while a low R 2 indicates that there is no or very low enumerator effect.…”

Las habilidades cognitivas y socioemocionales de los adultos y sus resultados en el mercado laboral en Colombia

“…Respondents may give socially desirable answers rather than answer how they truly behave, respond under the influence of how the exact questions are asked and how much response options they have, or constantly agree or disagree with the series of questions. Plus, they are not always consistent over short periods of time: a study in rural Colombia reveals that respondents tend to change their answers for survey questions related to the Big Five personality traits and other attitudes when asked two times with a three-week interval (Laajaj & Macours, 2018). 8 In addition, using step surveys including the Colombia one we use in this paper, Laajaj et al (2019) show that the scores of the Big Five personality traits based on survey questions do not always measure them as they are supposed to by design and thus cannot be confidently interpreted as measures of them.…”

“…While this method allows us to observe the relationship between labor market outcomes and the nine disaggregated measures of skills, estimations could be biased because of measurement error. Survey-based measures of skills can be poor, especially in low-and middle-income countries (Almlund et al, 2011;Laajaj & Macours, 2018;Laajaj et al, 2019). Respondents may give socially desirable answers rather than answer how they truly behave, respond under the influence of how the exact questions are asked and how much response options they have, or constantly agree or disagree with the series of questions.…”

Measuring socio-emotional skills in schools: simple questionnaires outperform behavioral tasks

“…They can also be affected by the fact that different people can have different standards or reference points, which may lead two people that are objectively identical on a given trait (say self-control or grit) to report different scores on that trait because the reference point of their group is different. In addition, responses may be affected by the cognitive skills of the respondent and her ability to understand the questions asked (29,30). Thirdparty questionnaires are also exposed to the "reference bias" and may be further flawed by subjective impressions or by misinterpretations of behavior, as well as poor observation and thus information on that behavior.…”