Identifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI2030

Abstract

Challenges in measuring early childhood development (ECD) at scale have been documented, yet little is known about the specific difficulties related to questionnaire design and question interpretation. The purpose of this paper is to discuss the challenges of measuring ECD at scale in the context of household surveys and to show how to overcome them. The paper uses examples from the cognitive interviewing exercises that were conducted as part of the methodological work to develop a measure of ECD outcomes, the ECDI2030. It describes the methodological work carried out to inform the selection and improvement of question items and survey implementation tools as a fundamental step to reduce and mitigate systematic measurement error and improve data quality. The project consisted of a total of five rounds of testing, comprising 191 one-on-one, in-depth cognitive interviews across six countries (Bulgaria, India, Jamaica, Mexico, Uganda, and the USA). Qualitative data analysis methods were used to determine matches and mismatches between intention of items and false positives or false negative answers among subgroups of respondents. Key themes emerged that could potentially lead to systematic measurement error in population-based surveys on ECD: (1) willingness of child to perform task versus ability of child to perform task; (2) performing task versus performing task correctly; (3) identifying letters or numbers versus recognizing letters or numbers; (4) consistently performing task versus correctly performing task; (5) applicability of skills being asked versus observability of skills being asked; and (6) language production versus language comprehension. Through an iterative process of testing and subsequent revision, improvements were made to item wording, response options, and interviewer training instructions. Given the difficulties inherent in population-level data collection in the context of global monitoring, this study’s findings confirm the importance of cognitive testing as a crucial step in careful, culturally relevant, and sensitive questionnaire design and as a means to reduce response bias in cross-cultural contexts.

Feeding, caregiving practices, and developmental delay among children under five in lowland Nepal: a community-based cross-sectional survey

“…The low ECDI scores may be explained by the low proportion of children who are ‘on track’ in the socio-emotional domain, which includes items that may be strongly influenced by social norms that require children to be calm and compliant. Moreover, a recent study on the cognitive testing of ECDI2030 highlights the development and refinement of population-level measures of ECD should be culturally relevant to reduce response bias [ 35 ].…”

“…Third, the information on feeding and caregiving practices was self-reported by mothers, which may be liable to recall bias and social desirability bias. Fourth, the developmental potential of children at 36–59 months of age was assessed using the ECDI score, which has limited coverage of cognition and culturally specific developmental milestones [ 35 , 62 ]. Fifth, we did not analyse the factors affecting maternal and paternal caregiving practices separately.…”

Gender similarities and differences in early childhood development in low- and middle-income countries

“…A few limitations to our study deserve attention, in part because they offer important topics for future research. Although the ECDI was designed as a brief tool to track and compare basic skills in early childhood across geographically, socioeconomically, and culturally diverse contexts, there has been ongoing debate on its validity and precision to measure specific developmental subdomains (Cappa et al, 2021; Munoz-Chereau et al, 2021; Richter et al, 2019). Moreover, the ECDI has been criticized because its target age range (i.e., 36–59 months) and four domains were not clearly aligned with the SDG indicators (UNICEF, n.d.).…”

School readiness among children born to women living with HIV in Dar es Salaam, Tanzania: a cohort study protocol

“…As a secondary measure of development, we will implement the Early Child Development Index 2030 (ECDI2030), which is commonly used in population-based surveys to assess whether a child is meeting expected developmental milestones in motor, language, math, literacy, executive functioning and socioemotional domains. 15 The ECDI2030 is used to evaluate progress towards SDG target 4.2 and is administered as 20 close-ended questions to the mother or primary caregiver, where they indicate whether their child has exhibited the behaviour in each question. The ECDI2020 indicator is defined as the proportion of children 24–59 months of age who have achieved the minimum number of milestones expected for their age according to age-specific cut-off scores compared with all children aged 24–59 months of age.…”