Can health workers reliably assess their own work? A test–retest study of bias among data collectors conducting a Lot Quality Assurance Sampling survey in Uganda

Beckworth, Colin; Davis, Rosemary; Faragher, Brian; Valadez, Joseph J.

doi:10.1093/heapol/czt110

Cited by 6 publications

(9 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These data were integrated into a superset, and in this study we analysed mothers’ responses to the question “Where did you give birth?”, their age at the time of the survey (in years) and their education level (none, primary, secondary, post-secondary). Uganda LQAS data reliability studies are available for review [ 41 , 42 ].…”

Section: Methodsmentioning

confidence: 99%

Assessing delivery practices of mothers over time and over space in Uganda, 2003–2012

Sprague

Jeffery

Crossland

et al. 2016

Emerg Themes Epidemiol

View full text Add to dashboard Cite

BackgroundIt is well known that safe delivery in a health facility reduces the risks of maternal and infant mortality resulting from perinatal complications. What is less understood are the factors associated with safe delivery practices. We investigate factors influencing health facility delivery practices while adjusting for multiple other factors simultaneously, spatial heterogeneity, and trends over time.MethodsWe fitted a logistic regression model to Lot Quality Assurance Sampling (LQAS) data from Uganda in a framework that considered individual-level covariates, geographical features, and variations over five time points. We accounted for all two-covariate interactions and all three-covariate interactions for which two of the covariates already had a significant interaction, were able to quantify uncertainty in outputs using computationally intensive cluster bootstrap methods, and displayed outputs using a geographical information system. Finally, we investigated what information could be predicted about districts at future time-points, before the next LQAS survey is carried out. To do this, we applied the model to project a confidence interval for the district level coverage of health facility delivery at future time points, by using the lower and upper end values of known demographics to construct a confidence range for the prediction and define priority groups.ResultsWe show that ease of access, maternal age and education are strongly associated with delivery in a health facility; after accounting for this, there remains a significant trend towards greater uptake over time. We use this model together with known demographics to formulate a nascent early warning system that identifies candidate districts expected to have low prevalence of facility-based delivery in the immediate future.ConclusionsOur results support the hypothesis that increased development, particularly related to education and access to health facilities, will act to increase facility-based deliveries, a factor associated with reducing perinatal associated mortality. We provide a statistical method for using inexpensive and routinely collected monitoring and evaluation data to answer complex epidemiology and public health questions in a resource-poor setting. We produced a model based on this data that explained the spatial distribution of facility-based delivery in Uganda. Finally, we used this model to make a prediction about the future priority of districts that was validated by monitoring and evaluation data collected in the next year.Electronic supplementary materialThe online version of this article (doi:10.1186/s12982-016-0049-8) contains supplementary material, which is available to authorized users.

show abstract

Section: Methodsmentioning

confidence: 99%

Assessing delivery practices of mothers over time and over space in Uganda, 2003–2012

Sprague

Jeffery

Crossland

et al. 2016

Emerg Themes Epidemiol

View full text Add to dashboard Cite

show abstract

“…The questionnaire was adapted so that questions for which the answer could change between the test and the retest were excluded. The questionnaire was the same one as used in a previous smaller LQAS reliability study [ 11 ]. Therefore, the results for this study are directly comparable to the previous LQAS reliability study.…”

Section: Methodsmentioning

confidence: 99%

“…An initial, albeit small scale, study assessing whether local data collectors are a source of bias in LQAS survey [ 11 ], found no evidence to support the hypothesis that they bias the data they collect. However, that study was restricted to one district, and the second set of dis-interested data collectors came from the same district; also the sample size was small consisting of 76 participants.…”

Section: Introductionmentioning

confidence: 99%

Can local staff reliably assess their own programs? A confirmatory test-retest study of Lot Quality Assurance Sampling data collectors in Uganda

Beckworth

Anguyo

Kyakulaga

et al. 2016

BMC Health Serv Res

View full text Add to dashboard Cite

BackgroundData collection techniques that routinely provide health system information at the local level are in demand and needed. LQAS is intended for use by local health teams to collect data at the district and sub-district levels. Our question is whether local health staff produce biased results as they are responsible for implementing the programs they also assess.MethodsThis test-retest study replicates on a larger scale an earlier LQAS reliability assessment in Uganda. We conducted in two districts an LQAS survey using 15 local health staff as data collectors. A week later, the data collectors swapped districts, where they acted as disinterested non-local data collectors, repeating the LQAS survey with the same respondents. We analysed the resulting two data sets for agreement using Cohens’ Kappa.ResultsThe average Kappa score for the knowledge indicators was k = 0.43 (SD = 0.16) and for practice indicators k = 0.63 (SD = 0.17). These scores show moderate agreement for knowledge indicators and substantial agreement for practice indicators. Analyses confirm that respondents were more knowledgeable on retest; no evidence of bias was found for practice indicators.ConclusionThe findings of this study are remarkably similar to those produced in the first reliability study. There is no evidence that using local healthcare staff to collect LQAS data biases data collection in an LQAS study. The bias observed in the knowledge indicators was most likely due to a ‘practice effect’, whereby respondents increased their knowledge as a result of completing the first survey; no corresponding effect was seen in the practice indicators.Electronic supplementary materialThe online version of this article (doi:10.1186/s12913-016-1655-4) contains supplementary material, which is available to authorized users.

show abstract

“…Other comparisons of LQAS with demographic surveillance systems have proved to have an excellent agreement of results, but in those occasions the indicators were identical [13]. Similarly, reliability studies of LQAS have recently compared data collected by managers who use LQAS results to improve their own programmes with data collected by disinterested data collectors; the concordance of the two data sets was very high [14].…”

Section: Surveys Are Complementary Not Redundantmentioning

confidence: 99%

Comparing two survey methods of measuring health‐related indicators: Lot Quality Assurance Sampling and Demographic Health Surveys

Anoke

Mwai

Jeffery

et al. 2015

Tropical Med Int Health

View full text Add to dashboard Cite

Abstractobjectives Two common methods used to measure indicators for health programme monitoring and evaluation are the demographic and health surveys (DHS) and lot quality assurance sampling (LQAS); each one has different strengths. We report on both methods when utilised in comparable situations.methods We compared 24 indicators in south-west Uganda, where data for prevalence estimations were collected independently for the two methods in 2011 (LQAS: n = 8876; DHS: n = 1200). Data were stratified (e.g. gender and age) resulting in 37 comparisons. We used a two-sample two-sided Ztest of proportions to compare both methods.results The average difference between LQAS and DHS for 37 estimates was 0.062 (SD = 0.093; median = 0.039). The average difference among the 21 failures to reject equality of proportions was 0.010 (SD = 0.041; median = 0.009); among the 16 rejections, it was 0.130 (SD = 0.010, median = 0.118). Seven of the 16 rejections exhibited absolute differences of <0.10, which are clinically (or managerially) not significant; 5 had differences >0.10 and <0.20 (mean = 0.137, SD = 0.031) and four differences were >0.20 (mean = 0.261, SD = 0.083).conclusion There is 75.7% agreement across the two surveys. Both methods yield regional results, but only LQAS provides information at less granular levels (e.g. the district level) where managerial action is taken. The cost advantage and localisation make LQAS feasible to conduct more frequently, and provides the possibility for real-time health outcomes monitoring.keywords monitoring and evaluation, stratified sampling, cluster sampling, lot quality assurance sampling, demographic and health survey, Uganda

show abstract

Can health workers reliably assess their own work? A test–retest study of bias among data collectors conducting a Lot Quality Assurance Sampling survey in Uganda

Cited by 6 publications

References 9 publications

Assessing delivery practices of mothers over time and over space in Uganda, 2003–2012

Assessing delivery practices of mothers over time and over space in Uganda, 2003–2012

Can local staff reliably assess their own programs? A confirmatory test-retest study of Lot Quality Assurance Sampling data collectors in Uganda

Comparing two survey methods of measuring health‐related indicators: Lot Quality Assurance Sampling and Demographic Health Surveys

Contact Info

Product

Resources

About