Finding Quality Responses: The Problem of Low-Quality Survey Responses and Its Impact on Accountability Measures

Chen, Pu-Shih Daniel

doi:10.1007/s11162-011-9217-4

Cited by 26 publications

(21 citation statements)

References 47 publications

(55 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Such strategies can have severe consequences for the results of a survey, single-item results can be biased, reliability of the scales can be reduced or inflated, and the associations between scales can be increased. All these sources are a threat to the validity of the results, leading to lower statistical power and potentially erroneous conclusions (Barge and Gehlbach 2012;Chen 2011). Therefore, it is of very high importance to implement appropriate measures to detect or preferably prevent these behaviors.…”

Section: Data Collectionmentioning

confidence: 99%

“…Several measures to identify satisficers have been combined to ensure that only extreme satisficers were removed to prevent possible systematic errors from participant elimination (Chen 2011). These measures include the number of times the instructional manipulation check was failed (more three times), overly extreme stated prices (smaller than 40 % or larger than 160 % of reference value) and the answers on four duplicate questions.…”

Section: Data Collectionmentioning

confidence: 99%

See 1 more Smart Citation

Multichannel Commerce

Trenz

2015

Progress in IS

View full text Add to dashboard Cite

Section: Data Collectionmentioning

confidence: 99%

Section: Data Collectionmentioning

confidence: 99%

Multichannel Commerce

Trenz

2015

Progress in IS

View full text Add to dashboard Cite

“…In addition to concerns about response homogeneity, some researchers have also questioned the quality of undergraduate data. Chen utilized data from the National Survey of Student Engagement (NSSE) involving undergraduates from 587 US colleges and universities [11]. About 11% of first year and 7% of fourth year undergraduates failed to answer 30% or more of the 85 Web-based survey questions.…”

Section: Introductionmentioning

confidence: 99%

Assessing Diabetes-Relevant Data Provided by Undergraduate and Crowdsourced Web-Based Survey Participants for Honesty and Accuracy

2017

View full text Add to dashboard Cite

BackgroundTo eliminate health disparities, research will depend on our ability to reach select groups of people (eg, samples of a particular racial or ethnic group with a particular disease); unfortunately, researchers often experience difficulty obtaining high-quality data from samples of sufficient size. ObjectivePast studies utilizing MTurk applaud its diversity, so our initial objective was to capitalize on MTurk’s diversity to investigate psychosocial factors related to diabetes self-care.MethodsIn Study 1, a “Health Survey” was posted on MTurk to examine diabetes-relevant psychosocial factors. The survey was restricted to individuals who were 18 years of age or older with diabetes. Detection of irregularities in the data, however, prompted an evaluation of the quality of MTurk health-relevant data. This ultimately led to Study 2, which utilized an alert statement to improve conscientious behavior, or the likelihood that participants would be thorough and diligent in their responses. Trap questions were also embedded to assess conscientious behavior.ResultsIn Study 1, of 4165 responses, 1246 were generated from 533 unique IP addresses completing the survey multiple times within close temporal proximity. Ultimately, only 252 responses were found to be acceptable. Further analyses indicated additional quality concerns with this subsample. In Study 2, as compared with the MTurk sample (N=316), the undergraduate sample (N=300) included more females, and fewer individuals who were married. The samples did not differ with respect to race. Although the presence of an alert resulted in fewer trap failures (mean=0.07) than when no alert was present (mean=0.11), this difference failed to reach significance: F1,604=2.5, P=.11, ƞ²=.004, power=.35. The modal trap failure response was zero, while the mean was 0.092 (SD=0.32). There were a total of 60 trap failures in a context where the potential could have exceeded 16,000.ConclusionsPublished studies that utilize MTurk participants are rapidly appearing in the health domain. While MTurk may have the potential to be more diverse than an undergraduate sample, our efforts did not meet the criteria for what would constitute a diverse sample in and of itself. Because some researchers have experienced successful data collection on MTurk, while others report disastrous results, Kees et al recently identified that one essential area of research is of the types and magnitude of cheating behavior occurring on Web-based platforms. The present studies can contribute to this dialogue, and alternately provide evidence of disaster and success. Moving forward, it is recommended that researchers employ best practices in survey design and deliberately embed trap questions to assess participant behavior. We would strongly suggest that standards be in place for publishing the results of Web-based surveys—standards that protect against publication unless there are suitable quality assurance tests built into the survey design, distribution, and analysis.

show abstract

“…Several issues related to poor implementation have been proposed. First, scholars have questioned the quality of institutional data in higher education, particularly self-reported data and information from students (Adams & Umbach, 2012;Chen, 2011;Sax et al, 2003). Also, leaders may not involve employees at all organization levels in data practices because frontline academic employees often lack the time, resources, and understanding of data to effectively utilize data in everyday practice (Blaich & Wise, 2011;Bresciani et al, 2009).…”

mentioning

confidence: 99%

From Democratic to “Need to Know”: Linking Distributed Leadership to Data Cultures in the Florida College System

et al. 2020

View full text Add to dashboard Cite

This multiple case study examines the extent and ways in which leaders and administrators in Florida College System (FCS) institutions engage in distributed leadership through data sharing with frontline staff. Based on focus groups and individual interviews with administrators, faculty, and staff (659 participants) from 21 state colleges, we found a continuum of three data cultures ranging from democratic data cultures to blended data cultures to “need to know” data cultures. We triangulate these results with survey data from FCS institutional leaders and find considerable variation in the extent of data sharing and perceptions of effectiveness of institutional data use. Institutions with democratic data cultures tended to have distributed leadership that encouraged information sharing and collaboration among staff to use data to inform change. Need-to-know institutions faced challenges, including weak data quality, concerns about adequate time and resources among staff for reviewing data, and perceptions that staff lack data literacy skills.

show abstract

Finding Quality Responses: The Problem of Low-Quality Survey Responses and Its Impact on Accountability Measures

Cited by 26 publications

References 47 publications

Multichannel Commerce

Multichannel Commerce

Assessing Diabetes-Relevant Data Provided by Undergraduate and Crowdsourced Web-Based Survey Participants for Honesty and Accuracy

From Democratic to “Need to Know”: Linking Distributed Leadership to Data Cultures in the Florida College System

Contact Info

Product

Resources

About