Coding Text Answers to Open-ended Questions: Human Coders and Statistical Learning Algorithms Make Similar Mistakes

He, Zhoushanyue; Schonlau, Matthias

doi:10.12758/mda.2020.10

Cited by 5 publications

(12 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The equipment necessary to accomplish automated/semi-automated machine learning methods of coding, as well as the computer software and user skill necessary to develop coding dictionaries for use in the computer-assisted content analysis may be prohibitive. Further, though technology-aided methods of content analysis have mitigated the time-consuming nature of manual analysis methods, these methods have been found to have higher error rates than human coders (He & Schonlau, 2021, Schonlau & Couper, 2016). To address these potential sources of human error associated with manual coding (He & Schonlau, 2021, Schonlau & Couper, 2016, FORT-CAST incorporates a clearly defined codebook (the initial categorization scheme of coding categories and their definitions) on each tab of the EXCEL file to guide coders in their data analysis, frame-ofreference training, and practice coding sessions with feedback from primary coders.…”

Section: Discussionmentioning

confidence: 99%

“…Further, though technology-aided methods of content analysis have mitigated the time-consuming nature of manual analysis methods, these methods have been found to have higher error rates than human coders (He & Schonlau, 2021, Schonlau & Couper, 2016). To address these potential sources of human error associated with manual coding (He & Schonlau, 2021, Schonlau & Couper, 2016, FORT-CAST incorporates a clearly defined codebook (the initial categorization scheme of coding categories and their definitions) on each tab of the EXCEL file to guide coders in their data analysis, frame-ofreference training, and practice coding sessions with feedback from primary coders. These strategies serve to ensure clarity and understanding among multidisciplinary coders and enhance inter-rater reliability of the members of the coding team.…”

Section: Discussionmentioning

confidence: 99%

“…Neither automated, semi-automated, computer-assisted, nor manual methods of coding are without error; however, the sources of error vary by method. Semi-automated systems have been found to have higher error rates, only classifying up to 58% of textual data correctly (Schonlau & Couper, 2016) with the primary error reported as generalization error, or out-of-the-sample error, a measure of how accurately a machine algorithm can detect meaningful patterns in data (He & Schonlau, 2021). Conversely, errors in manual coding are largely due to human error, unclear coding manuals, or misunderstanding the meaning of textual responses.…”

Section: Manual Data Analysis Methodsmentioning

confidence: 99%

“…Conversely, errors in manual coding are largely due to human error, unclear coding manuals, or misunderstanding the meaning of textual responses. Nevertheless, the rigor of all methods is challenged when coding more lengthy, ambiguous text responses (He & Schonlau, 2021).…”

Section: Manual Data Analysis Methodsmentioning

confidence: 99%

“…If the estimated categorization probability threshold exceeds the computed threshold, automated coding is accepted; if the estimated probability is lower than the computed threshold, manual coding is required (Schonlau & Couper, 2016). While both fully automated and semiautomated coding methods have been introduced in an effort to efficiently analyze formidable sets of open-ended comments, these methods have been found to have higher error rates than human coders (He & Schonlau, 2021), automatically classifying only 58% of textual data accurately (Schonlau & Couper, 2016).…”

Section: Semi-automated Codingmentioning

confidence: 99%

See 4 more Smart Citations

Frame of reference training for content analysis with structured teams (FORT‐CAST): A framework for content analysis of open‐ended survey questions using multidisciplinary coders

Galura

Horan

Parchment

et al. 2022

Research in Nursing & Health

View full text Add to dashboard Cite

In the context of a global pandemic, the need for reliable analysis of qualitative data in healthcare has never been more pressing. Open-ended questions are a feasible way for both researchers and organizational stakeholders to gain deeper insight into complex situations when timely research is needed. However, the interpretation of brief, textual responses can prove problematic. Both manual and automated/ semiautomated methods of coding qualitative data have been associated with errors and costly temporal delays. Data obtained from the qualitative analysis of openended questions have been questioned for lacking robust insights. The present article introduces an innovative, manual, team-based method of analyzing responses to open-ended survey questions. This method was developed and implemented at the outset of the COVID-19 pandemic to understand the needs of nurses and their perceptions of organizational strategies that were implemented to address pandemic-related challenges. This framework utilizes a dedicated project management structure, general purpose software for data collection and analysis, frame-ofreference training designed for an interdisciplinary team of coders, and data analysis procedures that align with qualitative content analysis procedures. In concert, these techniques empower researchd team members with varying backgrounds and disparate levels of experience to provide unique human insights to data analysis procedures, refine the coding process, and support the abstraction of meaningful themes that were used to prioritize organizational strategies and further support nurses as the pandemic progressed.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Manual Data Analysis Methodsmentioning

confidence: 99%

Section: Manual Data Analysis Methodsmentioning

confidence: 99%

Section: Semi-automated Codingmentioning

confidence: 99%

See 3 more Smart Citations

Frame of reference training for content analysis with structured teams (FORT‐CAST): A framework for content analysis of open‐ended survey questions using multidisciplinary coders

Galura

Horan

Parchment

et al. 2022

Research in Nursing & Health

View full text Add to dashboard Cite

show abstract

Quantification of Textual Responses To Open-Ended Questions In Big Data

Merkys

Bubelienė

2023

Communications in Computer and Information Science

View full text Add to dashboard Cite

Challenges and opportunities in current vaccine technology and administration: A comprehensive survey examining oral vaccine potential in the United States

Rupassara¹,

Kindt²,

Kazi

et al. 2022

Human Vaccines & Immunotherapeutics

View full text Add to dashboard Cite

This study provides a snapshot of the current vaccine business ecosystem, including practices, challenges, beliefs, and expectations of vaccine providers. Our team focused on providers’ firsthand experience with administering vaccines to determine if an oral vaccine (e.g. pill or oral-drop) would be well-received. We interviewed 135 healthcare providers and vaccine specialists across the US, focusing questions on routine vaccinations, not COVID-19 vaccines. Improving workflow efficiency is a top concern among vaccine providers due to shrinking reimbursement rates—determined by pharmacy benefit managers (PBMs)—and the time-intensiveness of injectable vaccines. Administering injectable vaccines takes 23 minutes/patient on average, while dispensing pills takes only 5 minutes/patient. An average of 24% of patients express needle-fear, which further lengthens the processing time. Misaligned incentives between providers and PBMs could reduce the quality and availability of vaccine-related care. The unavailability of single-dose orders prevents some rural providers from offering certain vaccines. Most interviewees (74%) believe an oral vaccine would improve patient–provider experience, patient-compliance, and workflow efficiency, while detractors (26%) worry about the taste, vaccine absorption, and efficacy. Additional research could investigate whether currently non-vaccinating pharmacies would be willing to offer oral vaccines, and the impact of oral vaccines on vaccine acceptance.

show abstract

Coding Text Answers to Open-ended Questions: Human Coders and Statistical Learning Algorithms Make Similar Mistakes

Cited by 5 publications

References 6 publications

Frame of reference training for content analysis with structured teams (FORT‐CAST): A framework for content analysis of open‐ended survey questions using multidisciplinary coders

Frame of reference training for content analysis with structured teams (FORT‐CAST): A framework for content analysis of open‐ended survey questions using multidisciplinary coders

Quantification of Textual Responses To Open-Ended Questions In Big Data

Challenges and opportunities in current vaccine technology and administration: A comprehensive survey examining oral vaccine potential in the United States

Contact Info

Product

Resources

About