A Web-based program for coding open-ended response protocols

Ames, Susan L.; Gallaher, Peggy; Park, Sun; Pearce, Steve G.; Zogg, Jennifer B.; Houska, Brian; Leigh, Barbara C.; Stacy, Alan W.

doi:10.3758/bf03192716

Cited by 8 publications

(6 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…That is, responses were entered verbatim into the computer, and then coded by two independent judges into the relevant myth and fact content categories (see below). We used a computerized coding and reliability report program to expedite and formalize the coding and data entry process for the open-ended measures (see Ames et al, 2005). Overall agreement between judges on the Myth responses coded ranged from 81% to 90%.…”

Section: Methodsmentioning

confidence: 99%

Prevention Education Effects on Fundamental Memory Processes

et al. 2012

View full text Add to dashboard Cite

This study evaluated effects of a key session from a nationally recognized drug abuse prevention program on basic memory processes in 211 high-risk youth in Southern California. In a randomized, between-subject design, the authors manipulated assignment to a Myth and Denial program session and the time of assessment (immediate vs. one-week delay). The authors examined program decay effects on memory accessibility and judgment errors. Those participants exposed to the program session generated more myths and facts from the program than those in the control group, suggesting that even a single program session influenced students’ memory for program information and this was retained at least one week and detectable with indirect tests of memory accessibility. However, consistent with basic research perspectives, participants in the program delayed assessment group erroneously generated more fact-related information from the session to the prompt “It is a myth that_____” than the participants in the program immediate assessment group; that is, they retained more facts as myths. These types of program effects, anticipated by basic memory theory, were not detected with a traditional judgment task in the present sample. The results suggest that basic science approaches offer a novel way of conceptually recasting prevention effects to more completely understand how these effects may operate. Implications for program evaluation and conceptualization are discussed.

show abstract

Section: Methodsmentioning

confidence: 99%

Prevention Education Effects on Fundamental Memory Processes

et al. 2012

View full text Add to dashboard Cite

show abstract

“…In practice, intercoder disagreement is common when a text is coded by two or more coders (Ames et al, 2005;Carley, 1993;Crittenden & Hill, 1971;Popping & Roberts, 2009;Schonlau, 2015). Intercoder agreement is typically measured by Cohen's k coefficient, a measure that takes into account chance agreement among the human coder (Fleiss, Levin, & Paik, 2003).…”

Section: Whether and How Intercoder Disagreement Affect Statistical Lmentioning

confidence: 99%

Automatic Coding of Text Answers to Open-Ended Questions: Should You Double Code the Training Data?

Schonlau

2019

Social Science Computer Review

View full text Add to dashboard Cite

Open-ended questions in surveys are often manually coded into one of several classes (or categories). When the data are too large to manually code all texts, a statistical (or machine) learning model must be trained on a manually coded subset of texts. Uncoded texts are then coded automatically using the trained model. The quality of automatic coding depends on the trained statistical model, and the model relies on manually coded data on which it is trained. While survey scientists are acutely aware that the manual coding is not always accurate, it is not clear how double coding affects the classification errors of the statistical learning model. We investigate several budget allocation strategies when there is a limited budget for manual classification: single coding versus various options for double coding where the number of training texts is reduced to maintain the fixed budget. Under fixed budget, double coding improved prediction of the learning algorithm when the coding error is greater than about 20–35%, depending on the data. Among double-coding strategies, paying for an expert to resolve differences performed best. When no expert is available, removing differences from the training data outperformed other double-coding strategies. When there is no budget constraint and the texts have already been double coded, all double-coding strategies generally outperformed single coding. As under fixed budget, having an expert to solve disagreement in training texts improves accuracy most, followed by removing differences.

show abstract

“…consult Ames et al (2005) for a suitable and flexible Webbased program. The use of Web-based ratings allows for the entire process, from data collection through scoring, to be conducted off-site.…”

Section: Author Notementioning

confidence: 99%