Oral and Written Symbol Comprehension Testing: The Benefit of Cognitive Interview Probing

Abstract

Traditionally, symbol comprehension is tested using open-ended, written responses. However, responses are often so brief that they may fail to indicate a participant's true understanding of some symbols. In the present study, several test methods were compared to the standard written method to determine if they produce better symbol comprehension performance. The four alternative methods included: written test with probe questioning after all responses were provided, oral test without probe questioning, oral test with probe questioning after responses for all symbols were provided, and oral test with probe questioning after each partially correct or incorrect response. The probe or follow-up questioning technique is taken from the cognitive interview procedure used in eyewitness identification research to elicit more detailed responses. Participants reported their interpretations of 31 safety symbols in one of the 5 test method conditions. Results showed that the test methods that included follow-up questioning elicited more information from participants, and increased comprehension rates in both oral and written test formats. The results have implications for cost-effective symbol design and evaluation.

Tools for Symbol Development – Safety Symbol Response Taxonomy and Graphic Compatibility Verification

“…Moderators of in-person testing can often get clarity for these types of responses by using a non-leading follow-up prompt such as "What do you mean by...?" (Brantley & Wogalter, 1999;Wisniewski, Isaacson & Hall, 2007;ANSI Z535.3, 2011).…”

Multicultural, Rapid, Collaborative, Iterative Design and Evaluation of a Warning Pictorial

“…Following completion of each questionnaire, the responses were reviewed to determine whether there was enough detail to determine 1) the specific nature of the hazard, and 2) how the participant would avoid the hazard. If responses were not specific enough, the participant was asked questions to elicit more specific details (Brantley & Wogalter, 1999). Once additional details were obtained, the experimenter asked the participant to explain each of his answers to understand which elements of the pictorial led to each response.…”

Comprehension Testing of Product Safety Symbols: Lessons Learned

“…Testing can be conducted through written or oral questions, or a combination of the two. Comprehension scores can be affected when an oral component and probing questions are added to a written questionnaire (Brantley and Wogalter, 1999), and oral testing allows people who are illiterate (often a target audience for symbols) or uncomfortable with writing to participate. Participants will at times leave a question blank or respond with a written "don't know," but they will generally always respond to oral questioning.…”