Evaluating the Use and Interpretation of the TOEIC® Listening and Reading Test Score Report: Perspectives of Test Takers in Japan

Abstract

Researchers suggest that claims about the meaningfulness of test score interpretations and consequences of test use should be backed by evidence that stakeholders understand the definition of the construct assessed (meaningfulness) and score reports (consequences). Evaluation of stakeholders' actual uses and interpretations of score reports in large‐scale standardized language proficiency tests, however, remains limited in the score reporting literature. This study investigates how test takers, as an important stakeholder group, use and interpret the score report of the TOEIC® Listening and Reading (TOEIC L&R) test. Data were collected from 834 TOEIC L&R test takers in Japan, who represented a wide range of English language proficiency based on their TOEIC L&R total scores. The participants responded to an online survey that asked about their uses and interpretations of the test results and their comprehension of the performance feedback presented in the score report. The results showed that the participants used the test results largely as intended, providing an important piece of validity evidence to support the proposed uses of the TOEIC L&R test. Some of the score reporting elements such as the performance feedback and footnote message, however, were not easy to understand for all participants, revealing a need to improve the interpretability of the score report. The study findings have implications for designing informative score reports and the usefulness around reporting test performance feedback.

Interpretation and Use of a Workplace English Language Proficiency Test Score Report: Perspectives of TOEIC® Test Takers and Score Users in Taiwan

“…The average percentages taken across the five comprehension questions indicated that the test takers and institutional score users did roughly equally well (54.3% for test takers; 55.6% for institutional score users), whereas the corporate score users performed slightly less well with an average of 48.2%. Compared to the findings of Hsieh (2023) where TOEIC L&R test takers in Japan ( N = 834) answered, on average, 60.9% of the same five comprehension questions correctly, the relatively lower percentages seen in the current study raised the concern that the TOEIC L&R test takers and score users in Taiwan may have more challenges in understanding the abilities‐measured information and the explanatory texts in the footnote message. As mentioned earlier, the TOEIC L&R score report in Hsieh (2023) was presented in Japanese to the test takers and research participants in Japan and the one used in the current study was presented in the English language.…”

“…Compared to the findings of Hsieh (2023) where TOEIC L&R test takers in Japan ( N = 834) answered, on average, 60.9% of the same five comprehension questions correctly, the relatively lower percentages seen in the current study raised the concern that the TOEIC L&R test takers and score users in Taiwan may have more challenges in understanding the abilities‐measured information and the explanatory texts in the footnote message. As mentioned earlier, the TOEIC L&R score report in Hsieh (2023) was presented in Japanese to the test takers and research participants in Japan and the one used in the current study was presented in the English language. It was speculated that this difference in the language used in the score reports might have impacted stakeholders' comprehensions of the abilities‐measured information.…”

“…It should be noted that the current study was part of a larger study that examined the use and interpretation of the TOEIC L&R test scores and the abilities‐measured information. The study was also a follow‐up study of Hsieh (2023). In that study, only perspectives of test takers in Japan were included, which limited the generalizability of the study findings to other TOEIC L&R test stakeholder groups and test‐taking population.…”