Mixed Methods Research in Language Testing and Assessment

Abstract

As an alternative paradigm, mixed methods research (MMR), in general, endorses pluralism to understand the complex nature of a social world from multiple perspectives and multiple methodological lenses, each of which offers partial, yet valuable, insights. This methodological mixing is not limited to mixing of methods, but extends to the entire inquiry process. Researchers in language testing and assessment (LTA) are increasingly turning to MMR in order to understand the complexities of language acquisition and interaction among various language users, and also to expand opportunities to investigate validity claims beyond the three traditional facets of construct, content, and criterion validity. We use current conceptualizations of validity as a guiding framework to review 32 empirical MMR studies that have been published in LTA since 2007. Our systematic review encompassed multiple areas of foci, including the rationale for the use of MMR, evidence of collaboration, and synergetic effects. The analyses revealed several key trends including: (a) triangulation and complementarity were the prevalent uses of MMR in LTA; (b) the majority of the studies took place predominantly in higher education learning contexts with adult immigrant or university populations; (c) aspects of writing assessment were most frequently the focus of the studies (compared to other language modalities); (d) many of the studies explicitly addressed facets of validity, and others had significant implications for expanding notions of validity in LTA; (e) the majority of the studies avoided mixing at the data analysis stage by distinguishing data types and reporting results separately; and (f) integration occurred primarily at the discussion stage. We contend that LTA should embrace MMR through creative designs and integrative analytic strategies to seek new insights into the complexities and contexts of language testing and assessment.

Raters’ Decision Making Variations in Scoring Writing Samples

“…For example, one rater indicated an intuitive first reading followed by a systematic second reading. This rating behavior is referred to by Jang et al (2014) as "a pragmatic two-scan read". The patterns of comments also provided evidence that raters were sometimes coming to their decision as a result of a "pragmatic two-scan read" where an initial scan could not lead to a grade, leading to a second more systematic treatment.…”

“…In addition, Jang et al (2014) identified a "provisional mark" technique where some raters (but not all) initially decided upon a grade early in reading a text and continued reading to confirm the provisional mark. Perhaps this technique was used more by raters with a tendency to spontaneous decision making.…”

“…A rating scale was a 4-point focused-holistic rating scale developed by Turner and Upshur (2002), and validation studies indicated that the scale functioned acceptably for the intended purpose in subjectively scoring writing tasks (Crossly, Jang, Wagner & Park, 2014;Kyle & McNamara, 2016;Plakans & Gebril, 2013;Shrestha & Coffin, 2012). To estimate inter-and intra-coder reliability, five pages of the coded data were randomly chosen.…”

“…A great deal of this work has focused on accounting for systematic variability in rater scoring. Research on rater training has so far suggested that training is useful in increasing rater consistency (Jang, Wagner & Park, 2014;McNamara, 1996;Plakans & Gebril, 2013 ;Weigle, 1999Weigle, , 2002Weir, 2005), but there continues to be unexplained variability that resists training (Crossley, Kyle & MacNamara, 2016;Hoyt & Kerns, 1999;Plakans & Gebril, 2013;Shrestha & Coffin, 2012).…”

Using Authentic Writing Contests to Prepare Third Graders for High Stakes Standardized Assessments

“…Writing contests meet the same curricular goals through authentic tasks and increase students' interest and motivation in writing (Duke et al, 2006;Jocson et al, 2006). This mixed-methods study integrated qualitative and quantitative data to understand students' writing development related to their writing scores, emotions, and experiences (Jang, Wagner, & Park, 2014). The data demonstrated using writing contest tasks was equally or more beneficial than traditional test preparation tasks for supporting students' writing development.…”