Encouraging professional learning communities to increase the shared consensus in writing assessments

Gasse, Roos Van; Lesterhuis, Marije; Verhavert, San; Bouwer, Renske; Vanhoof, Jan; Petegem, Peter Van; Mæyer, Sven De

doi:10.1108/jpcc-08-2018-0021

Cited by 4 publications

(5 citation statements)

References 35 publications

(44 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The texts were evaluated using Comproved (https://comproved.com/), a software developed specifically for benchmarking competencies in a holistic manner (Van Gasse et al, 2019). All texts written by the students were contrasted by showing two texts (Text A and Text B) to qualified assessors and offering the op-tion to choose one of them as the "best elaboration of the task" according to the given instructions.…”

Section: Research Questions Methodology and Datamentioning

confidence: 99%

Source-Based Writing in Professional Spanish as a Foreign Language: Some Pedagogical Insights Based on a Quali-Quantitative Approach

Leijten¹,

Vandermeulen²,

Vangehuchten³

2022

RLA. Rev. Ling.

View full text Add to dashboard Cite

Source-based writing is a crucial skill for communication professionals. This study investigates how master students in professional communication incorporate sources in synthesis writing tasks in their L1 (Dutch), L2 (English) and FL (Spanish and French). To show the complexity that characterizes the use of sources, we will first present a case study of two students in Spanish as a foreign language. We will then link it to the results of a factor analysis that points out which variables are descriptive indicators of the way in which master's students (N = 209) deal with sources during the writing process. There are three components that can determine source use (75% of the total variance in the data): a) initial reading time, b) interaction with sources, and c) degree of variation in source use. In L1 there seems to be a correlation between the initial reading time and interaction with sources, on the one hand, and the quality of the writing, on the other; also, the time spent consulting sources compared to writing time is considerably longer in FL than in L1 and L2. These findings allow us to reflect upon how to develop effective teaching strategies to

show abstract

Section: Research Questions Methodology and Datamentioning

confidence: 99%

Source-Based Writing in Professional Spanish as a Foreign Language: Some Pedagogical Insights Based on a Quali-Quantitative Approach

Leijten¹,

Vandermeulen²,

Vangehuchten³

2022

RLA. Rev. Ling.

View full text Add to dashboard Cite

show abstract

“…From the literature on rater training emerges mainly that such a training provides a frame of reference to raters, may reduce rater biases and result in a shared understanding and more consistency between raters (Bachman & Palmer, 2010;Lumley, 2002;Shohamy et al, 1992;Wang et al, 2017;Weigle, 1994). Recently, it has also been demonstrated that by collectively comparing a series of example texts of varying performance levels raters can learn how to conceptualize text quality (Van Gasse et al, 2019).…”

Section: 4mentioning

confidence: 99%

“…Trained raters may still differ in focus, weighting and decision-making processes (Eckes, 2008;Vaughan, 1991). These different rater perspectives should rather be embraced than avoided, as they are essential for understanding the complexity and multidimensionality of text quality and sharing norms and standards (Jølle, 2015;Van Gasse et al, 2019). Key is then to make room for comparing and discussing different norms and perspectives on the evaluation of text quality.…”

Section: 4mentioning

confidence: 99%

Guidelines for the Validation of Writing Assessment in Intervention Studies

Bouwer,

Van Steendam,

Lesterhuis

2023

Conceptualizing, Designing, Implementing, and Evaluating Writing Interventions

View full text Add to dashboard Cite

To draw valid conclusions about the effectiveness of writing interventions, the assessment design is key. Within a writing assessment, observations of students' performance on a particular writing task are generally used to make inferences about their writing proficiency. To do that, researchers have to make various decisions, e.g., about the kind of text(s) students have to write, how the texts are scored and by whom. Moreover, they have to underpin these decisions with arguments and evidence to make sure the conclusions they draw based upon text quality scores are valid interpretations of students' writing proficiency. This requires a thorough and comprehensive analysis of theoretical, empirical, and analytical evidence of writing assessment information, which is often not manageable for researchers whose primary focus is on the writing intervention (Shaw et al., 2012).The aim of this chapter is to provide specific guidelines for researchers on how to make evidence-based choices when assessing writing in a writing intervention study. As such, the guidelines are meant to empower researchers to argue and support the validity of their decisions in and about the assessment of writing. To do that, we will use an existing framework for an argument-based approach to validity (Kane, 2006; and apply it to writing by reviewing theoretical and empirical evidence from writing assessment research.The chapter starts with a brief discussion of Kane's framework that presents a chain of inferences which researchers need to make on the basis of assessment scores. An in-depth discussion of Kane's framework remains outside this chapter's scope, but we will use it to specify the inferences that are generally made within the context of a writing intervention study and the kind of evidence that is needed to substantiate those inferences. For each proposed inference, we will use insights from writing assessment research to provide evidence-based guidelines for researchers to carefully select the most appropriate assessment procedures and to build their own case for the validity of these inferences.

show abstract

“…The work of Lesterhuis et al (2018) is also relevant as their findings report that the considered construct of quality in a comparative method is multidimensional and notes that teachers use their experience to give meaning to the evidence. Not only can the "bottom up" approach to standards setting unleash the expertise of teachers, but building on the insights shared by Van Gasse et al (2019) where they highlight the change in the conception of the assessor following a CJ intervention, it can also support the development of the teacher in terms of refining their constructs of capability. The richness of varying perspectives discussing the concrete context of the assignment, ensures task-orientated and not examiner-orientated focus.…”

Section: Research Agendamentioning

confidence: 99%

Using Teachers’ Judgments of Quality to Establish Performance Standards in Technology Education Across Schools, Communities, and Nations

2022

View full text Add to dashboard Cite

The establishment and maintenance of national examination standards remains a serious issue for teachers and learners, whilst the levers of control remain firmly in the hands of Awarding Bodies and supervising politicians. Significantly, holistic assessment presents an agility and collective approach to establishing in the minds of teachers “what is of value” when determining the comparative evidence of pupil performance. It is argued in this paper that the collation of the comparative judgment process can initially identify and subsequently maintain standards of performance that can be defined on a cluster, regional or even national level. Much comparative judgment research centers on the formative benefits for learners, but here we place the focus on teachers operating in collaborative groups to establish standards within and beyond their own schools, and ultimately across the nation. We model a proof-of-concept research project. A rank is produced by the collective consensus of the participating teachers and used to simulate a definition of standard. Extrapolations are statistically modeled to demonstrate the potential for this approach to establishing a robust definition of national standards. But central to the process is what is going on in the minds of teachers as they make their judgements of quality. The research aims to draw out teachers’ constructs of quality; to make them explicit; to share them across classrooms and schools; and to empower teachers to debate and agree their standards across schools. This research brings to the fore the symbiotic relationship between teaching, learning and assessment.

show abstract

Encouraging professional learning communities to increase the shared consensus in writing assessments

Cited by 4 publications

References 35 publications

Source-Based Writing in Professional Spanish as a Foreign Language: Some Pedagogical Insights Based on a Quali-Quantitative Approach

Source-Based Writing in Professional Spanish as a Foreign Language: Some Pedagogical Insights Based on a Quali-Quantitative Approach

Guidelines for the Validation of Writing Assessment in Intervention Studies

Using Teachers’ Judgments of Quality to Establish Performance Standards in Technology Education Across Schools, Communities, and Nations

Contact Info

Product

Resources

About