Automatic assignment of reviewers in an online peer assessment task based on social interactions

Anaya, Antonio R.; Luque, Manuel; Letón, Emilio; Hernández-del-Olmo, Félix

doi:10.1111/exsy.12405

Cited by 9 publications

(12 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At the individual user level we have proposed the following metrics, which were explained in depth in our previous research (Anaya et al, 2019): the number of conversations that a student has started (Initiative); the number of messages that a student has sent (Messages); the number of replies to messages sent by a student (Popularity); Closeness centrality (Closeness); Betweenness centrality (Betweeness); Eigenvector centrality (Eigenvector).…”

Section: Student Social Behavior Attributes and The Social Network An...mentioning

confidence: 99%

“…To have a full implementation of the ID, we need to elicit the numerical values corresponding to the probability and utility parameters. By refining the scheme of elicitation used by (Anaya et al, 2019), we propose a more sophisticated approach by partitioning the set of numerical parameters into three sets: 1. Those parameters that are taken from expert knowledge and that do not depend on the course to which the model is applied.…”

Section: Numerical Parametersmentioning

confidence: 99%

See 1 more Smart Citation

Team assembly approach based in social modelling

Anaya

Letón

Luque

2023

Preprint

View full text Add to dashboard Cite

Team assembly is an important issue when collaborative or cooperative activities between people are carried out. In the educational context, it seems relevant to us to consider in the assembly process how students interact with others. Therefore, it is appropriate to carry out a process to model social behavior. In turn, the experts will be able to use the modeling information to establish the appropriate criteria to differentiate between assembly strategies according to the educational contexts. Our social behavior modeling process proposal is based on the information that can be extracted from the interactions and is guided by experts. We propose an approach divides into steps. Firstly, social network analysis metrics are used to study students’ social interactions. Secondly, a set of social interactions attributes are offered so that experts can organize them hierarchically. Thirdly a quantitative information on the students’ relationship that an influence diagram can infer due to the attributes hierarchically ordered. Once the modelling process is out, the qualitative information may be used to improve the team assembly. We describe in this paper two experiences where we have put it into practice our approach. The students made a peer review task. However, the pedagog-ical strategies that experts followed were different for each experience.

show abstract

Section: Student Social Behavior Attributes and The Social Network An...mentioning

confidence: 99%

Section: Numerical Parametersmentioning

confidence: 99%

Team assembly approach based in social modelling

Anaya

Letón

Luque

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In the selection rules for the review of learners' work, it is possible to consider adding a mandatory assignment selection method, or to combine the mode of learners' arbitrary choice of assignment and the mode of the system to automatically push assignments. In the current online peer assessment, there are mainly two selection rules for the review of learners' work: one is that learners choose the work to review according to their own preferences, and the other is assignment assigned by the system (Anaya et al, 2019). Since participation behaviour positively and significantly affects learners' social network prestige, combined with directional research in social network characteristics, it can be inferred that learners who submit assignments late and with poor quality have difficulty getting more feedback to help them reflect, especially from a high-prestige group.…”

Section: Implications For Education and Practicementioning

confidence: 99%

A model of factors influencing in-service teachers’ social network prestige in online peer assessment

Ma¹,

Du²,

2022

AJET

View full text Add to dashboard Cite

The rise of teacher training in online interactive learning environments has contributed to teachers’ professional development and brought new vitality to the informatisation of education. Many researchers have reported that there is a participation gap in online interactive learning environments. Research on the factors influencing this is very important. Social network prestige, which measures the degree to which learners gain peer attention in directed social networks, is one of the important metrics to characterise the participation gap. In this study, we offered an online teacher training course, and 1438 in-service teachers from primary and secondary schools attended. Among them, we selected 457 in-service teachers who participated in the three peer assessment activities as the final participants. To analyse the factors influencing learners’ social network prestige in online peer assessment, we first conducted a partial least squares structural equation modelling analysis to construct a model of factors influencing social network prestige. Then, we adopted several semi-structured interviews to investigate learners' perspectives to provide an in-depth analysis of the factors influencing social network prestige. The purpose of this study was to gain insight into the participation gap in online interactions and make effective suggestions on how to improve learning performance in online peer assessment. Implications for practice or policy: Course designers could improve the design of the introduction to peer assessment to motivate learners and enhance their acceptance of the activities. Course designers could reduce participation gap by assigning work from low-prestige learners to high-prestige learners in a non-mandatory way later in the course.

show abstract

“…Students in the study by Alkhalifa and Devlin (2021) revealed this, clarifying that suitable pairing in peer assessment could increase their engagement. Anaya et al (2019) and Capuano and Caballé (2015) observed that many studies ignore the selection of reviewers in peer assessment, as reviewers are determined randomly or manually by experts. Random or manual matching is not viable, especially in online environments which involve a large number of students.…”

Section: Introductionmentioning

confidence: 99%

“…Random or manual matching is not viable, especially in online environments which involve a large number of students. Moreover, the social interactions of peer assessments (e.g., characteristics of peers) have still not been addressed in depth (Anaya et al, 2019;van Gennip et al, 2009), despite the fact that social interactions between peers play an important role in peer assessment (Bouguessa & Romdhane, 2015). Thus, this study examined such issues in the context of addressing authors' doubts about the value of the feedback received in peer assessment.…”

Section: Introductionmentioning

confidence: 99%

Matching Authors and Reviewers in Peer Assessment Based on Authors’ Profiles

2022

View full text Add to dashboard Cite

Aim/Purpose: To encourage students’ engagement in peer assessments and provide students with better-quality feedback, this paper describes a technique for author-reviewer matching in peer assessment systems – a Balanced Allocation algorithm. Background: Peer assessment concerns evaluating the work of colleagues and providing feedback on their work. This process is widely applied as a learning method to involve students in the progress of their learning. However, as students have different ability levels, the efficacy of the peer feedback differs from case to case. Thus, peer assessment may not provide satisfactory results for students. In order to mitigate this issue, this paper explains and evaluates an algorithm that matches the author to a set of reviewers. The technique matches authors and reviewers based on how difficult the authors perceived the assignment to be, and the algorithm then matches the selected author to a group of reviewers who may meet the author’s needs in regard to the selected assignment. Methodology: This study used the Multiple Criteria Decision-Making methodology (MCDM) to determine a set of reviewers from among the many available options. The weighted sum method was used because the data that have been collected in user profiles are expressed in the same unit. This study produced an experimental result, examining the algorithm with a real collected dataset and mock-up dataset. In total, there were 240 students in the real dataset, and it contained self-assessment scores, peer scores, and instructor scores for the same assignment. The mock-up dataset created 1000 records for self-assessment scores. The algorithm was evaluated using focus group discussions with 29 programming students and interviews with seven programming instructors. Contribution: This paper contributes to the field in the following two ways. First, an algorithm using a MCDM methodology was proposed to match authors and reviewers in order to facilitate the peer assessment process. In addition, the algorithm used self-assessment as an initial data source to match users, rather than randomly creating reviewer – author pairs. Findings: The findings show the accurate results of the algorithm in matching three reviewers for each author. Furthermore, the algorithm was evaluated based on students’ and instructors’ perspectives. The results are very promising, as they depict a high level of satisfaction for the Balanced Allocation algorithm. Recommendations for Practitioners: We recommend instructors to consider using the Balanced Allocation algorithm to match students in peer assessments, and consequently to benefit from personalizing peer assessment based on students' needs. Recommendation for Researchers: Several MCDM methods could be expanded upon, such as the analytic hierarchy process (AHP) if different attributes are collected, or the artificial neural network (ANN) if fuzzy data is available in the user profile. Each method is suitable for special cases depending on the data available for decision-making. Impact on Society: Suitable pairing in peer assessment would increase the credibility of the peer assessment process and encourage students’ engagement in peer assessments. Future Research: The Balanced Allocation algorithm could be applied using a single group, and a peer assessment with random matching with another group may also be conducted, followed by performing a t-test to determine the impact of matching on students’ performances in the peer assessment activity.

show abstract

Automatic assignment of reviewers in an online peer assessment task based on social interactions

Cited by 9 publications

References 45 publications

Team assembly approach based in social modelling

Team assembly approach based in social modelling

A model of factors influencing in-service teachers’ social network prestige in online peer assessment

Matching Authors and Reviewers in Peer Assessment Based on Authors’ Profiles

Contact Info

Product

Resources

About