Holistic rubric vs. analytic rubric for measuring clinical performance levels in medical students

Yune, Sojung; Lee, Sang Yeoup; Im, Sun Ju; Kam, Beesung; Baek, Sun‐Yong

doi:10.1186/s12909-018-1228-9

Cited by 18 publications

(18 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the light of this finding, it can be said no significant difference in ranking would be seen between the use of analytic rubric scoring and general impression scoring for those assessments aiming to rank the examinees in terms of their ability. This research finding corroborates that "there is a high correlation between ability estimations calculated according to different scoring methods" in studies carried out with teachers/instructors/expert raters, and that focusing on comparing analytical and holistic rubrics (Anita, 2011;Chi, 2001;Ghalib & Al-Hattami, 2015;Hunter, Jones, Radhawa, 1996;Yune, Lee, Im, Kam, & Baek, 2018) is also valid for the scoring via analytic rubric and general impression in peer assessment.…”

Section: Discussion Conclusion and Suggestionssupporting

confidence: 81%

Akran değerlendirmede analitik rubrikle ve genel izlenimle yapılan puanlamaların incelenmesi

Öztürk¹,

Şahin

İlhan

2019

Turkish Journal of Education

View full text Add to dashboard Cite

The aim of this research was to analyze and compare analytic rubric and general impression scoring in peer assessment. A total of 66 university students participated in the study and six of them were chosen as peer raters on a voluntary basis. In the research, students were supposed to prepare a sample study within the scope of scientific research methods course and were also expected to present their studies in class. While the students were giving a presentation, their course instructor and peer raters conducted scoring, firstly by using the analytic rubric and subsequently by using general impressions. Collected data were analyzed using the Rasch model. Consequently, it was found that students were distinguished from one another at a highly reliable rate using both scoring methods. Additionally, it was discovered that the differences between students' ability levels were better revealed when analytic rubric was used. It was ascertained that there was a high level positive correlation between the ability estimations obtained from the scores performed by the peers and the instructor, regardless of the scoring method used. Finally, it was determined that ability estimations, corresponding peer raters' analytic rubric and general impression scoring, held a positive and highly strong relation.

show abstract

Section: Discussion Conclusion and Suggestionssupporting

confidence: 81%

Akran değerlendirmede analitik rubrikle ve genel izlenimle yapılan puanlamaların incelenmesi

Öztürk¹,

Şahin

İlhan

2019

Turkish Journal of Education

View full text Add to dashboard Cite

show abstract

“…, 2016). Employing an analytic rubric enabled us to capture multiple, distinct ideas present in the same student response in individual rubric categories (Yune et al. , 2018).…”

Section: Methodsmentioning

confidence: 99%

Mixed Student Ideas about Mechanisms of Human Weight Loss

Sripathi

Moscarella

Yoho

et al. 2019

LSE

View full text Add to dashboard Cite

Recent calls for college biology education reform have identified “pathways and transformations of matter and energy” as a big idea in biology crucial for students to learn. Previous work has been conducted on how college students think about such matter-transforming processes; however, little research has investigated how students connect these ideas. Here, we probe student thinking about matter transformations in the familiar context of human weight loss. Our analysis of 1192 student constructed responses revealed three scientific (which we label “Normative”) and five less scientific (which we label “Developing”) ideas that students use to explain weight loss. Additionally, students combine these ideas in their responses, with an average number of 2.19 ± 1.07 ideas per response, and 74.4% of responses containing two or more ideas. These results highlight the extent to which students hold multiple (both correct and incorrect) ideas about complex biological processes. We described student responses as conforming to either Scientific, Mixed, or Developing descriptive models, which had an average of 1.9 ± 0.6, 3.1 ± 0.9, and 1.7 ± 0.8 ideas per response, respectively. Such heterogeneous student thinking is characteristic of difficulties in both conceptual change and early expertise development and will require careful instructional intervention for lasting learning gains.

show abstract

“…Multiple concepts can be present within the same response, and therefore, the response could be scored as 1 in multiple analytic rubric bins. Some studies have found that analytic rubrics are more reliable in that they check key content components of reasoning and provide specific feedback to students (Jönsson and Svingby 2007;Jescovitch et al 2019b;Yune et al 2018).…”

Section: Challenges In Applying Coding Approaches To Machine Learningmentioning

confidence: 99%

“…Capturing complex content interactions is claimed to be easier with holistic rubrics than analytic rubrics (Tomas et al 2019). Holistic rubrics provide a comprehensive evaluation of a response and are thought of as easy to use (Jescovitch et al 2019b;Yune et al 2018).…”

Section: Challenges In Applying Coding Approaches To Machine Learningmentioning

confidence: 99%

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

et al. 2020

View full text Add to dashboard Cite

We systematically compared two coding approaches to generate training datasets for machine learning (ML): (i) a holistic approach based on learning progression levels and (ii) a dichotomous, analytic approach of multiple concepts in student reasoning, deconstructed from holistic rubrics. We evaluated four constructed response assessment items for undergraduate physiology, each targeting five levels of a developing flux learning progression in an ion context. Human-coded datasets were used to train two ML models: (i) an 8-classification algorithm ensemble implemented in the Constructed Response Classifier (CRC), and (ii) a single classification algorithm implemented in LightSide Researcher’s Workbench. Human coding agreement on approximately 700 student responses per item was high for both approaches with Cohen’s kappas ranging from 0.75 to 0.87 on holistic scoring and from 0.78 to 0.89 on analytic composite scoring. ML model performance varied across items and rubric type. For two items, training sets from both coding approaches produced similarly accurate ML models, with differences in Cohen’s kappa between machine and human scores of 0.002 and 0.041. For the other items, ML models trained with analytic coded responses and used for a composite score, achieved better performance as compared to using holistic scores for training, with increases in Cohen’s kappa of 0.043 and 0.117. These items used a more complex scenario involving movement of two ions. It may be that analytic coding is beneficial to unpacking this additional complexity.

show abstract

Holistic rubric vs. analytic rubric for measuring clinical performance levels in medical students

Cited by 18 publications

References 20 publications

Akran değerlendirmede analitik rubrikle ve genel izlenimle yapılan puanlamaların incelenmesi

Akran değerlendirmede analitik rubrikle ve genel izlenimle yapılan puanlamaların incelenmesi

Mixed Student Ideas about Mechanisms of Human Weight Loss

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

Contact Info

Product

Resources

About