Assessing the Reliability of a Chemical Engineering Problem-solving Rubric when Using Multiple Raters

Duckett, Timothy Ryan; Liberatore, Matthew W.; Asogwa, Uchenna; Mentzer, Gale; Malefyt, Amanda P.

doi:10.18260/1-2--32124

Cited by 4 publications

(6 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At least 3 raters individually scored each student solution to minimize rater bias. To ensure agreement among raters, an interrater reliability was completed and detailed previously . All PROCESS scores across cohorts and groups are reported by individual problem in the Supporting Information (Tables S.1–S.3).…”

Section: Methods and Frameworkmentioning

confidence: 99%

Video-Inspired, Student-Written Problems vs Textbook Problems: Comparing Difficulty and Problem-Solving Skills between Two Cohorts in Chemical Engineering

et al. 2023

Self Cite

View full text Add to dashboard Cite

Complex problem solving is one of the skills desired in the 21st century workforce. Students can improve their problem-solving skills by working on homework problems, and creating new problems can be a stimulating and inventive exercise for student as well. Our previous studies explored the impacts of a novel homework pedagogy where students reverse-engineered videos to create course-related homeworkquality problems. The present study investigated differences in problem difficulty and student problem-solving ability across two cohorts when solving video-inspired studentwritten (VISW) problems versus Textbook problems. The current study deployed similar mixed methods research on an additional cohort of undergraduate students in an engineering course focused on material and energy balances to verify and reproduce previous results. For each cohort, findings were drawn from ∼2000 task-load surveys and ∼2500 problem solving scores compiled by multiple raters. In general, outcomes for the second cohort replicated previous findings. The student-written problems were perceived equally as difficult as the Textbook problems. PROCESS scores from the Treatment group revealed better problem-solving skills when students solved VISW than Textbook problems. Students in the Treatment group displayed an increase in problem-solving ability across the ∼4-month long course with small to medium effect sizes (Hedges g = 0.6 and 0.1). Additionally, students in the Treatment group exhibited better problem-solving skills on the most difficult VISW problems compared to the most difficult Textbook problems.

show abstract

Section: Methods and Frameworkmentioning

confidence: 99%

Video-Inspired, Student-Written Problems vs Textbook Problems: Comparing Difficulty and Problem-Solving Skills between Two Cohorts in Chemical Engineering

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…Overall, stating that auto-graded activities are scaffolded has some justification based on the fraction correct as a function of question order. Using a single metric -fraction correct -to quantify scaffolding is a limitation of the study, and future work on problem difficulty and scaffolding could use multiple expert raters or use validated tools, e.g., NASA Task Load Index (Duckett et al 2019;Asogwa et al 2021). Question order will be examined further in the discussion of the Copy sheet button next.…”

Section: Question Type and Ordermentioning

confidence: 99%

Deliberate practice of spreadsheet skills when using copiable, randomized, and auto-graded questions within an interactive textbook

Gorbett¹

2022

Advances

View full text Add to dashboard Cite

Spreadsheets are a core computational tool for practicing engineers and engineering students.While Microsoft Excel, Google Sheets, and other spreadsheet tools have some differences, numerous formulas, functions, and other tasks are common across versions and platforms. Building upon learning science frameworks showing that interactive activities are advantageous, an interactive textbook from zyBooks was created to provide students the opportunity to acquire spreadsheet skills by doing. Interactive components include stepping through animations, completing multiple choice and matching questions, and evaluating spreadsheet formulas and functions using 100+ autograded, randomized questions. While the interactive reading participation was discussed in previous work, the focus here is on auto-graded questions, sometimes called online homework. Fraction correct regardless of attempts, number of attempts before correct, and number of attempts after correct provided metrics to examine deliberate practice across three cohorts encompassing over 250 students. Sections grouped as General spreadsheet skills or Functions generally showed greater median correct, ranging from 76 to 90%, than sections categorized as Advanced spreadsheet skills, which led to median correct of 68 to 81%. Median correct also varied between different question types and decreased with question order, which aligns with questions being scaffolded. Finally, a hypothesis was tested: Adding a Copy sheet feature to the auto-graded questions would encourage deliberate practice and improve fraction correct between cohorts. Over 80% of students used the new Copy sheet button, and a statistically significant increase with large effect size in fraction correct between cohorts was found, which supports the hypothesis.

show abstract

“…The vast majority of survey feedback about YouTube Fridays was positive and students felt they had a better understanding of the course topic and even felt confident solving problems. Also, students reported the ability to relate course to real world phenomena since YouTube pedagogy provided students a mechanism to apply classroom concepts to open-ended, real world situations [27][28][29][30].…”

Section: Introductionmentioning

confidence: 99%

Student Attitudes When Solving Homework Problems that Reverse Engineer YouTube Videos

Asogwa

Liberatore

Duckett

et al.

2020 ASEE Virtual Annual Conference Content Access Proceedings

Self Cite

View full text Add to dashboard Cite

show abstract

Assessing the Reliability of a Chemical Engineering Problem-solving Rubric when Using Multiple Raters

Cited by 4 publications

References 14 publications

Video-Inspired, Student-Written Problems vs Textbook Problems: Comparing Difficulty and Problem-Solving Skills between Two Cohorts in Chemical Engineering

Video-Inspired, Student-Written Problems vs Textbook Problems: Comparing Difficulty and Problem-Solving Skills between Two Cohorts in Chemical Engineering

Deliberate practice of spreadsheet skills when using copiable, randomized, and auto-graded questions within an interactive textbook

Student Attitudes When Solving Homework Problems that Reverse Engineer YouTube Videos

Contact Info

Product

Resources

About