Enhancing the effectiveness of concept inventories using textual analysis: investigations in an electrical engineering subject

Goncher, Andrea; Boles, Wageeh

doi:10.1080/03043797.2017.1410523

Cited by 5 publications

(6 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By combining effective questioning and carefully designed distractors, these tests can assess students' understanding and identify misconceptions. Despite this, they are lacking in their ability to evaluate guessed selections, provide evidence for the cause or reasoning behind students' misconceptions or provide insight into the nature of students' understanding (Goncher & Boles, 2017).…”

Section: Automated Assessment Approachesmentioning

confidence: 99%

Applying natural language processing to automatically assess student conceptual understanding from textual responses

Somers

Cunningham-Nelson

Boles

2021

AJET

Self Cite

View full text Add to dashboard Cite

In this study, we applied natural language processing (NLP) techniques, within an educational environment, to evaluate their usefulness for automated assessment of students’ conceptual understanding from their short answer responses. Assessing understanding provides insight into and feedback on students’ conceptual understanding, which is often overlooked in automated grading. Students and educators benefit from automated formative assessment, especially in online education and large cohorts, by providing insights into conceptual understanding as and when required. We selected the ELECTRA-small, RoBERTa-base, XLNet-base and ALBERT-base-v2 NLP machine learning models to determine the free-text validity of students’ justification and the level of confidence in their responses. These two pieces of information provide key insights into students’ conceptual understanding and the nature of their understanding. We developed a free-text validity ensemble using high performance NLP models to assess the validity of students’ justification with accuracies ranging from 91.46% to 98.66%. In addition, we proposed a general, non-question-specific confidence-in-response model that can categorise a response as high or low confidence with accuracies ranging from 93.07% to 99.46%. With the strong performance of these models being applicable to small data sets, there is a great opportunity for educators to implement these techniques within their own classes. Implications for practice or policy: Students’ conceptual understanding can be accurately and automatically extracted from their short answer responses using NLP to assess the level and nature of their understanding. Educators and students can receive feedback on conceptual understanding as and when required through the automated assessment of conceptual understanding, without the overhead of traditional formative assessment. Educators can implement accurate automated assessment of conceptual understanding models with fewer than 100 student responses for their short response questions.

show abstract

Section: Automated Assessment Approachesmentioning

confidence: 99%

Applying natural language processing to automatically assess student conceptual understanding from textual responses

Somers

Cunningham-Nelson

Boles

2021

AJET

Self Cite

View full text Add to dashboard Cite

show abstract

“…The topics on the continuous time SSCI are background mathematics, linearity and time invariance (LTI), convolution, transform representations, and filtering. Despite their limitations [5], concept inventories are useful for testing a large number of students and comparing results across universities. Researchers typically give the 25-question SSCI in a pre/post-test format and calculate average gain statistics, <g> = 100 • post−pre 25−pre , to measure how many concepts students learn as a percentage of how many that they did not know before taking SS.…”

Section: Previous Research On Conceptual Understandingmentioning

confidence: 99%

Work in Progress: A Longitudinal Study of Students’ Conceptual Understanding of Signals and Systems

Crockett¹,

Finelli²,

Powell³

2020 ASEE Virtual Annual Conference Content Access Proceedings

View full text Add to dashboard Cite

show abstract

“…CIs utilize multiple-choice questions (MCQs) and specifically designed response selections to help identify misconceptions. CI fail, however, to provide evidence of the causes of the misconceptions or the nature of student conceptual understanding [19]. Although MCQ-based CIs can measure student learning and understanding, they do not provide measures of higher-level thinking.…”

Section: Introductionmentioning

confidence: 99%

“…It has been found that textual analysis on student written explanations can provide better judgements of their conceptual understanding. Goncher and Boles [19] described a framework to analyze assessment instruments that utilize textual, short-answer responses. However, their framework has not yet been tested for large sample sizes.…”

Section: Introductionmentioning

confidence: 99%

Development of Bloom’s-level Graduated Instrument for Assessing Transport Concepts in Hands-on Learning

Khan

Kaiphanliam

Thiessen

et al.

2019 ASEE Annual Conference &Amp; Exposition Proceedings

View full text Add to dashboard Cite

Bio-sketchAminul Islam Khan has received B.Sc and M.Sc in Mechanical Engineering from the most regarded and reputed engineering university of Bangladesh, Bangladesh University Engineering and Technology (BUET). In his B.Sc degree, he had received the department Gold medal for his outstanding achievements.Aminul Islam Khan has joined to BUET in 2011 as a Lecturer in Mechanical Engineering Department. In 2015, he has become an Assistant Professor in the same department of BUET. In 2016, he has joined to School of Mechanical and Materials Engineering of WSU as a PhD student. From that time, he has been working as a Research Assistant. As a research assistant, he has been working to improve learning/teaching methods in undergraduate engineering education along with his scientific research. He is currently working on drug delivery approach modeling for treatment of neurodegenerative diseases. Aminul Islam Khan is committed to excellence in teaching as well as research and always promotes a student-centered learning environment. He has a keen ability to teach, advise, and recruit students. He has also proven himself to be a very effective researcher by publishing several journal articles. His resume has a substantial list of publications, including peer-reviewed articles in national and international journals and conferences. Moreover, he has joined in several reputed conferences, for example American Physical Society (APS), and presented his scholarly works. Kitana Kaiphanliam, Washington State UniversityKitana Kaiphanliam is a first-year doctoral student in the Chemical Engineering program at Washington State University (WSU). Her research interests include biomanufacturing for immunotherapy applications and miniaturized hands-on learning devices for engineering education. Kitana is an active member of the American Institute of Chemical Engineers (AIChE) at WSU, and serves as their Graduate Student Chair for the 2018-19 academic year.

show abstract

Enhancing the effectiveness of concept inventories using textual analysis: investigations in an electrical engineering subject

Cited by 5 publications

References 8 publications

Applying natural language processing to automatically assess student conceptual understanding from textual responses

Applying natural language processing to automatically assess student conceptual understanding from textual responses

Work in Progress: A Longitudinal Study of Students’ Conceptual Understanding of Signals and Systems

Development of Bloom’s-level Graduated Instrument for Assessing Transport Concepts in Hands-on Learning

Contact Info

Product

Resources

About