An accurate and practical method for assessing science and engineering problem-solving expertise

Price, Argenta; Salehi, Shima; Burkholder, Eric; Kim, Candice; Isava, Virginia; Flynn, Michael R.; Wieman, Carl

doi:10.1080/09500693.2022.2111668

Cited by 7 publications

(4 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our research group has done extensive work to characterize and assess authentic problem-solving expertise across science, engineering, and medicine domains (Salehi, 2018;Price et al, 2021Price et al, , 2022. Drawing on these work, we now characterize problems in science and engineering domains along two dimensions: context and data specificity (Figure 1).…”

Section: Figurementioning

confidence: 99%

Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving

Wang,

Burkholder,

Wieman

et al. 2024

Front. Educ.

Self Cite

View full text Add to dashboard Cite

The study explores the capabilities of OpenAI's ChatGPT in solving different types of physics problems. ChatGPT (with GPT-4) was queried to solve a total of 40 problems from a college-level engineering physics course. These problems ranged from well-specified problems, where all data required for solving the problem was provided, to under-specified, real-world problems where not all necessary data were given. Our findings show that ChatGPT could successfully solve 62.5% of the well-specified problems, but its accuracy drops to 8.3% for under-specified problems. Analysis of the model's incorrect solutions revealed three distinct failure modes: (1) failure to construct accurate models of the physical world, (2) failure to make reasonable assumptions about missing data, and (3) calculation errors. The study offers implications for how to leverage LLM-augmented instructional materials to enhance STEM education. The insights also contribute to the broader discourse on AI's strengths and limitations, serving both educators aiming to leverage the technology and researchers investigating human-AI collaboration frameworks for problem-solving and decision-making.

show abstract

Section: Figurementioning

confidence: 99%

Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving

Wang,

Burkholder,

Wieman

et al. 2024

Front. Educ.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Template for Designing Decisions-Based Instruction Worksheets (Adapted From Price et al, 2022) The worksheet starts with a realistic problem scenario that contains realistic irrelevant and incomplete information and has multiple possible solution paths. To solve should require knowledge that learners are expected to know or learn during the process.…”

Section: Figurementioning

confidence: 99%

Improved Teaching of Science and Engineering Using Deliberate Practice of Problem-Solving Decisions

Argenta Price,

Carl Wieman

2023

ITL

View full text Add to dashboard Cite

The ability to solve complex realistic problems is the most important requirement for being a good scientist or engineer. Here we present a theoretical design for improving the effectiveness and efficiency in teaching technical problem solving at the post-secondary level. The approach brings together theoretical ideas and empirical results from (a) evidence-based instructional practices, (b) development of expertise through deliberate practice, and (c) a decision-based framework for complex problem solving. This design involves explicit practice and feedback on making the set of decisions that define the problem-solving process of skilled scientists and engineers. We provide a template for instruction based on this design. This design also provides a way to assess problem-solving skills that is more accurate than traditional examinations.

show abstract

“…Additionally, the conventional evaluation processes used to assess students' performance in these tasks often lack the ability to fully capture their PS skills or their capacity to make nuanced PS decisions, which are essential for achieving true mastery (Price, A. et al, 2022). Therefore, there is an urgent need for innovative instructional methods that emphasize the development of students' PS abilities, along with a fresh approach to accurate performance assessment.…”

Section: Introductionmentioning

confidence: 99%

Design and Validation of a Rubric to Assess Problem-Solving Skills of Students in a Chemistry Context

Garcés,

F. Sánchez-Barba,

Pérez-Quintanilla

et al. 2023

Proceedings of the 6th International Conference on New Trends in Teaching and Education

View full text Add to dashboard Cite

Interest in developing 21st Century Skills, particularly Problem-Solving (PS) abilities in Science Education, has grown in recent years. This study presents the design and validation of an analytic rubric to quantitatively assess PS performance in 23 students enrolled in the Water Resources (WR) degree program at Rey Juan Carlos University in Madrid, Spain. The rubric evaluated students' PS abilities through conveniently designed ill-structured problems within a General Chemistry course. The rubric consists of three dimensions, each representing different levels of PS mastery, with a total possible score of 60 points. By employing indicators ranked on a three-level scale, the rubric enables a comprehensive evaluation of students' problemsolving efforts in written solutions. Categorizing students into five stages of competence acquisition allows effective assessment of their PS skills within each level. evaluation. The rubric is a valuable educational tool, empowering continuous improvement and excellence in PS skills.

show abstract

An accurate and practical method for assessing science and engineering problem-solving expertise

Cited by 7 publications

References 33 publications

Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving

Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving

Improved Teaching of Science and Engineering Using Deliberate Practice of Problem-Solving Decisions

Design and Validation of a Rubric to Assess Problem-Solving Skills of Students in a Chemistry Context

Contact Info

Product

Resources

About