The Development of Knowledge Tests When Studying Academic Texts in Higher Education

Abstract

In order to measure students' learning performance, research on higher education mainly relies on the use of knowledge mastery tests. Although these tests are widely being used, the psychometric quality of these measurements has been given little attention. The aim of this study is to outline the development and validation of two knowledge mastery tests, serving as indicators for learning performance, through the use of item response theory. As studying academic texts is a major source for learning domain-specific content in higher education, both tests focus on knowledge mastery after studying an academic text on, respectively "problem solving theory" and "metacognition". Both tests were administered to first-year university students (n=180 for problem solving; n=249 for metacognition) enrolled for a Bachelor programme in Educational Sciences. The items were controlled for dimensionality, model-data fit, local item dependence and monotonicity. The final measure consisted of 16 out of 30 items for the knowledge test about the problem solving text. For the knowledge test about metacognition, 21 out of 30 items remained relevant and informative. The final results indicate that both knowledge tests are reliable, in particular for median ability levels. This study highlights the importance of developing knowledge mastery tests that provide accurate and valid measures for learning from academic tests.