Effects of COVID-19 Pandemic on Progress Test Performance in German-Speaking Countries

Abstract

Background. The COVID-19 pandemic has been the source of many challenges for medical students worldwide. The authors examined short-term effects on the knowledge gain of medical students in German-speaking countries. Methods. The development of the knowledge gain of medical students during the pandemic was measured by comparing the outcomes of shared questions within Berlin Progress Test (PT) pairs. The PT is a formative test of 200 multiple choice questions at the graduate level, which provides feedback to students on knowledge and knowledge gain during their course of study. It is provided to about 11,000 students in Germany and Austria around the beginning of each semester. We analyzed three successive test pairs: PT36-PT41 (both conducted before the pandemic), PT37-PT42 (PT37 took place before the pandemic; PT42 was conducted from April 2020 onwards), and PT38-PT43 (PT38 was administered before the pandemic; PT43 started in November 2020). The authors used mixed-effect regression models and compared the absolute variations in the percentage of correct answers per subject. Results. The most recent test of each PT pair showed a higher mean score compared to the previous test in the same pair (PT36-PT41 : 2.53 (95% CI: 1.31–3.75), PT37-PT42 : 3.72 (2.57–4.88), and PT38-PT43 : 5.66 (4.63–6.69)). Analogously, an increase in the share of correct answers was observed for most medical disciplines, with Epidemiology showing the most remarkable upsurge. Conclusions. Overall, PT performance improved during the pandemic, which we take as an indication that the sudden shift to online learning did not have a negative effect on the knowledge gain of students. We consider that these results may be helpful in advancing innovative approaches to medical education.

Ensino remoto emergencial na Medicina: aspectos positivos e negativos no ensino e na aprendizagem em tempos de pandemia

“…Uma das possibilidades de análise para avaliação de desempenho é a aplicação de Testes de Progresso, visto que foi observado por Sehy et al 77 ganho de conhecimento de estudantes de Medicina em países de língua alemã, em que o desempenho no Teste de Progresso melhorou durante a pandemia. Os autores, portanto, consideraram esse dado uma indicação de que a mudança repentina para o aprendizado on-line não teve um efeito negativo no ganho de conhecimento dos alunos.…”

“…O fato de a maioria dos artigos tratar de desafios relacionados à adaptação do ensino ao contexto remoto, somado às questões metodológicas mencionadas anteriormente, indica como, na maioria das situações, o ensino remoto foi tido como uma ferramenta emergencial para possibilitar a não interrupção completa do curso 6,10,11,[13][14][15][16]40,47 Com o exposto em vista, é possível relacionar a duração de sessões mais curtas sendo mais proveitosas com o fato de que, no ambiente virtual, os papéis de professores e alunos estão, de certa forma, invertidos 77 . Isso faz com que fique ainda mais evidente que o papel do professor contemporâneo é de servir como um guia, um mediador e um facilitador, auxiliando os estudantes no desenvolvimento de suas habilidades básicas (pensamento crítico, reflexão e comunicação) e específicas para o contexto on-line (autoaprendizado, aprendizado de longa data, administração de mudanças e administração de conhecimento) 77 .…”

The Identification of Guessing Patterns in Progress Testing as a Machine Learning Classification Problem

“…The Progress Test Medizin (PTM) is a computer-based test issued by the Charité-Universitätsmedizin Berlin and administered at 17 German, Austrian and Swiss medical schools [22]; it consists of 200 multiple choice questions at graduate level [23], the same for all participating faculties. The exam is conducted through four different platforms (ePT, Moodle, LPlus and ILIAS), among which the ePT platform collects the most extensive information about student performance and test-taking behaviour.…”

Discovering unknown response patterns in progress test data to improve the estimation of student performance

“…Such a personalized analysis is a known factor for effective feedback (e.g. [46] and references therein) and is requested by PTM students, as shown by PTM surveys [15,16]. For example, students can visually compare their response patterns with the response patterns of all clusters and thus get a better overview of their current level of knowledge.…”

“…In addition, total scores cannot be used to infer questionlevel responses: scores from questions of different content with different confidence levels and difficulty indices may add up to the same total score. Surveys on PTM have shown that students wish to compare their performance to that of other participating universities [15,16]. We aim to identify groups of students by inferring response patterns using only the response status, correctness of, and confidence in all answers, excluding pre-defined criteria such as test-taking effort ("seriousness"), cohorts, and curricular differences across universities.…”