Continuous Evaluation in Ethics Education: A Case Study

Abstract

A great need for systematic evaluation of ethics training programs exists. Those tasked with developing an ethics training program may be quick to dismiss the value of training evaluation in continuous process improvement. In the present effort, we use a case study approach to delineate how to leverage formative and summative evaluation measures to create a high-quality ethics education program. With regard to formative evaluation, information bearing on trainee reactions, qualitative data from the comments of trainees, in addition to empirical findings, can ensure that the training program operates smoothly. Regarding summative evaluation, measures examining trainee cognition, behavior, and organization-level results provide information about how much trainees have changed as a result of taking the ethics training. The implications of effective training program evaluation are discussed.

Fostering a Research Integrity Culture: Actionable Advice for Institutions

“…Research integrity experts from both research performing and research funding organizations agree that the current approach to ethics training should be updated to match new developments such as open science, reproducibility, and environmental responsibility (LERU, 2020;Tomić et al, 2021). Preferably, a systematic approach is used (Antes, 2014;McIntosh et al, 2018), which starts with a needs assessment determining what should be learned, incorporating the input of professionals with relevant knowledge of practice. These needs are formulated using active verbs (e.g., "describe", "identify", "analyze", "reflect",… ) in terms of awareness, knowledge, skills, attitudes and/or behavior (Antes, 2014;Marusic et al, 2016;Pizzolato et al, 2020).…”

“…These needs are formulated using active verbs (e.g., "describe", "identify", "analyze", "reflect",… ) in terms of awareness, knowledge, skills, attitudes and/or behavior (Antes, 2014;Marusic et al, 2016;Pizzolato et al, 2020). Subsequently, learning materials and procedures to bridge the gap between the current and the desired situation should be planned and created (Antes, 2014;McIntosh et al, 2018). To prevent institutions having to reinvent the wheel and to aid institutions with limited resources, creating repositories of trainings with proven effectiveness has been suggested (Abdi, Pizzolato et al, 2021;European Science Foundation, 2010) and various resources are already available online -such as those of the Online Ethics Center for Engineering and Science (Available from https://onlineethics.org/resources), the ORI, and the European Network of Research Ethics and Research Integrity and the European Network of Research Integrity Offices (ENERI-ENRIO).…”

“…Thoroughly and systematically evaluating whether ethics training reaches its educational goals and whether it is up to date and future-proof is as important as its planning (Antes, 2014;Forsberg et al, 2018;Marusic et al, 2016;McIntosh et al, 2018;Mumford et al, 2015). Pre-post designs with control groups provide the most conclusive evidence for program effectiveness, but a multitude of designs and instruments is used (e.g., Antes et al, 2009;Marusic et al, 2016;McIntosh et al, 2018;Mumford et al, 2015;Zollitsch et al, 2021) and often only one or two outcome criteria are used (Marusic et al, 2016;McIntosh et al, 2018;Steele, Mulhearn et al, 2016). As a result, the available evidence is often unreliable, with a high risk of bias (Marusic et al, 2016;Zollitsch et al, 2021).…”

“…As a result, the available evidence is often unreliable, with a high risk of bias (Marusic et al, 2016;Zollitsch et al, 2021). Mumford et al (2015) proposed a "multilevel approach" in which outcome measures are tailored to the training objectives -awareness, knowledge, skills, attitudes and/or behavior (Marusic et al, 2016;McIntosh et al, 2018) -and evidence of different strength is combined into a balanced, global evaluation. For example, student ratings or answers to open-ended questions fall short of hard evidence.…”

Assessing clinical research coordinator knowledge of good clinical practice: An evaluation of the state of the art and a test validation study

“…Evidence indicates that training programs in research ethics differ significantly in effectiveness: While programs in general are not associated with any positive outcomes [7,8], some programs are associated with improved knowledge and decision-making skills [9]. It is therefore essential to assess training programs to determine whether they are effective [10,11]. Moreover, assessment enables programs to determine when an individual has adequately mastered important content [11].…”