Development of a written assessment for a national interprofessional cardiotocography education program

Abstract

BackgroundTo reduce the incidence of hypoxic brain injuries among newborns a national cardiotocography (CTG) education program was implemented in Denmark. A multiple-choice question test was integrated as part of the program. The aim of this article was to describe and discuss the test development process and to introduce a feasible method for written test development in general.MethodsThe test development was based on the unitary approach to validity. The process involved national consensus on learning objectives, standardized item writing, pilot testing, sensitivity analyses, standard setting and evaluation of psychometric properties using Item Response Theory models. Test responses and feedback from midwives, specialists and residents in obstetrics and gynecology, and medical and midwifery students were used in the process (proofreaders n = 6, pilot test participants n = 118, CTG course participants n = 1679).ResultsThe final test included 30 items and the passing score was established at 25 correct answers. All items fitted a loglinear Rasch model and the test was able to discriminate levels of competence. Seven items revealed differential item functioning in relation to profession and geographical regions, which means the test is not suitable for measuring differences between midwives and physicians or differences across regions. In the setting of pilot testing Cronbach’s alpha equaled 0.79, whereas Cronbach’s alpha equaled 0.63 in the setting of the CTG education program. This indicates a need for more items and items with a higher degree of difficulty in the test, and illuminates the importance of context when discussing validity.ConclusionsTest development is a complex and time-consuming process. The unitary approach to validity was a useful and applicable tool for development of a CTG written assessment. The process and findings supported our proposed interpretation of the assessment as measuring CTG knowledge and interpretive skills. However, for the test to function as a high-stake assessment a higher reliability is required.Electronic supplementary materialThe online version of this article (doi:10.1186/s12909-017-0915-2) contains supplementary material, which is available to authorized users.

The impact of a national cardiotocography education program on neonatal and maternal outcomes: A historical cohort study

“…However, the extent, form and content of this teaching were not uniform. In contrast, the present CTG education program was standardized and consisted of an e‐learning program and a 1‐day course with a 30‐item CTG multiple‐choice question test …”

“…Completion of the e‐learning program was a prerequisite for attending the 1‐day course. The e‐learning and the 1‐day course are described in previous publications …”

“…In contrast, the present CTG education program was standardized and consisted of an e-learning program and a 1-day course with a 30-item CTG multiple-choice question test. 18 All physicians and midwives responsible for laboring women at a Danish maternity unit were obliged to participate in the program.…”

Development and validation of a written credentialing examination for overseas‐educated dietitians

“…Just as much thought, discussion and debate must go into their development as for any other assessment instrument used for high‐stakes purposes . Research papers continue to emerge from various health professional contexts presenting data that support the validity of using MCQs for credentialing purposes, and such validity studies are important in helping shore up public confidence in the decisions made on the basis of the results of MCQ examinations …”

“…4, 6 Research papers continue to emerge from various health professional contexts presenting data that support the validity of using MCQs for credentialing purposes, and such validity studies are important in helping shore up public confidence in the decisions made on the basis of the results of MCQ examinations. [7][8][9][10][11][12] Ideas about validity have also evolved, from being initially regarded as an inherent property of a test to an argument for the particular interpretations and decisions based on the results of the test. 13 In earlier conceptualisations, the validity of a test tended to be sought in specific and often isolated claims about the construct, content and criterion validity, 14 with emphasis frequently given to the latter (as 'predictive validity') in relation to subsequent real-life performance.…”