Construct validation of progress testing to measure knowledge and visual skills in radiology

Ravesloot, Cécile J.; Schaaf, Marieke van der; Haaring, Cees; Kruitwagen, Cas; Beek, Erik van; Cate, Olle ten; Schaik, Jan van

doi:10.3109/0142159x.2012.716177

Cited by 22 publications

(34 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Regarding construct validity, we only found significant differences in mean test scores between lower training years. This finding had been reported by the authors and others previously [5,16]. The reliability per year group is inconsistent.…”

Section: Discussionsupporting

confidence: 58%

An exploratory study into the effect of time-restricted internet access on face-validity, construct validity and reliability of postgraduate knowledge progress testing

Dijksterhuis

Jozwiak

Braat³

et al. 2013

BMC Med Educ

View full text Add to dashboard Cite

BackgroundYearly formative knowledge testing (also known as progress testing) was shown to have a limited construct-validity and reliability in postgraduate medical education. One way to improve construct-validity and reliability is to improve the authenticity of a test. As easily accessible internet has become inseparably linked to daily clinical practice, we hypothesized that allowing internet access for a limited amount of time during the progress test would improve the perception of authenticity (face-validity) of the test, which would in turn improve the construct-validity and reliability of postgraduate progress testing.MethodsPostgraduate trainees taking the yearly knowledge progress test were asked to participate in a study where they could access the internet for 30 minutes at the end of a traditional pen and paper test. Before and after the test they were asked to complete a short questionnaire regarding the face-validity of the test.ResultsMean test scores increased significantly for all training years. Trainees indicated that the face-validity of the test improved with internet access and that they would like to continue to have internet access during future testing. Internet access did not improve the construct-validity or reliability of the test.ConclusionImproving the face-validity of postgraduate progress testing, by adding the possibility to search the internet for a limited amount of time, positively influences test performance and face-validity. However, it did not change the reliability or the construct-validity of the test.

show abstract

Section: Discussionsupporting

confidence: 58%

An exploratory study into the effect of time-restricted internet access on face-validity, construct validity and reliability of postgraduate knowledge progress testing

Dijksterhuis

Jozwiak

Braat³

et al. 2013

BMC Med Educ

View full text Add to dashboard Cite

show abstract

“…How expertise in radiology develops and which factors influence its development is largely unknown (10,11). In a previous cross-sectional study (12), we found that scores on knowledge and image questions in a radiology test differ between 1st-, 2nd-, and 3rd-year residents, but were similar for residents in postgraduate year 4 and 5. This suggested that knowledge and image interpretation skill development slows down during residency; however, this was not evaluated in a longitudinal study that tracked individual score development over time.…”

Section: Implication For Patient Carementioning

confidence: 79%

“…We hypothesized that development of radiologic knowledge and image interpretation skill show a curvilinear growth (12). We also hypothesized that image interpretation skill develops faster in a nonacademic training environment because of higher reading volumes than in academic hospitals (13).…”

Section: Methodsmentioning

confidence: 99%

“…Questions with low reliability were removed from the test, and therefore maximum scores ranged from 188 to 197 (the total number of questions in the test). A detailed description of the test and its quality can be found in Ravesloot et al (12) and in Table E2 (online). In this previous study, test data from Dutch Radiology Progress Tests from 2005 to 2009 were evaluated for quality (12).…”

Section: Instrumentationmentioning

confidence: 99%

“…A detailed description of the test and its quality can be found in Ravesloot et al (12) and in Table E2 (online). In this previous study, test data from Dutch Radiology Progress Tests from 2005 to 2009 were evaluated for quality (12). Indicators of test quality were good.…”

Section: Instrumentationmentioning

confidence: 99%

See 2 more Smart Citations

Predictors of Knowledge and Image Interpretation Skill Development in Radiology Residents

et al. 2017

Self Cite

View full text Add to dashboard Cite

Purpose To investigate knowledge and image interpretation skill development in residency by studying scores on knowledge and image questions on radiology tests, mediated by the training environment. Materials and Methods Ethical approval for the study was obtained from the ethical review board of the Netherlands Association for Medical Education. Longitudinal test data of 577 of 2884 radiology residents who took semiannual progress tests during 5 years were retrospectively analyzed by using a nonlinear mixed-effects model taking training length as input variable. Tests included nonimage and image questions that assessed knowledge and image interpretation skill. Hypothesized predictors were hospital type (academic or nonacademic), training hospital, enrollment age, sex, and test date. Results Scores showed a curvilinear growth during residency. Image scores increased faster during the first 3 years of residency and reached a higher maximum than knowledge scores (55.8% vs 45.1%). The slope of image score development versus knowledge question scores of 1st-year residents was 16.8% versus 12.4%, respectively. Training hospital environment appeared to be an important predictor in both knowledge and image interpretation skill development (maximum score difference between training hospitals was 23.2%; P < .001). Conclusion Expertise developed rapidly in the initial years of radiology residency and leveled off in the 3rd and 4th training year. The shape of the curve was mainly influenced by the specific training hospital. RSNA, 2017 Online supplemental material is available for this article.

show abstract

Fourteen years of progress testing in radiology residency training: experiences from The Netherlands

et al. 2017

View full text Add to dashboard Cite

ObjectivesTo describe the development of the Dutch Radiology Progress Test (DRPT) for knowledge testing in radiology residency training in The Netherlands from its start in 2003 up to 2016.MethodsWe reviewed all DRPTs conducted since 2003. We assessed key changes and events in the test throughout the years, as well as resident participation and dispensation for the DRPT, test reliability and discriminative power of test items.ResultsThe DRPT has been conducted semi-annually since 2003, except for 2015 when one digital DRPT failed. Key changes in these years were improvements in test analysis and feedback, test digitalization (2013) and inclusion of test items on nuclear medicine (2016). From 2003 to 2016, resident dispensation rates increased (Pearson’s correlation coefficient 0.74, P-value <0.01) to maximally 16 %. Cronbach´s alpha for test reliability varied between 0.83 and 0.93. The percentage of DRPT test items with negative item-rest-correlations, indicating relatively poor discriminative power, varied between 4 % and 11 %.ConclusionsProgress testing has proven feasible and sustainable in Dutch radiology residency training, keeping up with innovations in the radiological profession. Test reliability and discriminative power of test items have remained fair over the years, while resident dispensation rates have increased.Key Points • Progress testing allows for monitoring knowledge development from novice to senior trainee. • In postgraduate medical training, progress testing is used infrequently. • Progress testing is feasible and sustainable in radiology residency training.

show abstract

Construct validation of progress testing to measure knowledge and visual skills in radiology

Abstract: We found support for the reliability and construct validity of the DRPT. However, assessment on visual skill development needs further exploration.

Cited by 22 publications

References 24 publications

An exploratory study into the effect of time-restricted internet access on face-validity, construct validity and reliability of postgraduate knowledge progress testing

An exploratory study into the effect of time-restricted internet access on face-validity, construct validity and reliability of postgraduate knowledge progress testing

Predictors of Knowledge and Image Interpretation Skill Development in Radiology Residents

Fourteen years of progress testing in radiology residency training: experiences from The Netherlands

Contact Info

Product

Resources

About