Programmatic assessment: Can we provide evidence for saturation of information?

Jong, Lubberta H. de; Bok, Harold G. J.; Kremer, W.D.J.; Vleuten, Cees van der

doi:10.1080/0142159x.2018.1555369

Cited by 19 publications

(17 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite our efforts to be comprehensive in our searches, it is possible that we inadvertently missed a paper even though we took steps such as hand searching the papers included to safeguard against this possibility. With respect to the holistic approach of meaningful triangulation, it is important to note that several studies in our review focused on a single assessment instrument, for example the mini-CEX [21,22,36], the progress test [27], or a single competence such as professionalism [25,29]. Additionally, our search strategy yielded a number of studies describing valuable lessons drawn from the local implementation process.…”

Section: Limitationsmentioning

confidence: 99%

Where the rubber meets the road — An integrative review of programmatic assessment in health care professions education

Schut¹,

Maggio²,

Heeneman³

et al. 2020

Perspect Med Educ

View full text Add to dashboard Cite

Introduction Programmatic assessment was introduced as an approach to design assessment programmes with the aim to simultaneously optimize the decision-making and learning function of assessment. An integrative review was conducted to review and synthesize results from studies investigating programmatic assessment in health care professions education in practice. Methods The authors systematically searched PubMed, Web of Science, and ERIC to identify studies published since 2005 that reported empirical data on programmatic assessment. Characteristics of the included studies were extracted and synthesized, using descriptive statistics and thematic analysis. Results Twenty-seven studies were included, which used quantitative methods (n = 10), qualitative methods (n = 12) or mixed methods (n = 5). Most studies were conducted in clinical settings (77.8%). Programmatic assessment was found to enable meaningful triangulation for robust decision-making and used as a catalyst for learning. However, several problems were identified, including overload in assessment information and the associated workload, counterproductive impact of using strict requirements and summative signals, lack of a shared understanding of the nature and purpose of programmatic assessment, and lack of supportive interpersonal relationships. Thematic analysis revealed that the success and challenges of programmatic assessment were best understood by the interplay between quantity and quality of assessment information, and the influence of social and personal aspects on assessment perceptions. Conclusion Although some of the evidence may seem compelling to support the effectiveness of programmatic assessment in practice, tensions will emerge when simultaneously stimulating the development of competencies and assessing its result. The identified factors and inferred strategies provide guidance for navigating these tensions.

show abstract

Section: Limitationsmentioning

confidence: 99%

Where the rubber meets the road — An integrative review of programmatic assessment in health care professions education

Schut¹,

Maggio²,

Heeneman³

et al. 2020

Perspect Med Educ

View full text Add to dashboard Cite

show abstract

“…One point concerned the need for data saturation for high-stakes decisions. There is some evidence that consensus amongst decision makers is independent of the number of datapoints exceeding the required minimum, suggesting that data saturation can be obtained in a given context, with a defined minimum of datapoints (de Jong et al 2019). Another point was raised on the need for psychological safety in teaching and working environments.…”

Section: Principlementioning

confidence: 99%

Ottawa 2020 consensus statement for programmatic assessment – 1. Agreement on the principles

et al. 2021

Self Cite

View full text Add to dashboard Cite

Introduction: In the Ottawa 2018 Consensus framework for good assessment, a set of criteria was presented for systems of assessment. Currently, programmatic assessment is being established in an increasing number of programmes. In this Ottawa 2020 consensus statement for programmatic assessment insights from practice and research are used to define the principles of programmatic assessment. Methods: For fifteen programmes in health professions education affiliated with members of an expert group (n ¼ 20), an inventory was completed for the perceived components, rationale, and importance of a programmatic assessment design. Input from attendees of a programmatic assessment workshop and symposium at the 2020 Ottawa conference was included. The outcome is discussed in concurrence with current theory and research. Results and discussion: Twelve principles are presented that are considered as important and recognisable facets of programmatic assessment. Overall these principles were used in the curriculum and assessment design, albeit with a range of approaches and rigor, suggesting that programmatic assessment is an achievable education and assessment model, embedded both in practice and research. Knowledge on and sharing how programmatic assessment is being operationalized may help support educators charting their own implementation journey of programmatic assessment in their respective programmes.

show abstract

“…In 2010, a programmatic approach to assessment was implemented in a 3-year competency-based veterinary curriculum in the Netherlands that was mainly organized around clinical rotations ( 11 , 23 ). Over the last decade, several scientific reports have appeared that describe in detail the different elements of the assessment program ( 4 , 23 , 27 ). In short, the program sought to motivate students to collect PRI through workplace-based assessments (WBAs).…”

Section: Context That Shaped Our Perception Of High-stakes Decision Makingmentioning

confidence: 99%

“…After 2 years and at the end of the program, two members of a competence committee independently performed a high-stakes judgment for promotion and/or licensure purposes based on their judgment of PRI across outcomes and rotations over time ( 27 ). The judgment resulted in narrative comments on each learning outcome (e.g., competency domain), reports on the strengths, points for improvement, and a grade on a 4–10 scale, with 6 or higher indicating that the student had passed.…”

Section: Context That Shaped Our Perception Of High-stakes Decision Makingmentioning

confidence: 99%

“…The judgment resulted in narrative comments on each learning outcome (e.g., competency domain), reports on the strengths, points for improvement, and a grade on a 4–10 scale, with 6 or higher indicating that the student had passed. Research indicated that the rate of consensus between the independent portfolio assessors was substantial, students collected a sufficient amount of PRI, and saturation of information within the repository was attained ( 27 ). On average, a high-stakes judgment of a given student's repository took about 45 min.…”

Section: Context That Shaped Our Perception Of High-stakes Decision Makingmentioning

confidence: 99%

See 1 more Smart Citation

“Prevention Is Better Than Cure”: A Plea to Emphasize the Learning Function of Competence Committees in Programmatic Assessment

2021

Self Cite

View full text Add to dashboard Cite

Programmatic assessment: Can we provide evidence for saturation of information?

Cited by 19 publications

References 23 publications

Where the rubber meets the road — An integrative review of programmatic assessment in health care professions education

Where the rubber meets the road — An integrative review of programmatic assessment in health care professions education

Ottawa 2020 consensus statement for programmatic assessment – 1. Agreement on the principles

“Prevention Is Better Than Cure”: A Plea to Emphasize the Learning Function of Competence Committees in Programmatic Assessment

Contact Info

Product

Resources

About