Presenting a comprehensive multi-scale evaluation framework for participatory modelling programs: A scoping review

Abstract

Introduction Systems modelling and simulation can improve understanding of complex systems to support decision making, better managing system challenges. Advances in technology have facilitated accessibility of modelling by diverse stakeholders, allowing them to engage with and contribute to the development of systems models (participatory modelling). However, despite its increasing applications across a range of disciplines, there is a growing need to improve evaluation efforts to effectively report on the quality, importance, and value of participatory modelling. This paper aims to identify and assess evaluation frameworks, criteria, and/or processes, as well as to synthesize the findings into a comprehensive multi-scale framework for participatory modelling programs. Materials and methods A scoping review approach was utilized, which involved a systematic literature search via Scopus in consultation with experts to identify and appraise records that described an evaluation framework, criteria, and/or process in the context of participatory modelling. This scoping review is registered with the Open Science Framework. Results The review identified 11 studies, which varied in evaluation purposes, terminologies, levels of examination, and time points. The review of studies highlighted areas of overlap and opportunities for further development, which prompted the development of a comprehensive multi-scale evaluation framework to assess participatory modelling programs across disciplines and systems modelling methods. The framework consists of four categories (Feasibility, Value, Change/Action, Sustainability) with 30 evaluation criteria, broken down across project-, individual-, group- and system-level impacts. Discussion & conclusion The presented novel framework brings together a significant knowledge base into a flexible, cross-sectoral evaluation effort that considers the whole participatory modelling process. Developed through the rigorous synthesis of multidisciplinary expertise from existing studies, the application of the framework can provide the opportunity to understand practical future implications such as which aspects are particularly important for policy decisions, community learning, and the ongoing improvement of participatory modelling methods.

Leveraging artificial intelligence and software engineering methods in epidemiology for the co-creation of decision-support tools based on mechanistic models

“…As academic models are not directly usable as DST by non-modeller end-users, participatory methods (Voinov et al, 2018; Voinov and Bousquet, 2010) aim at the co-design of a model and the associated DST at the same time, by modellers and stakeholders, for a given practical question, often resulting in an improved engagement by stakeholders in the process and higher confidence in model outcomes (Lee et al, 2022). This however results in a model that focuses on a narrow application-oriented question, e.g.…”

“…However, installing and configuring those platforms, developing epidemiological models usable in real situations, and running simulations with relevant outputs again requires technical skills in modelling and computer science which makes them difficult to be used by health managers in an autonomous way. As academic models are not directly usable as DST by non-modeller end-users, participatory methods (Voinov et al, 2018;Voinov and Bousquet, 2010) aim at the co-design of a model and the associated DST at the same time, by modellers and stakeholders, for a given practical question, often resulting in an improved engagement by stakeholders in the process and higher confidence in model outcomes (Lee et al, 2022). This however results in a model that focuses on a narrow application-oriented question, e.g.…”

An overview of the Ugandan food system: outcomes, drivers & activities

“…The presence of aflatoxins in these staples contributes to health issues, such as liver cancer and stunting, while also causing economic losses due to rejected exports and reduced market values. The country has lost 577 million USD annually as a result of around 3,700 aflatoxin-induced liver cancer cases (Lee et al, 2022).…”

Assessing Approaches to Strengthen Monitoring and Evaluation in Development Projects: Evidence from Donor-Funded Projects in Machakos County, Kenya

“…The study draws on data from literature and experiences from youth-led evaluations in Oaktree (Australia's largest youth-led organization focusing on international development). The study calls for greater youth participation and involvement in development programme evaluations and urges communities to work with young people to advance the important role of youth participation in M & E. Lee et al (2022) examines how participatory frameworks influence evaluations. In a scoping review, the study assesses how evaluation frameworks and criteria are influenced by participatory approaches.…”