Assessment of Evaluation Tools for Integrated Surveillance of Antimicrobial Use and Resistance Based on Selected Case Studies

Abstract

Regular evaluation of integrated surveillance for antimicrobial use (AMU) and resistance (AMR) in animals, humans, and the environment is needed to ensure system effectiveness, but the question is how. In this study, six different evaluation tools were assessed after being applied to AMU and AMR surveillance in eight countries: (1) ATLASS: the Assessment Tool for Laboratories and AMR Surveillance Systems developed by the Food and Agriculture Organization (FAO) of the United Nations, (2) ECoSur: Evaluation of Collaboration for Surveillance tool, (3) ISSEP: Integrated Surveillance System Evaluation Project, (4) NEOH: developed by the EU COST Action “Network for Evaluation of One Health,” (5) PMP-AMR: The Progressive Management Pathway tool on AMR developed by the FAO, and (6) SURVTOOLS: developed in the FP7-EU project “RISKSUR.” Each tool was scored using (i) 11 pre-defined functional aspects (e.g., workability concerning the need for data, time, and people); (ii) a strengths, weaknesses, opportunities, and threats (SWOT)-like approach of user experiences (e.g., things that I liked or that the tool covered well); and (iii) eight predefined content themes related to scope (e.g., development purpose and collaboration). PMP-AMR, ATLASS, ECoSur, and NEOH are evaluation tools that provide a scoring system to obtain semi-quantitative results, whereas ISSEP and SURVTOOLS will result in a plan for how to conduct evaluation(s). ISSEP, ECoSur, NEOH, and SURVTOOLS allow for in-depth analyses and therefore require more complex data, information, and specific training of evaluator(s). PMP-AMR, ATLASS, and ISSEP were developed specifically for AMR-related activities—only ISSEP included production of a direct measure for “integration” and “impact on decision making.” NEOH and ISSEP were perceived as the best tools for evaluation of One Health (OH) aspects, and ECoSur as best for evaluation of the quality of collaboration. PMP-AMR and ATLASS seemed to be the most user-friendly tools, particularly designed for risk managers. ATLASS was the only tool focusing specifically on laboratory activities. Our experience is that adequate resources are needed to perform evaluation(s). In most cases, evaluation would require involvement of several assessors and/or stakeholders, taking from weeks to months to complete. This study can help direct future evaluators of integrated AMU and AMR surveillance toward the most adequate tool for their specific evaluation purpose.

One Health Evaluation of Antimicrobial Use and Resistance Surveillance: A Novel Tool for Evaluating Integrated, One Health Antimicrobial Resistance and Antimicrobial Use Surveillance Programs

“…Our method and tool did not attempt to comment on actions or next steps for the specific AMR/AMU programs, as our focus was on the overall development and sustainability of programs to achieve effective integrated monitoring. The utility and feasibility of 12 other tools for surveillance or One Health evaluation were assessed by the CoEval-AMR Network project team since 2019 ( 28 ). These included AMR/AMU surveillance tools previously reviewed by Nielsen et al ( 29 ) using 10 criteria to rank the utility and feasibility of the frameworks.…”

Evaluating Integrated Surveillance for Antimicrobial Use and Resistance in England: A Qualitative Study

“…It was also highlighted that work was ongoing to explore options to use AMR data from animals generated by private laboratories. Currently, these data are not accessible and only some resistance data from small animal private laboratories are collected through the Small Animal Veterinary Surveillance Network (SAVSNET) 7 . The following quote illustrate this: "The private labs, for example, we are looking into developing that and work more closely with the private sector and explore options to work more with them, because obviously there are private labs that do testing for resistance.…”

“…These tools range in application from assessing laboratory capacity in the surveillance system to the assessment of governance, but none of them provides a focussed approach for the evaluation of OH AMU/AMR surveillance. A recent study documented the strengths and weaknesses of six evaluation tools namely, SurvTools, NEOH, ISSE, ATLASS developed by the Food and Agriculture Organization (FAO), the Progressive Management Pathway tool on AMR (PMP-AMR) developed by the FAO, and the Evaluation of Collaboration for Surveillance (ECoSur) tool (7). These tools were applied to a range of case studies 1 https://guidance.fp7-risksur.eu/.…”

Data_Sheet_1.DOCX

“…PMP-AMR, ATLASS, ECoSur and NEOH are evaluation tools that provide a scoring system to obtain semi-quantitative results, whereas ISSE and SurvTools would result in a plan for how to conduct evaluation(s). The NEOH and ISSE were perceived as the best tools for evaluation of OH aspects, and ECoSur as best for evaluation of the quality of collaboration (7). Depending on the evaluation questions, assessors will need to select a tool that suits their needs (7).…”

“…The NEOH and ISSE were perceived as the best tools for evaluation of OH aspects, and ECoSur as best for evaluation of the quality of collaboration (7). Depending on the evaluation questions, assessors will need to select a tool that suits their needs (7). To guide users in choosing a suitable evaluation tool, an international network of scientists developed guidance for choosing an assessment approach from an inventory of tools suitable for evaluating integrated AMU and AMR surveillance systems in the project "Co-Eval-AMR-Convergence in evaluation frameworks for integrated surveillance of AMU and AMR."…”