Considering Unintended Consequences: Evidence From Recent Evaluations of U.S. Foreign Assistance Programs

Abstract

This article examines factors that could have influenced whether evaluations of U.S. government–funded foreign assistance programs completed in 2015 had considered unintended consequences. Logit regression models indicate that the odds of considering unintended consequences were increased when all or most of seven standard data collection methods had been used, and there were some agency effects; however, no associations were found with evaluation type (impact vs. performance), timing, or whether evaluations were conducted by external evaluators. These results suggest that some proportion of the evaluations that did not consider unintended consequences would have considered such consequences if they had employed more data collection methods. Moreover, few of the evaluations that considered unintended consequences provided many details about their investigative methodologies or the consequences themselves, which raises further concerns that users might not have received information that would have helped for learning or accountability purposes.

Assessing International Development Cooperation: Becoming Intentional about Unintended Effects

“…This has been reaffirmed in the latest guidance of 2019, which states that evaluations focusing on impact ought to assess "The extent to which the intervention has generated or is expected to generate significant positive or negative, intended or unintended, higher-level effects" [10]. However, the little evidence available indicates that these guidelines are not followed systematically [16]. For example, a meta-evaluation of the United States Agency for International Development (USAID) monitoring and evaluation reports shows that only in 15% of the evaluations from 2009 to 2012 were unintended effects taken into consideration [17].…”

“…For example, a meta-evaluation of the United States Agency for International Development (USAID) monitoring and evaluation reports shows that only in 15% of the evaluations from 2009 to 2012 were unintended effects taken into consideration [17]. This percentage rose slightly from 2012 to 2015 according to de Alteriis [16]. An assessment of the evaluations of the Norwegian aid agency (NORAD) showed just 40% of Terms of References (ToRs) mentioned unintended effects.…”

“…While these categorizations help to differentiate unintended effects once they are identified, they do not facilitate the identification of potential unintended effects in a systematic way. As a consequence, current overviews of unintended effects of international cooperation efforts in evaluations not only report a paucity of unintended effects (e.g., de Alteriis found 49 distinct unintended effects in 1369 monitoring reports and evaluations [16]) but also display a certain randomness and superficiality. Researchers often accidentally come across unintended effects in their evaluations and report these in a few lines towards the end of a report.…”

“…Researchers often accidentally come across unintended effects in their evaluations and report these in a few lines towards the end of a report. They had neither planned to investigate them systematically nor had they developed appropriate research methods [7,16]. Intentionality about identifying unintended effects is necessary to address this.…”

Leading and Learning through Dynamic Performance Management in Government

“…We preface our discussion by acknowledging that since federal organizations are complex and open systems, in order for them to achieve their missions, their performance management systems need to extend beyond measuring the success of organizational components. They must deliberately identify, measure, and monitor expected linkages among internal actions and desired outcomes; measure and monitor synergies and emergencies among outcomes; predict, monitor, and respond to changes in ecosystems; and deliberately measure unintended consequences of agency actions to dynamically adapt strategy to ecosystem changes (Bamberger, Tarsilla, and Hesse‐Bibler ; de Alteriis ). Given the rapid change in intelligence brought on by “big data” and by technologies and tools that allow for data integration and analytics, as well as the many positional leaders charged with overseeing different analytical silos (e.g., chief performance improvement officers and chief data officers), we articulate here how centralized and strategically located leaders—knowledge brokers—might build and sustain linkages and synergies among the many players and data sources in the new normal of federal organizational performance.…”