SROI as a Method for Evaluation Research: Understanding Merits and Limitations

Abstract

Please direct correspondence to florentine.maier@wu.ac.at.Abstract: NPOs and their funders are increasingly drawn to the Social Return on Investment (SROI) method to evaluate the social impact of programs, organizations or organization networks. While many claims about the benefits of SROI have been expressed, various points of criticism have also been raised.On the basis of both current research and our own experience in conducting SROI analyses, we develop a comprehensive assessment of this method, which is structured along two dimensions: the observer's paradigmatic perspective, on the one hand, and positive or negative valuation, on the other. We identify two major merits: SROI analysis can provide legitimacy to NPOs or their funders, and it can assist in allocating resources. We identify limitations from three perspectives: From an interpretative-sociological perspective, criticism of commensuration and utilitarianism calls the method as a whole into question.From a technical-instrumental perspective, there are a number of difficulties that could however be overcome as the method matures. From an intermediary perspective, a number of limitations become apparent that, while inherent to SROI analysis, are no reason for abandoning it, as long as they are thoroughly understood. We conclude by providing suggestions for the responsible use of SROI analysis.The question of how to advance the common good is increasingly seen as a matter of finding out and implementing "what works" rather than of following ideologies. Among NPOs and government institutions that consider themselves cutting-edge, the formulation of explicit theories of change (see, for example, Fulbright-Anderson & Auspos, 2006) as well as the regular development of evidence-based policies (Pawson, 2006:vii) are expected. In managing performance, the focus has clearly shifted from inputs and outputs to impacts and public value (Barman, 2007; Lynch-Cerullo & Cooney, 2011; Moore, 2013). On the one 1 hand, these developments are to be welcomed as progress towards more open-mindedness and discursive rationality. On the other hand, the ideology of anti-ideology comes with its own trappings. The old quip that "[p]ractical men, who believe themselves to be quite exempt from any intellectual influence, are usually the slaves of some defunct economist" (Keynes, 1965:383) still applies. The present article contributes to a deeper understanding of this dilemma by focusing on a particular method that is widely used in the search for impact and evidence: the analysis of Social Return on Investment (SROI). It is the aim of this paper to contribute to a balanced and comprehensive understanding of its merits and limitations.Out of more than 40 approaches that have been developed for measuring social impact (Stevenson, Taylor, Lyon, & Rigby, 2010), SROI analysis is one of the most widespread. It is a mixed-method-approach to assess the social, economic, and environmental impact of interventions. Its most prominent feature is the SROI ratio. This ratio aims to dete...

Measuring Social Return on Investment

“…A comprehensive, field‐vetted list of best practices for SROI measurements is not possible at this point, given both the limited use of these measurements and the even more limited systematic evaluation and research analysis of them. Still, the challenges and lessons learned from this research, as well as from advice given in previous research on SROI measures (for example, Arvidson et al ; Maier et al ) point us toward a number of best practices for organizations and social investors who desire to utilize SROI techniques. We are not making a recommendation for or against the use of SROI as a measure of performance.…”

“…The challenge is to measure that social return in effective, valid, and reliable ways that can provide a common language and metric for comparing various potential social impact ventures. There are many different methods available for performance and social impact measurement, although none has become widespread in the venture philanthropy field, and most users of these methods continue to struggle with the specific techniques for assessing social value (Maier et al ).…”

“…Yet more challenges noted in previous assessments include overcoming resource and capacity limitations and the lack of incentives to implement SROI measures (Barman and MacIndoe ), determining who is responsible to invest the time and resources necessary to ensure data integrity (Bugg‐Levine and Emerson ; Maier et al ), and dealing with inadequate commitment among stakeholders to gather and track necessary information (Tuan ). In fact, REDF, the pioneer in SROI methodology, now acknowledges the complexity and limitations of the existing approaches and even discourages traditional SROI analysis for most organizations (Gair ).…”

The social impact of advice during disability welfare reform: from social return on investment to evidencing public value through realism and complexity

“…Some authors (e.g. Maier et al 2015;Banke-Thomas et al 2015) suggest that if methodological issues can be addressed then it may be possible to arrive a rigorous and reliable ratio that is representative of social impact. Methodological SROI problems include understanding what causes social impacts, as outcomes may be influenced by different, overlapping factors that are difficult to disentangle (Moody, Littlepage, and Paydar 2015).…”

Ambiguity in the Attribution of Social Impact: A Study of the Difficulties of Calculating Filter Coefficients in the SROI Method

“…To cope with this and determine the real impact of the project, filter coefficients are introduced. However, their estimation has some shortcomings [4,26,27]. Even though the filters are perfectly defined within the SROI methodology, when we fix them at a certain percentage, we cannot find a method or reference that enables these filters to be calculated with precision and ease [28][29][30].…”