6 min read
Measuring impact is hard.
Why is this so?
Impact means different things to different people. Every different sector – including non-profits, impact investors, foundations and corporations – have differing views on the definition of impact. While these perspectives differ in many ways, a few themes are common:
Impact is subjective.
Impact includes non-financial metrics.
Impact is hard to aggregate.
Impact takes a long time.
Maximising Social Impact
“Impact management and measurement is one of the most common pain points we hear from our members”, says Allison Hollowell, Chief Strategy Officer for AVPN. “It is a complex challenge for all sectors to grapple with whether they are investing for impact returns only, for a combination of impact and financial returns, and/or to understand the intersection of sustainable impact and business value”.
AVPN understands that maximising social impact requires continual awareness and work. As part of its ongoing efforts, AVPN, together with members Investing in Women and Mekong Economics, held two workshops on Measuring Impact in Vietnam, in Hanoi and Ho Chi Minh City.
Why Vietnam?
“There is an increasing need for local capital to play a role in shaping the social sector. Vietnam is very early in its development of philanthropy, impact investment, and strategic CSR. In addition, since the country has developed into a middle-income country traditional foreign aid is exiting. There is a gap and we need to fill it in with knowledge and expertise to ensure social impact does not lag behind” affirms Hollowell.
AVPN experienced first-hand this hunger within Vietnam to learn more about impact. Its two workshops – were filled with representatives from all sectors. There is a wish to better understand long-term work and apply impact measurement across all areas.
Corporations want to ensure that their CSR translates into durable and sustainable impact. Impact investors witness enormous opportunities, particularly given the high number of SMEs in the country that take in account social and environmental factors in their businesses. Non-profits and foundations are looking to guarantee the long-term viability of their organisations by ensuring they have the correct metrics in place. “Those working in the Vietnamese market are keen to understand the impacts that their investments have, since there is an increasing recognition that organizations that include and work on social and environmental impact have a higher rate of success”, confirms Hollowell.
But Where to Start?
“With buckets of common sense!”, exclaims Adam McCarty from Mekong Economics , adding that before one starts measuring, one needs to understand clearly what the desired impact is. “Impact is why you do things. Impact is the end of the ‘so what’ causal chain story. It is the end consequences of some action or investment. Simply put, it is the reason for the investment of time and money”. It has to answer the question whether the investment provides additionality by increasing social outcomes beyond what would have otherwise occurred.
Story of Causation
In fact, one should work backwards, starting with the impact and answering the question, “What is the problem, the opportunity or the change needed?”, punctuates McCarty. Then move into filling ‘the causal chain’ by using a logical framework (log frame) of activities: the outputs with key numbers, and outcomes on what actually happened (see example provided by trainer).
Added to this are questions of attribution, assumptions, proxies and other effects that might occur through your investment. McCarty highlights an important issue that is often forgotten: the ‘funnel of attrition’. “It is common knowledge that approximately 50% of start-ups fail. We also need this insight into this kind of investment, since we can learn from failure to tweak or improve our work”.
Theory of Change
This type of ‘log frame’ can be built into a descriptive and illustrative ‘Theory of Change’ diagram to show the changes that are expected in a particular context.
“It is focused in particular on mapping out or “filling in” what has been described as the “missing middle” between what a program or change initiative does (its activities or interventions) and how these lead to desired goals being achieved”, McCarty adds. “I am a big fan of this process. Frameworks are poor substitutes for careful specification of a theory of change”.
More than M&E
Many view evaluation and monitoring as the measurement tool for impact. While this is often an important part of checking you are on the right path to impact, it is not impact. “Impact is the net good”, McCarty emphasizes, “It is how you compare what changed because of your ‘investment’ – by comparing the baseline with the end line”.
McCarty is clear that evaluation can be expensive but needs to be proportional to the size of a project: “Big money requires higher levels of data”. He then points out that the smaller the investiment “pot” the few KPIs are necessary. In his opinion, chosen KPIs, “should flow logically and sensibly from a causal story, not try to exaggerate the impact of the project. KPIs should take into account the additionality, and they should be SMART (Specific, Measurable, Assignable, Realistic, Time-Related)”. It is often easier to measure inputs than outputs.
Hollowell adds to this that “the most important challenge for impact measurement is moving from measuring outputs (number of students educated) to outcomes (improved learning or increased job placements). We can use these outcomes as a proxy to signal that better education outcomes or access to better jobs will lead to improved livelihoods in the long-term”.
A range of different measurement tools are available, depending on the needs of various sectors and what one wants to measure. For example, impact investors have a range of tools available such as IMP and the updated GIIN’s IRIS+, among other options. These resources are helpful to structure measurement frameworks and generate ideas for what indicators can be used for measuring impact, drawing on the aggregated research of many other impact investors.
Gender-Lens Investing
A strong component of the agenda brought by the lead event partner Investing in Women is the importance of viewing impact investments through a gender-lens. This strengthens access to finance for women-owned and led SMEs. “AVPN is taking a deeper dive into certain thematic areas and has partnered with Investing in Women. Gender-sensitive impact investing is not a particularly well-developed topic in Vietnam, hence we are able to educate wider groups of stakeholders about the importance of integrating gender into different learning opportunities. We are promoting the integration of gender issues into causal chains and the design of data collection”, Hollowell adds.
And How Did It End?
With countless feelings!
Many attending the two workshops are at different points along their impact journey. Overall, comments ranged from “relieved that it is not rocket science” to “I am encouraged that we are on the right track”. The workshops also generated additional questions such as “how can we streamline and systemise collection of this data without adding more work?”; to honest viewpoints for instance, “I am still struggling with the concept but understand that it is difficult but not impossible”.
Hollowell understands these comments and hopes that all participants take away one key message: “Measurement is the end of the journey for Impact Management. Many want to jump directly into indicators and metrics without properly understanding the bigger picture. It is critical to develop a clear purpose or justification for your impact work (‘the why?’) and then work backwards from there on a ‘theory of change’ (aka strategy or causal story) to understand the building blocks to get from status quo to achieving impact”.
One key and lasting comment was that the course was attended by so many participants from diverse sectors which demonstrates a collective interest; that impact is not for one sector alone; and bodes well for continued long-term engagement.
AVPN looks forward to pursuing these topics and to strengthening learning in this important area. There is an AVPN / GIIN webinar on the 13th December 2019 on the topic of IRIS+ using Gender Lens Investing as a thematic focus. Please see further details here.
