Impact evaluation drives effective public sector change

The value of impact initiatives—purpose-driven efforts designed to create meaningful social or economic outcomes – is universally recognized by governments and non-profits. But how do organizations ensure that these public sector initiatives are genuinely effective? The answer lies in rigorous impact evaluation.

Impact evaluation is no longer just a nice-to-have; it's a critical process that helps jurisdictions forecast and anticipate the impact of programs and interventions and accurately measure outcomes.

Governments of all types have long been exploring ways to harness new types of insights that support data-driven and informed policy making. This kind of next-generation anonymized and aggregated data can be leveraged to track the impact of specific events or public realm changes across different industry segments in near real time.

Mastercard has formed public-private partnerships with governments and non-profits to measure the impact of their initiatives and investments through tools like Mastercard’s Test & Learn®. This blog highlights our work on two impact evaluation projects: New York City’s Open Streets holiday initiative and St. Louis' 911 Call Diversion program.

We also delve into the growing focus on impact evaluation, its rationale and the obstacles to overcome.

Impact initiatives are purpose-driven, organized, time-bound efforts targeting a specific social or economic outcome. Government agencies, non-profit organizations or private entities in the public sector can carry them out. As public and non-profit spending comes under more scrutiny, the ability to clearly and quantitatively showcase impact is a growing priority.

Data-driven impact measurement is critical in assessing the effectiveness of government and NGO-led social and economic programs and provides accountability to donors and taxpayers. While it may be obvious that program owners would want to know if their initiatives are broadly effective, it is even more valuable to understand the main drivers of the program’s impact. Unravelling the 'how' and 'why' behind a success or failure is more helpful than a binary 'yes' or 'no' and can help stakeholders adjust programs to deliver more optimal outcomes.

Take, for example, NYC’s Open Streets holiday initiative in which select streets were closed off to vehicular traffic to boost sales at local businesses during certain days in December. Impact evaluation revealed the potential increase in spending resulting from the initiative and the drivers behind incremental spending for businesses from pedestrian traffic. Moreover, the evaluation confirmed that the extended pedestrianization hours led to higher sales.

The City of St. Louis was able to reduce dispatcher and officer hours spent on behavioral health cases with its 911 Call Diversion Program. Non-violent mental and behavioral health calls are routed to crisis counselors rather than police officers. The city effectively identified the program's value by leveraging a rigorous test vs. matched control methodology.

Here are four ways that impact evaluations can show the value of these types of initiatives:

• Evidentiary insights: Empirical evidence can show what works, what doesn't, and under what conditions.
• Sustainability assessment: The longer-term sustainability of initiatives can also be measured, providing more confidence in long-term outcomes.
• Improved accountability: Justify the financial investment in an initiative and help with future fundraising.
• Learning and decision-making: Individual and organizational-level learning is encouraged and allows decision-makers and donors to make informed decisions.

The need for impact evaluation is straightforward. Still, several obstacles stand in the way of broader adoption:

• Funding & resource constraints: Impact evaluation studies can be seen as a ‘nice-to-have', leading to challenges in securing funding for data collection and data analysts.  
• Data availability & quality: Real-world data can often be noisy, low-quality and dated. Governments and NGOs often face difficulties in collecting accurate data that can be useful in assessing impact.  
• Time lag between intervention & impact: Interventions for public impact may take time to yield benefits, resulting in long delays before evaluations can be conducted and the benefit of the initiative assessed.
• Establishing baseline performance: The value of an intervention can only be determined if there is a clear understanding of the status quo, but establishing true ‘test’ and ‘control’ groups may not always be feasible.

Due to these challenges, governments and non-profits often struggle to measure the impact of their initiatives, preventing them from accurately forecasting impact and creating compelling business cases for essential funding.

While New York City had a sense that its holiday season pedestrianization effort was a success, conducting a comprehensive impact evaluation study wasn’t easy. The government tried to survey the local merchants, but the data gathered was sparse and biased. The uniquely commercial neighborhood setting made it even more difficult to analyze the initiative.

Similarly, when the City of St. Louis tried to test the effectiveness of its mental and behavioral health call diversion program, operators were unable to construct a reliable methodology to gauge effectiveness. Instead, officials had to rely primarily on anecdotal feedback, which lacked the necessary rigor to convince elected officials to renew funding.

Impact assessments can be more straightforward with a structured approach, spanning objective setting, intelligence, optimization and design, and the buy-in of key stakeholders and decision-makers throughout. Building processes and partnerships can help any public sector or not-for-profit customer become a sophisticated impact evaluator.

The first step in any effective impact evaluation program is to identify the organization’s primary objectives. Input from program owners, sponsors and funders can enable an evaluation design that prioritizes the most relevant focus areas. Answering a few key questions can help define organizational objectives:

• What are the program’s goals?  
• What is the expected timeframe for these goals?
• Are there other concurrent initiatives planned?

The next step is identifying indicators of success. These can be short-term metrics, such as real-time data that would enable immediate quantification of impact, or long-term indicators that can provide a view into systemic change over an extended timeframe. These long-term indicators can include housing rates, job program success and economic development indicators.

Stakeholder involvement in setting objectives and performance metrics can differ depending on the organization's size and structure. In small organizations, stakeholders with signatory authority, such as executive directors, can champion the organization's commitment to evaluation. However, in larger organizations, data collection teams should partner with internal teams to collect accurate and unbiased data. Some organizations may even have dedicated learning & impact teams responsible for designing, executing and interpreting evaluation initiatives.

An assessment of the data available to the organization, internally and through external sources, must be conducted to evaluate any gaps in the dataset. Through the Intelligence phase, Mastercard’s insights database and location assets could complement the initiative owner’s data for a more comprehensive evaluation. Some of these unique datasets include the Inclusive Growth Score™, which provides local planners, governments and impact investors with a clear, simple view of social and economic indicators at the neighborhood level.

As a part of Optimization, initiative owners will then use the data identified in the intelligence phase to perform short and long-term impact assessments through data analytics. Depending on the research objectives, this can take the form of testing and optimization through platforms such as Mastercard’s Test & Learn® or through location-based insights that marry publicly available metrics segmented geographically with spend insights.

The outcomes of the analysis should then guide the future strategy of the government or not-for-profit organization. Through the Design phase, impact initiatives can be further tailored and prioritized for funding to ensure they serve their communities and meet the organizational impact objectives.

Through this multi-step framework, Mastercard continues to partner with organizations in its communities to help them successfully evaluate their impact. These partnerships have helped non-profits and government organizations effectively drive impactful results and design future impact initiatives.

As a part of Optimization, initiative owners will then use the data identified in the intelligence phase to perform short and long-term impact assessments through data analytics. Depending on the research objectives, this can take the form of testing and optimization through platforms such as Mastercard’s Test & Learn® or through location-based insights that marry publicly available metrics segmented geographically with spend insights.

The outcomes of the analysis should then guide the future strategy of the government or not-for-profit organization. Through the Design phase, impact initiatives can be further tailored and prioritized for funding to ensure they serve their communities and meet the organizational impact objectives.

Through this multi-step framework, Mastercard continues to partner with organizations in its communities to help them successfully evaluate their impact. These partnerships have helped non-profits and government organizations effectively drive impactful results and design future impact initiatives.