Six tips to understanding and measuring policy impact

Measuring the impact of policies

If you’re working in programs (design or evaluation) and the performance management space, it is likely that you’ve heard this message consistently over the last few years. It’s a message that is being put forward by the Commonwealth (under the PGPA), agencies such as the ANAO and something that has been picked up and written about within the private sector/those interested in better practice.

And the industry seems to be getting the message. There appears to have been a definite increase in the understanding, willingness and intention to measure impacts, particularly within the program space.

While we’d argue that there has been a marked improvement in the measurement of program and service impacts, challenges still remain when it comes to policy. How to effectively measure the impacts of policy work is a challenging question being faced by evaluators and performance management experts.

Why are people afraid of measuring policy impacts?

If you start speaking to someone about policy performance, it is likely that you will get a broad range of reactions. From people who are motivated to try but don’t know where to start, to those that say it just isn’t possible or practical.

In our work, we hear a range of common justifications about why policy performance isn’t being measured:

> the outcomes of policy work are unclear – how can we measure what we don’t know?

> the performance of an organisation can’t be judged on the outcomes and impacts of policy work – we only write the policy and have little say in what is actually delivered/done

So how can you do it?

As with anything in the program evaluation and performance management sector there is no one size fits all approach to understanding policy impacts. But the key is to start the discussion and start trying.

1. Understand the purpose and intention of the policy

Policy work is another means of achieving an outcome. As with programs, each piece of policy is established with a particular aim or objective in mind. Understanding the intent behind the policy is the first step to being able to measure your impacts

2. Environment scanning

A common argument against measuring the impacts of policy work focuses on the number of external factors influencing what can be achieved. This is certainly something that must be taken into account, but isn’t a reason not to try. By conducting a market scan and understanding what else is happening in the market/public sphere that can influence the policy outcomes

3. Develop a program logic to understand how the policy is intended to work

Thinking about the inputs, activities and outcomes of your work is a great way to identify and plan to measure your policy impacts. This involves considering the intended causal chains that underpin the policy:

What resources are you using? What activities are you undertaking with these resources? What change will occur as a result of these resources? Why do you think this will be successful?

Where you are one of many actors working in the space it can be helpful to think of this holistically (i.e. across agencies) then work out exactly where you fit, and how far your reach extends.

4. Be realistic about the boundaries of your organisation and policy work

If you work in an enabling agency or area, it is inevitable that your outcomes will be different to those actually rolling out initiatives – that’s ok! Consider what success looks like for your area and policy. You may not be able to say that you directly save lives through your policy work, but you are probably making sure that those that do have the resources and information they need.

5. Focus on what you need to assess YOUR outcomes – it is ok to ignore the bigger picture for now

Once you have your logic and have considered where and how your policy or agency fits, you will be ready to start planning to measure impact. Think about what you would see/experience if your part of the policy was totally successful.

What changes would have occurred?

What impact would this have on attitudes, knowledge and behaviours?

This will give you a good head start on unpacking what data and information you need to collect and analyse to confirm whether this was realised in practice.

6. Work with related agencies and organisations

If you know that policy success will ultimately require input and action from other agencies, start a performance conversation with them now! Understand how they define policy success and what evaluation/monitoring activities they will be undertaking. While you may not be responsible for the outcomes of the policy, it is always useful to know what your work is ultimately leading/contributing to.

It is unlikely that you will get the measure right the first time, but there but there is a great deal of benefit to be gained from even attempting to measure impact. If you’re still not sure where to start, get in touch – we’d love to help. Reach out to our Public Sector Advisory Associate Director Dana Cross