Carbon Farming Futures Evaluation

Situation:

Our previously implemented the Carbon Farming Futures (CFF) Grant Program. This included three sub-Programs which, collectively, worked to support the identification, testing and extension of agricultural practice to reduce and/or abate carbon emissions.:

• Filling the Research Gap – designed to fund research into new technologies and processes
• Action on the Ground – designed to support field testing of technologies and processes
• Extension and Outreach – designed to support the promotion of new technologies and practices, encouraging uptake and use within the sector

Over $140M of funding was provided to approximately 200 projects across the three initiatives.

Task:

Grosvenor was engaged to conduct a summative evaluation of the overarching CFF Program, considering suitability of administration and design, as well as outcome realisation and impacts. Given the design of Carbon Farming Futures, the evaluation needed to consider performance and outcomes of each of the 200 projects, three sub-Programs and the overall CFF Program.

Actions:

A theory of change approach was applied to the design and delivery of the evaluation, incorporating mixed-methods data collection and analysis. Given the broad scope of the Carbon Farming Future Programs, the establishment of a shared and comprehensive program understanding (depicted in program logic models) was critical to the successful design and delivery of this evaluation.

Grosvenor worked in close collaboration with the Department to design a mixed-methods evaluation strategy with activities tailored to the overarching CFF Program and each sub-Program. A comprehensive evaluation framework was developed, incorporating:

• evaluation methodology and scope, detailing the purpose of evaluation activities at a project, program and CFF level. This informed development of Key Evaluation Questions to guide evaluation delivery
• nested program logics which demonstrated inputs, activities, outputs and outcomes for CFF as a whole and each sub-Program. These were designed to demonstrate the relationship between the three funding initiatives as well as how they each contributed to overall Program objectives
• data matrix detailing success factors, measures, indicators and data sources to inform analysis of the key evaluation questions. This was established specific to each sub-Program and the overarching CFF Program
• stakeholder analysis and consultation plan, detailing approaches to engagement that would enable the collection of data while minimising burden on stakeholders who had been involved across the suite of programs

A mixed method approach was applied to deliver each of the evaluation activities incorporating:

• qualitative data collection through consultation with key stakeholders from the Department, individual projects and broader agricultural sector
• desktop review of project and program level documentation, with data coded to enable quantitative analysis of performance, outcomes and key themes at a project, program and CFF level.

Evaluation activities occurred across a 15-month period.

Results:

Through delivery of the evaluation, Grosvenor was able to present evidence-based findings, conclusions and recommendations relevant to each of the nested programs as well as the overarching CFF Program. We were able to provide the Department with guidance on how to enhance data capture for future Grants Programs to streamline analysis processes. The client expressed a high level of satisfaction with the Grosvenor team, providing an NPS rating of 10/10.

Grosvenor was subsequently reengaged by the Department to evaluate other agricultural grants programs.

Conclusion:

The summative evaluation of the Carbon Farming Futures (CFF) Funding Program, conducted by Grosvenor, successfully provided the Department with a comprehensive understanding of the program’s design, administration, and outcomes. By applying a theory of change approach and using mixed-methods data collection, Grosvenor was able to evaluate the performance of each of the 200 projects and three sub-programs, as well as the overall CFF initiative. The use of detailed program logic models, a tailored evaluation framework, and a stakeholder consultation plan enabled a thorough analysis of key evaluation questions and outcomes. The Department highly valued the evidence-based insights and actionable recommendations provided, particularly regarding improvements in data capture for future grants. The success of this evaluation, reflected in a 10/10 NPS rating, resulted in Grosvenor being reengaged for additional evaluations, further solidifying the firm’s reputation for delivering high-quality, impactful evaluations in the agricultural sector.

 

For an independent program evaluation contact our Program Evalution team lead Dana Cross