Independent Program Evaluation

Situation:

The Climate Change Fund is a significant initiative under Australia’s Net Zero strategy, aimed at delivering energy savings, reducing emissions, and building resilience to climate change to achieve Net Zero emissions by 2050. As part of this initiative, the New South Wales Government established several programs, including the Energy Savings Scheme (ESS) and the Peak Demand Reduction Scheme (PDRS). These programs aim to drive energy efficiency improvements and manage peak energy demand to reduce overall emissions and support sustainability goals.

The Independent Pricing and Regulatory Tribunal (IPART) of New South Wales, responsible for overseeing the administration of these programs, sought an independent evaluation to measure their effectiveness, efficiency, appropriateness, and overall outcomes. The evaluation was intended to inform future decision-making and program management, as well as to ensure compliance with both state and national climate policies.

Given the complexity and public accountability associated with these schemes, IPART needed an evaluation that could withstand public scrutiny, demonstrate transparency in administration, and offer clear, actionable insights.

Task:

Grosvenor Consulting was engaged to deliver a comprehensive evaluation, ensuring that the program’s administration aligned with government expectations and standards. The task involved:

• Assessing Efficiency: evaluating how well the ESS and PDRS schemes utilised their resources, including funding, human resources, and time
• Evaluating Effectiveness: determining whether the schemes met their intended outcomes, such as energy savings and reduced emissions, while also identifying any unintended outcomes
• Judging Appropriateness: assessing whether the schemes’ objectives, strategies, and activities were aligned with broader policy goals and the needs of stakeholders
• Reporting Outcomes: providing recommendations for improvement, future program design, and potential areas for policy adjustment

The evaluation also required demonstrating the delivery of an impartial and evidence-based review that would enhance IPART’s decision-making capacity while maintaining public trust.

Action:

Scoping and Planning:

To begin, Grosvenor conducted a scoping phase to align with IPART’s goals and identify key evaluation questions (KEQs). This phase involved:

• reviewing foundational documents, including the program logic models, annual plans, governance frameworks, and key performance indicators
• conducting initial consultations with IPART leadership and program teams to understand expectations and refine the scope
• mapping out the program’s life cycle to ensure that our evaluation framework aligned with both current program status and future strategic needs

Following this, detailed evaluation plan was then developed, outlining the methodology, timeline, resources, and both quantitative and qualitative data collection approaches.

Data Collection and Stakeholder Engagement:

1.      Document Review and Preliminary Analysis:

• conducted an in-depth review of over 100 documents, including policy briefs, program performance reports, financial records, and administrative data
• developed a data matrix to ensure comprehensive coverage

2.      Stakeholder Consultations:

• conducted workshops and interviews with over 50 stakeholders, including officials, program administrators, regulators, and partners
• used mixed methods (interviews, focus groups, surveys) to capture diverse perspectives, guided by tailored interview protocols

3.      Field Observations and Site Visits:

• conducted site visits to observe energy-efficient and renewable energy projects, engaging with local teams to understand operational challenges and successes

4.      Data Analysis and Synthesis:

• applied qualitative and quantitative techniques (thematic coding, statistical analysis) to synthesise data
• triangulated findings to ensure validity, reduced bias, and facilitated internal workshops to identify patterns

5.      Capacity Building:

• conducted a two-day training workshop for IPART staff on evaluation methodologies, data analysis, and interpretation
• created guides and tools to build IPART’s internal evaluation capabilities

Drafting and Report Preparation:

• developed draft reports structured around specific KEQs, with executive summaries, background, findings, and recommendations
• collaborated with IPART through multiple review rounds to align reports with strategic objectives and regulatory standards
• prepared supplementary materials i.e. presentations and briefing documents for various stakeholders, including the government and the public

Results

1.      High-Quality Evaluation Deliverables:

• delivered a set of three comprehensive evaluation reports covering all aspects of the ESS and PDRS schemes, as well as a consolidated overview of findings across both programs
• reports were structured around the KEQs, ensuring clarity and direct relevance to IPART’s needs. Each report provided evidence-backed conclusions and clear, actionable recommendations

2.      Enhanced Stakeholder Trust and Engagement:

• the use of participatory evaluation methods ensured high levels of stakeholder engagement throughout the process. This approach led to increased buy-in from IPART staff and external stakeholders, enhancing the credibility and utility of the findings
• stakeholders provided positive feedback, noting the inclusive and transparent evaluation process, which increased trust and willingness to implement recommendations

3.      Actionable Insights for Program Improvement:

• the evaluation identified several areas for improvement, including the need for more robust change management processes, enhanced data collection and monitoring practices, and a clearer alignment of program activities with strategic objectives
• key recommendations included the development of a centralised data management system, the introduction of more flexible funding arrangements, and improvements in stakeholder communication strategies

4.      Strengthened Internal Evaluation Capacity:

• Grosvenor’s capacity-building initiatives significantly increased IPART’s ability to conduct future evaluations independently. The training and tools provided were highly valued, with IPART staff reporting increased confidence and competence in evaluation skills
• IPART is now better positioned to regularly assess its programs’ effectiveness and make data-driven decisions aligned with government policy

5.      Positive Strategic Impact:

• the evaluation’s findings and recommendations have been integrated into IPART’s strategic planning processes. Several recommendations have already been implemented, resulting in more streamlined program management and improved stakeholder satisfaction
• the evaluation has also supported IPART in demonstrating its commitment to transparency and continuous improvement, strengthening its reputation as a credible regulatory body

Conclusion:

Through a methodical and participatory evaluation approach, Grosvenor successfully delivered a comprehensive evaluation that met all IPART’s requirements. The project demonstrated our expertise in government evaluations, leveraging both qualitative and quantitative methods to provide a balanced assessment of program performance. The insights provided by our evaluation have been instrumental in shaping IPART’s future strategies and improving the administration of the Climate Change Fund programs.

Published Report: Evaluation of the Safeguard Administration Program

For an independent program evaluation contact our Program Evalution team lead Dana Cross