From theory based policy evaluation to SMART Policy Design: Lessons learned from 20 ex-post evaluations of energy efficiency instruments

Mirjam Harmelink and Robert Harmsen, Ecofys Netherlands, The Netherlands
Lars Nilsson, Lund University, Environment and Energy Systems Studies, Sweden

Keywords

ex-post evaluation, theory based policy evaluation, energy efficiency, policies

Abstract

This article presents the results of an in-depth ex-post analysis of 20 energy efficiency policy instruments applied across different sectors and countries. Within the AID-EE project, we reconstructed and analysed the implementation process of energy efficiency policy instruments with the aim to identify key factors behind successes and failures. The analysis was performed using a uniform methodology called “theory based policy evaluation”. With this method the whole implementation process is assessed with the aim to identify: (i) the main hurdles in each step of the implementation process, (ii) key success factors for different types of instruments and (iii) the key indicators that need to be monitored to enable a sound evaluation of the energy efficiency instruments.

Our analysis shows that:

Energy efficiency policies often lack quantitative targets and clear timeframes;
Often policy instruments have multiple and/or unclear objectives;
The need for monitoring information does often not have priority in the design phase;
For most instruments, monitoring information is collected on a regular basis. However, this information is often insufficient to determine the impact on energy saving, cost-effectiveness and target achievement of an instrument;
Monitoring and verification of actual energy savings have a relatively low priority for most of the analyzed instruments.
There is no such thing as the ‘best’ policy instrument. However, typical circumstances in which to apply different types of instruments and generic characteristics that determine success or failure can be identified.

Based on the assessments and the experience from applying theory based policy evaluation ex-post, we suggest that this should already be used in the policy formulation and design phase of instruments. We conclude that making policy theory an integral and mandated part of the policy process would facilitate more efficient and effective energy efficiency instruments.