Ex-ante impact evaluations of energy efficiency measures – how to increase their transparency

Panel: 4. Monitoring and evaluation for a wise, just and inclusive transition

This is a peer-reviewed paper.

Authors:
Barbara Schlomann, Fraunhofer Institute for Systems and Innovation Research, Germany
Iska Brunzema, Fraunhofer Institute for Systems and Innovation Research ISI, Germany

Abstract

Both at European and national level, the requirements for reporting on measures introduced to achieve EU-wide and national energy efficiency and climate targets have increased considerably in recent years. While ex-post impact evaluation of policies and measures is nationally as well as internationally rather well documented, ex-ante reporting processes are still very inconsistent. The goal of this presentation is to develop a consistent methodology for a good practice for ex-ante impact evaluations, which provides a common methodological guideline for the design and monitoring of energy efficiency and climate policy measures and programmes.

In our presentation, we derive the key factors to be taken into account for a transparent ex-ante impact evaluation of energy efficiency policies. The following factors will be discussed:

1. Relevant assessment criteria:

The most important assessment criterion for an ex-ante evaluation of energy efficiency policies is their climate protection effect, which is reflected in the effect on energy savings and greenhouse gas (GHG) reduction achieved by a measure as the main indicators. However, we will also briefly address other important criteria as e.g. economic effects (such as effects on energy costs, investments, employment or value added), distributional impacts, as well as acceptance and diffusion.

2. Assumptions on key framework data:

In order to consistently calculate the indicators for the energy savings impact of a measure, certain framework assumptions must be made. For bottom-up impact assessments, these include the following aspects in particular: Energy prices, lifetimes, emission and primary energy factors as well as the underlying reference development (baseline).

3. Methodological approach for determining gross impacts of the measure:

In principle, a distinction should be made between gross and net savings when reporting savings by means of indicators. The indicator values "influenced" by the measure are called the gross effect of a measure. Following the procedure of an ex-post evaluation, the impact model should first be determined also for an ex-ante evaluation. Such an impact model contains the idealised assumption of the impact of an action (here: implementation of a measure) in the sense of a causally justified sequence. However, a generally binding definition of how to determine the gross impact of a measure is not possible, as the individual policies are too heterogeneous and the method chosen depends on the specifics of each policy measure.

4. Effect adjustment approach for the calculation of net effects:

The net effect addresses the criterion of the effectiveness of a The question is to what extent the evaluated measure was "causal" for the determined gross impact and to what extent "additional" savings were achieved. To determine the net effect, it is necessary to adjust the gross values for corresponding effects as e.g. free rider, rebound or interaction effects.

5. Requirements for a transparent handling of uncertainties

Uncertainties exist at all described stages of the ex-ante evaluation process. In this context, it seems useful to distinguish between uncertainties regarding implementation and allocation of a policy measures as well as technological and political uncertainties. In our presentation, we will address some methods to reduce uncertainties which are used in practice. However, since inherent uncertainties cannot be completely avoided in ex-ante impact assessments, it is important to transparently present not only the assumptions on which the evaluation is based, but also the associated uncertainties.

In our presentation, we will not only discuss these criteria, but also illustrate them by a catchy policy example from the building sector.