Sharing experience to spread evaluation practices: evaluation does help to improve policies!

Panel: 3. Policy and governance

This is a peer-reviewed paper.

Authors:
Jean-Sébastien Broc, IEECP (Institute for a European Energy and Climate Policy), The Netherlands
Barbara Breitschopf, Fraunhofer ISI, Germany
Barbara Schlomann, Fraunhofer ISI, Germany
Fabian Voswinkel, Fraunhofer ISI, Germany
Lovorko Maric, EIHP, Croatia
Gregor Thenius, Austrian Energy Agency, Austria

Abstract

Experience sharing about evaluation of energy efficiency policies is often limited due to lack of time for stakeholders to disseminate or document their evaluation works, and to the many languages in Europe. This paper presents the findings from two resources developed to get a better view of current ex-post evaluation practices and facilitate experience sharing:

– a Knowledge Base that gathers 180 references encoded according to a set of criteria enabling various types of search;

– 23 case studies including detailed analyses of practical examples of evaluations.

The review of the Knowledge Base supports the thesis that methodological discussions precede empirical studies: an evaluation issue (e.g. rebound effect) is more likely to be taken into account in practice if clear guidance is available about it. The review gives evidence of a large variety of studies and approaches, but highlights the need to elaborate and use transparent and replicable evaluation methods.

Promoting a minimum level of documentation would be a first step in this direction.

Documentation may be tricky due to the variety of metrics/approaches used for quantifying energy savings.

The review thus discusses what metrics and factors to take into account for common evaluation criteria (target achievement, effectiveness, efficiency).

Expanding the scope of evaluation to include other impacts than energy savings (e.g., employment, health) also appears a promising way to enhance the added value of evaluation and improve policy design.

The review of the case studies drew practical lessons summarized in 11 messages structured along the evaluation process: from the no-brainer about early planning of evaluation, up to thinking ahead what evaluation results to communicate, how and to whom, as well as about the conditions for evaluation findings to be used. These 11 messages are based on feedback from stakeholders and illustrated by examples that show how evaluations have helped to improve policies.