Abstract

Several high-profile replication failures have called into question the reproducibility of results in medicine, neuroscience, genetics, psychology and economics (Camerer et al., 2016). A paper published in Science found that just 1/3 of psychology studies could be replicated when the study was run for a second time (OSI, 2015). To our knowledge, there have been no attempted replications of energy efficiency studies; so can we be confident that the estimated energy savings from the smart meter roll out will be realised? Or that electric vehicles will reduce carbon emissions by predicted levels? Or is energy heading towards its own reproducibility crisis? Researchers call for the increased use of randomised control trials (RCTs) to evaluate energy efficiency policy and the introduction of protocols or guidelines for conducting experiments (Vine et al., 2014; Frederiks et al., 2016). However, no guidelines for increasing reproducibility have been proposed. Moreover, RCTs are just one method for causal analysis and RCTs cannot answer all important causal questions. This paper will outline research methods for improved impact assessment of energy efficiency policy, including RCTs, but also quasi-experiments and systematic reviews, that go beyond the conclusions of single experiments. It will then present tools for increasing replicability: pre-registration of trials; pre-analysis plans; reporting standards; synthesis tools and; publication of datasets with associated code in data repositories. Based on work at our research group at the UCL Energy Institute, a leading centre in energy research, we recognize that not all of these tools (mostly from medical trials) provide ‘off-the-shelf’ models for energy efficiency evaluations, and so consider adaptations for energy research. Our aim is to stimulate discussion and get feedback from the research community at ECEEE so the toolkit can be developed and potentially adopted more widely.

Downloads

Download this paper as pdf: 8-299-17_Huebner.pdf

Download this presentation as pdf: 8-299-17_Huebner_presentation.pdf

Panels of

1. Foundations of future energy policy

2. Policy: governance, design, implementation and evaluation challenges

3. Local action

4. Mobility, transport, and smart and sustainable cities

5. Buildings and construction technologies and systems

6. Buildings policies, directives and programmes

7. Appliances, products, lighting and ICT

8. Monitoring and evaluation: building confidence and enhancing practices

9. Consumption and behaviour