Techniques for getting the most from an evaluation: Review of methods and results for attributing progress, non-energy benefits, net to gross, and cost-benefit

Panel: Panel 4. Market transformation

Author:
Lisa A. Skumatz, Skumatz Economic Research Associates, Inc.

Abstract

As background for several evaluation and attribution projects, the authors conducted research on best practices in a few key areas of evaluation. We focused on techniques used in measuring market progress, enhanced techniques in attributing net energy impacts, and examining omitted program effects, particularly net non-energy benefits. The research involved a detailed literature review, interviews with program managers and evaluators across the US, and refinements of techniques used by the authors in conducting evaluation work. The object of the research was to uncover successful (and unsuccessful) approaches being used for key aspects of evaluation work. The research uncovered areas of tracking that are becoming more commonly used by agencies to assess progress in the market.

In addition, detailed research by the authors on a number of impact and attribution evaluations have also led to recommendations on key practices that we believe comprise elements of best practices for assessments of attributable program effects. Specifically, we have identified a number of useful steps to improve the attribution of impacts to program interventions. Information on techniques for both attribution / causality work for a number of programs are presented - including market transformation programs that rely on marketing, advertising, training, and mid-stream incentives and work primarily with a network of participating mid-market actors. The project methods and results are presented and include:

Theory-based evaluation, indicators, and hypothesis testing;
Enhanced measurement of free riders, spillover, and other effects, and attribution of impacts using distribution and ranges of measure and intervention impacts, rather than less reliable point estimates;
Attribution of program-induced non-energy benefits;
Net to gross, benefit cost analysis, and incorporation of scenario / risk analysis of results;
Comparison of net to gross results across program types to explore patterns and important differences.

These extra steps improved the reliability and robustness of the results of the causality analysis and provided a better foundation to guide benefit-cost analysis and program and investment decisions - an important goal of an evaluation. The paper highlights benefits and impacts of these approaches, and provides comparisons, contrasts, lessons learned, and highlights successful approaches that may be transferable to other locations.

Finally, the authors suggest that there are important impacts from programs that are often omitted from evaluations - non-energy benefits. Suggested approaches for assessing and measuring these hard-to-measure impacts are presented, along with order of magnitude results that have been estimated in previous research. The research suggests that cost-benefit analyses incorporating scenario analyses related to credible ranges for NTG estimates, and incorporating subsets of NEBs may provide the best information on program impacts. Combined with good quality information, tracking market progress provides a strong basis for evaluating program effects and understanding the progress the program has made in transforming the market.