Addressing Uncertainty in the Evaluation of Market Transformation Activities

Kevin Cooney, Stratus Consulting
Dan Violette and Michael Ozog, Summit Blue Consulting

Keywords

Abstract

This paper describes the methodology and results of an evaluation of the market transformation efforts of the Northwest Energy Efficiency Alliance. The analysis presented here focuses on the methods used to bound the savings estimates attributed to the Alliance portfolio. The approach involved review of evaluation reports and market assessments conducted for about 30 programs over the past six years. From this review, a sample of programs was selected for analysis that was representative of Alliance activities across markets. For each program, the variables that influenced energy impacts were examined to determine which assumptions were pivotal drivers of savings claims. These pivot assumptions fall into one of two categories: field performance of the technology or number of units attributable to program activities.

To dimension uncertainty surrounding pivot assumptions, alternative hypotheses regarding each assumption’s value were sought from market actors in the region through an interview process. Combined with current literature on savings attributed to specific technologies, these alternative hypotheses helped establish a range for each variable (e.g., displaced watts for compact fluorescent lighting). A distribution was established for each pivot assumption, and a simulation analysis was conducted using 5,000 random draws from each distribution. Each draw was run through a cost-effectiveness model to produce a distribution for energy savings and a benefit/cost ratio attributable to each program. This process was completed for each program reviewed, and the results were summed to assess the overall energy impacts and levelized costs of the savings attributable to the Alliance portfolio.