Integrated, Real Time (IRT), on-going data collection for evaluation – benefits and comparative results

Susan L. Gordon and Lisa A. Skumatz, Skumatz Economic Research Associates, Inc.

Keywords

real time, on-going, data collection, evaluaion, comparative results, building programme evaluation

Abstract

Assessment of the energy and demand savings attributable to building energy efficiency programs are usually conducted using surveys of past program participants, and oftentimes past non-participants. However, retrospective surveys – calls back to participants and non-participants from past years – have drawbacks including potential recall issues, low response rates, and high cost. We propose and test one workable and effective approach that helps address these problems – integrating evaluation data collection as part of already-established points of contact with participants.

On-going data collection provides several advantages over traditional methods. Data are available at any point during the program’s implementation – so evaluations can be conducted quickly. The data are collected at a point closer to when the program participation decision is made – potentially improving the quality of the data and consistency with actual decision-making. Finally, if data are collected as part of the ordinary course of the program, then it saves the expense of phone (or other) surveys, and can considerably increase the sample sizes available for the analysis. This approach has been used by several utilities, and the paper will describe the experience to date, benefits, costs, and lessons learned. The paper will compare results from retrospective attribution surveys and the real time, program-integrated surveys, and discuss how both types of surveys compliment each other to provide a more complete attribution evaluation.