Using Program Theories to Align Performance Metrics

Jeff Erickson, PA Government Services Inc.
Oscar Bloch, Wisconsin Department of Administration

Keywords

Abstract

Wisconsin is implementing statewide energy efficiency programs with public benefits funds. The programs are being run through the state government and are being implemented and evaluated by private firms. The programs must address a variety of legislative goals including energy efficiency, system reliability, environmental protection, and rural economic development. At the same time, the program implementers must establish a portfolio of programs that balance market transformation and resource acquisition policy objectives. With such a range of issues to address, the process of designing effective and responsive programs and evaluations has taken substantial effort. Under direction from the state, the contractors were instructed to take a theory-based approach to program and evaluation design. However, the programs were quickly designed and fielded based on incompletely articulated program theories. As a result, substantial effort has been expended attempting to develop viable program theories for second-year contracts. The state has announced that key metrics in the program theories will be incorporated into second-year contracts to ensure that the implementers’ motivations are in line with the broader goals. This paper will describe the methods used to facilitate discussions between the state, evaluators, and program staff to develop and refine program theories that are consistent with performance metrics and overall public policy. It will describe the key ingredients needed for effectively supporting the development of viable and useful program theories. It will also present some guiding principals for judging whether a program theory is adequate or not. Along the way, the paper will present the pitfalls encountered, approaches to solving those pitfalls, and lessons learned from the effort.