Likert Scales are too simplistic – Better and more useful alternatives in four applications in energy efficiency

Panel: 4. Monitoring and evaluation for a wise, just and inclusive transition

This is a peer-reviewed paper.

Authors:
Dana D'Souza, Skumatz Economic Research Associates(SERA), USA
Lisa Skumatz, Skumatz Economic Research Associates (SERA), USA

Abstract

Likert Scales (5, 7, or 9-point) are in common use in energy efficiency (EE) evaluation, usually asking respondents to indicate a selection along a linear scale marked with labels on the extremes (very dissatisfied and very satisfied, or with labels at each numeric point. EE work commonly assigns equal differences in scores for each numeric increment, even though the originators warned against the practice. The evaluation, survey, and literature research conducted in this paper shows equally-spaced differences between the numeric or verbal extremes can be biased. This study provides an overview of Likert Scales, clarifies key analytical issues, and provides practical examples of more defensible approaches that should be considered in place of the simple analysis methods commonly applied with Likert scales. Specifically, we suggest adaptations of “labeled magnitude scaling (LMS)” or other Labeled Scales (LS) as more appropriate scoring values for the Likert increments.

This paper suggests improved practices in four example areas of evaluation where Likert Scales are commonly used:

• Process Evaluations: Likert scales are common in process evaluations (e.g., applied to “satisfaction”, and “ease of participation”). LMS or other labeled scale values provide more robust information and defensible estimates of average likelihoods and other factors.

• Net to Gross: The most sophisticated net-to-gross (NTG) methodologies use a combination of direct and corroborating questions to identify free ridership and spillover. Several questions involve Likert scales, and the current computations assume 25% / 50% / 75% / 100% “likelihoods” in the NTG computations. Transitioning to LMS scaling would provide likelihood percentages that are defensible and would reduce bias.

• Quantifying Better/Worse Program Changes: In matrices used to analyze or score program options, Likert or linear scales are often used for those elements that cannot be monetized. A LMS scale better represents differences than the “one unit” differences arising from Likert scales.

• Barriers Analysis: Assessment of program barriers in process evaluations commonly rely on Likert scales. Changes in barriers from, say, 3.4 to 3.6 do not provide implementable information. Using an extension of LMS (non-energy benefits or non-energy impacts NEB/NEI) provides explicit recommendations on the dollar size of the barrier and the incentive / intervention value needed to erase the barrier.

While none of these changes may be large, the LMS approach allows for greater defensibility, may support additional computations and meaning from surveys and applications, and will lead to more robust and defensible Evaluation, Measurement, and Verification (EM&V) results.