Introduction

The assessment of energy use intensity (EUI) at the campus-level is inherently challenging because of the range of property types that may or may not be present, unique energy demands, differences in space utilization and occupancy rates, differences in average building age, local climatic/weather conditions and other factors. The way that EUI is assessed in STARS has evolved substantially over the years in response to these challenges and the availability of relevant data. The latest version – STARS 3.0 – features a simplified approach to EUI that leverages the statistical significance of institution type to benchmark performance within peer groups.

Recently, Cornell University and Colorado State University expressed concern about how performance is normalized under the new version and asked AASHE to reinstate some elements of the v2 methodology. We appreciate them raising these concerns and look forward to further engagement with the community on this subject. In the spirit of accountability and to help inform the discussion about these issues and how best to proceed, this post provides an overview of the data and analyses on which the changes to normalization factors were based, the impact on scoring that we anticipate and potential next steps for improving energy assessment in STARS.

Analysis of energy normalization in STARS v2

In the course of developing STARS 3.0, AASHE staff analyzed the energy data reported under STARS 2.2, as well as the findings of the ENERGY STAR Higher Education Benchmarking Initiative (HEBI). The results (summarized below) were surprising and indicated that the normalization factors used in v2 were not as effective as hoped in ensuring comparability between institutions. 

Energy-intensive space

Scoring in v2 was based in part on a complex metric: annual site energy consumption per gross square foot of EUI-adjusted floor area per degree day. To the best of our knowledge, EUI-adjusted floor area is a concept that is unique to STARS. It is calculated by increasing each institution’s actual floor area of building space to reflect the extent of property types such as laboratories that typically have high energy demands relative to office, classroom and residential space. 

Contrary to expectation, our analysis of the energy data reported under STARS 2.2 found that the self-reported extent of energy-intensive space on campus is not correlated with EUI. Only 2% of the variance in unadjusted EUI is predicted by the extent of energy-intensive space (R² = 0.024). This finding is consistent with the results of the HEBI:

“While unexpected, there were not clear performance trends with energy-intensive floor area, perhaps due to a lack of standardization in how institutions define such space and/or data quality issues in how it gets reported.”

Indeed, we have seen differences in how institutions report energy-intensive space.  For example, some institutions are reporting all buildings that include teaching laboratories as energy-intensive space and others are only reporting research facilities that have a measured EUI that is extraordinarily high relative to other campus buildings. In the absence of universal building-level energy metering (as tracked in ENERGY STAR Portfolio Manager, for example),  it is difficult to see how reporting could be sufficiently standardized across diverse institutions.

While the EUI-adjusted floor area concept was adopted in an effort to improve comparability between institutions, these findings indicate that it has not successfully done so. Indeed, our analysis suggests that normalizing by EUI-adjusted floor area is actually reducing comparability. 

Degree days

Degree days are a way to represent data on the outside air temperature of any particular location and its potential influence on the demand for heating and cooling. Again contrary to expectation, our analysis of the data reported under STARS v2 found that the total number of heating and cooling degree days is not strongly correlated with unadjusted EUI. Only 5% of the variance in EUI is predicted by the total number of degree days (R² = 0.05).

Because the concept of degree days is not well understood outside the field of energy management, this result may be due in part to data quality issues in how degree days are reported. It may also be due to other known challenges associated with degree-day-based normalization.  These challenges can be somewhat mitigated with regression analysis that accounts for building-specific factors such as property type, base temperature, baseload energy and occupancy. ENERGY STAR Portfolio Manager does this, for example. However this methodology requires that energy use be metered and reported at the building level rather than the campus level. Attempting to incorporate such an approach in STARS would be markedly more complicated than the current approach and would likely be inaccessible to institutions that don’t have comprehensive building-level energy metering in place.

Although the HEBI did not address degree days directly, it did explore the relationship between climate zone and EUI, with mixed results. The first round found EUI to be generally higher in colder regions, as expected, however the second round found that:

“Climate zone 6 (cold) showed the lowest median energy use, with the highest median energy use in zone 5 (cool). There were not clear trends across the other climate zones.”

Overall, as a result of these different factors, we do not have confidence that normalizing by degree days is meaningfully improving comparability between institutions. 

Institution type

Our analysis of the energy data reported under STARS 2.2 found a clear relationship between institution type and EUI, a finding that is also consistent with the HEBI results:

“EUI… increase[s] with Carnegie Classification as Doctoral Universities have significantly higher EUI… than all other types of institutions.”

There are likely multiple reasons for this relationship, but it seems fair to assume that the greater research intensity at Doctoral institutions is part of the explanation. 

This finding suggests that institution type may serve as a reasonable proxy for some of the key factors presumed to influence campus energy use, including research intensity. 

Changes for STARS 3.0

In response to the analysis outlined above, we initially proposed that quantitative assessment of campus-level EUI be dropped from STARS 3.0 altogether as impractical. The feedback received on that early draft was mixed, however, with some expressing the view that EUI was too important to leave out of STARS. The STARS Steering Committee (SC) therefore elected instead to modify the v2 methodology based on the staff’s analysis and the HEBI findings. This resulted in the following changes:

• Energy-intensive space and degree days were dropped as normalization factors because, as explained above, they seem to have been ineffective in (and perhaps detrimental to) ensuring greater comparability between institutions.
• Peer groups were established based on institution type, with a different benchmark calculated for each group. This change was made to leverage the significance of institution type and its potential as a proxy for factors such as research intensity. (Learn more about how these benchmarks were created.)
• A consistent approach to normalization was established across Operations whereby energy consumption, for example, is normalized by both floor area and the full-time equivalent of students and employees. This change was made to acknowledge that population can also be a significant driver of energy use and that the application of both normalization factors is likely to provide a more well-rounded assessment of performance than normalizing by floor area alone.

Anticipated impacts from these changes

This new approach was designed so that average aggregate energy scores would not change significantly from STARS 2.2. We anticipate, however, that a relatively small number of institutions (~6% of current STARS participants according to our estimates) are likely to experience a pronounced decrease in their EUI-based scores under the new methodology (and we also expect ~4% to experience a pronounced increase).  In some cases, these changes are likely a result of incorrect reporting of degree days or energy-intensive space that artificially inflated or decreased v2.2 scores. It does also seem to be the case though, as suggested by Cornell and CSU, that research-intensive institutions and institutions in colder climates are somewhat more likely to experience substantive decreases in these scores compared to other institutions. 

The extent of the changes made for v3.0 – including the substitution of an indicator on per capita energy use for an indicator on historical reductions – means that it is not possible to fully isolate the impacts of dropping degree days and energy-intensive space in favor of peer group benchmarking. Likewise, the results of reintroducing normalization for weather/climate and/or energy-intensive space in a future version would depend on the definitions used, how the thresholds and targets were set, whether the peer groups were retained and so on. It is reasonable to expect though that any substantial change would create new groups of “winners and losers” whose scores have changed because of the change in methodology rather than a change in performance. 

Next steps

Although the available evidence on campus-level energy use does not seem to support normalizing the data by energy-intensive space or degree days (at least not in the way we attempted in v2), there is ample evidence at the building level indicating that property/building type and climate/weather are important drivers of energy demand. We therefore understand the impulse to try to control for these factors in some way when comparing campus energy data. Likewise, we understand that a change in scoring (especially a negative change) that occurs without a significant change in the underlying performance is concerning for participants. This suggests that we should continue to explore alternative approaches to assessing and comparing campus energy efficiency. 

One tentative next step could be to convene a benchmarking task force to explore different options and recommend updates for STARS 3.1. Other possibilities include investigating the prospects of leveraging the use of ENERGY STAR Portfolio Manager (at least in the US and Canada) and/or adjusting the peer group benchmarks to account for climate zones.

We have shared the letters from Cornell and CSU with the STARS SC and will discuss how to proceed in the SC’s next meeting. We will keep participants informed through the online STARS Community and events like the STARS Town Hall at the AASHE Conference & Expo. As always, we welcome your suggestions and ideas! Please feel free to share them via the STARS Suggestion Box or by email to stars@aashe.org.