This article was downloaded by: [The University of British Columbia]

On: 29 October 2014, At: 19:00

Publisher: Taylor & Francis
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,
37-41 Mortimer Street, London W1T 3JH, UK

Journal of Building Performance Simulation

Publication details, including instructions for authors and subscription information:
http://www.tandfonline.com/loi/tbps20

Trial application of ASHRAE 1051-RP: calibration

method for building energy simulation
a a
Michael J. Gestwick & James A. Love
a
Faculty of Environmental Design, University of Calgary, 2500 University Drive N.W.,
Calgary, Alberta, Canada T2N1N4
Published online: 03 Dec 2013.

To cite this article: Michael J. Gestwick & James A. Love (2014) Trial application of ASHRAE 1051-RP: calibration method for
building energy simulation, Journal of Building Performance Simulation, 7:5, 346-359, DOI: 10.1080/19401493.2013.838698

To link to this article: http://dx.doi.org/10.1080/19401493.2013.838698

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained
in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no
representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the
Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and
are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and
should be independently verified with primary sources of information. Taylor and Francis shall not be liable for
any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever
or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of
the Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematic
reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any
form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://
www.tandfonline.com/page/terms-and-conditions
 Journal of Building Performance Simulation, 2014
Vol. 7, No. 5, 346–359, http://dx.doi.org/10.1080/19401493.2013.838698

Trial application of ASHRAE 1051-RP: calibration method for building energy simulation
Michael J. Gestwick and James A. Love∗
Faculty of Environmental Design, University of Calgary, 2500 University Drive N.W., Calgary, Alberta, Canada T2N1N4
(Received 19 June 2013; accepted 24 August 2013 )

ASHRAE research project 1051-RP generated a method to improve the process of calibrating whole building energy sim-
ulation models based on monthly utility data. The approach, using manual generation of simulation model variations, was
applied to a 12,000 m2 high-performance, dual energy, cold climate building. This led to 27 models that met the ASHRAE
Guideline 14 monthly goodness-of-fit criteria for electricity, but had fit values for gas that were about 5–7 times the normalized
Downloaded by [The University of British Columbia] at 19:00 29 October 2014

mean bias error (NMBE) acceptance threshold. Five models met the criteria for natural gas and had acceptable coefficient of
variation of the root-mean-square error for electricity, but NMBE was about 100% too high. Use of finer interval monitored
data yielded a model with electrical NMBE about 60% above the acceptance threshold, and gas use about 200% above.
Hourly analysis of the thermal energy demand on the plant showed wide discrepancies with the estimates on an hourly and
half daily basis.
Keywords: calibrated simulation; measurement and verification; energy; monitoring

1. Introduction applying it to three case study office building designs: two

Calibrated whole building simulation is essential for assess- all-electric and one “synthetic” dual energy (gas and elec-
ing energy efficiency measures, allowing quantification of tricity). Despite the importance of this work, no accounts
system interactions. Calibration is the process of adjusting were found in the literature of subsequent attempts to apply
the simulation model parameters so that energy-use esti- the method to real buildings. In the project reported here,
mates match actual energy use with acceptable accuracy. the method was applied to a high-performance dual energy
Calibration of commercial and institutional building mod- building, the Child Development Centre (CDC) at the
els is challenging due to the number of systems involved University of Calgary (2013a). In a second stage, finer reso-
and uncertainty regarding factors such as dynamic perfor- lution (e.g. hourly) data were used to extend the calibration
mance of building systems and energy use (e.g. typically effort.
limited to monthly utility bills); it “relies on user knowledge,
past experience, statistical expertise, engineering judge- 1.1. 1051-RP method
ment, and an abundance of trial and error” (Reddy and Simulators have imperfect knowledge of a building dur-
Maor 2006). ASHRAE funded research project 1051- ing calibration, necessitating educated judgements. Reddy
RP to and Maor (2006) sought to minimize simulator variabil-
cull the best tools, techniques, approaches, procedures from ity by (a) identifying specific points where judgement is
the existing body of research and develop a coherent and required and (b) developing standard, repeatable processes
systematic calibration methodology and a well-documented
toolkit of robust procedures to help practicing energy for vetting simulator judgements and calibrating param-
engineers successfully reconcile computer simulations to eters. The model is deemed calibrated when the output
measured data from actual buildings. (Reddy and Maor matches the utility data with acceptable accuracy, such
2006, 221) as the goodness-of-fit (GOF) criteria stated in ASHRAE
The result was a method, dubbed “analytical calibration” Guideline 14 (ASHRAE 2002). Many parameters affect the
(Reddy, Maor, and Panjapornpon 2007b), envisaged pri- energy performance of a building, from static (e.g. wall
marily for use with monthly energy data, that identified transmittance) to dynamic (e.g. occupancy); “calibrating
influential parameters, discretized them, arranged them as a detailed energy simulation program involving numerous
unique input vectors, and simulated all unique input vec- input parameters is a highly underdetermined problem (i.e.,
tor combinations to deliver a set of calibrated solutions the presence of too many parameters is likely to result in
rather than a single calibrated model. Reddy, Maor, and any solution being non-unique)”. Reddy and Maor (2006)
Panjapornpon (2007b) demonstrated the method by advocated that a set of solutions be generated in lieu of a

∗ Corresponding author. Email: [email protected]

© 2013 International Building Performance Simulation Association (IBPSA)

 Journal of Building Performance Simulation 347

single “calibrated” model on the basis that exploring a range (5) use a set of the most plausible input vector solutions
of solutions would provide a more robust understanding of for the final calibration, Reddy and Maor arbitrarily
building performance: suggested using the top 20 (2006).
The conventional wisdom is that once a simulation model
is calibrated with actual utility bills, the effect of differ- Reddy, Maor, and Panjapornpon (2007a) wrote that while
ent intended ECMs can be predicted with some degree “the methodology is applicable to any building simula-
of confidence by making changes to one or more of the tion program, the scope [of their application examples]
model input parameters that characterize the ECM. Such . . . was restricted to the DOE-2 program”. They noted
thinking is clearly erroneous . . . While performing cali- that
bration, the many degrees of freedom may produce good
calibration overall even though the individual parameters it is necessary to be able to perform automated batch runs of
may be incorrectly identified. Subsequently, altering one or the [simulation] program. Several existing interfaces have
more of these incorrectly identified parameters to mimic been reviewed . . . but they all require the use of a text edi-
the intended ECM is very likely to yield biased predictions. tor to program the required input changes, and one cannot
(Reddy, Maor, and Panjapornpon 2007a, 230) import or export from external programs . . . [therefore] we
have decided to develop our own interface because of the
Downloaded by [The University of British Columbia] at 19:00 29 October 2014

The “analytical calibration” method takes an input vector

type of limitations in how to specify variations in the inputs
approach rather than focusing on individual input param- for multiple batch runs and also how to transfer simulation
eters. It involves four concepts: (1) sensitivity analysis output to a spreadsheet program for further analysis (Reddy
(identify strong influential parameters), (2) “identifiability and Maor 2006, 82).
analysis” (determine number and selection of parameters
for tuning), (3) numerical optimization (determine param- They also commented that “the scope . . . was most per-
eter values), and (4) uncertainty analysis (determine the tinent to medium and large commercial [and presumably
range of variation of these parameters) (Reddy and Maor institutional] buildings with relatively complex HVAC&R
2006). It is computationally intensive relative to more [heating, ventilating, air-conditioning and refrigeration]
heuristic calibration methods, using randomized sampling equipment”.
to evaluate the influence of input parameters. The steps are At the time of writing, there were roughly three times
(Reddy, Maor, and Panjapornpon 2007a) as follows: as many users working with simulation programs that have
graphical user interfaces (GUIs) that would be difficult to
(1) develop a realistic and error-free input file by use with a batch program (Table 1). Based on their experi-
verifying inputs; ence, the authors believe that few consulting practices use
(2) reduce “dimensionality of the parameter space” text-based simulation models.
using walk-through auditing and heuristics (i.e.
reduce the quantity of uncertain input parameters);
1.1.1. Influential parameters for calibration
heuristically define a set of influential parameters
that corresponds to simulation inputs; To alleviate the “curse of dimensionality”, Reddy and Maor
(3) perform a restricted coarse grid calibration with the (2006) stressed reducing the parameter set to the most sen-
parameters defined in (2) using Monte-Carlo sim- sitive variables. They used 24 parameters for the “complex”
ulation to identify “weak” parameters and further building (gas and electricity). The breakdown of the primary
reduce the dimensionality of the parameter space; parameters by type is shown in Table 2. The four addi-
(4) refine the calibration using the “strong” parameters tional parameters used in the “complex” case were “plant”
identified in (3); parameters.

Table 1. Selected commonly used whole building energy simulation programs.

Program User base as per USDOE (2013a) Comment

Programs more easily run in a scripted environment

DOE2 1000 user organizations, of 1–20 individuals
EnergyPlus 3,000 users (ConstructionPro Network, n.d.)
ESP-r Hundreds of users
Programs that are difficult to run in a scripted environment
EE4 CBIP 1000 users DOE2.1 simulation engine, probably 100 downloads;
authors believe user base closer to 200
EnergyPro 6000 licences DOE2.1 simulation engine
eQUEST 10,000 downloads annually DOE2.2 simulation engine
HAP 5000 users
TRACE 700 1200 worldwide, including single and site/LAN licences
 348 M.J. Gestwick and J.A. Love

Table 2. Influential parameters used in the 1051-RP case studies.

Parameter type No. of parameters Parameter description

Loads – schedules 7 Lighting, equipment, fans, space heating/cooling temperature, ventilation, auxiliary
electrical loads
Loads – envelope 3 Window shading coefficient, window and wall thermal transmittances
Loads – internal 2 Lighting and equipment power densities
Systems 7 Supply fan power, minimum outdoor air, minimum supply air, economizer, off/on
hours control, energy efficiency ratio
Plant 4 Cooling tower (fan power and control), cooling pumps and boiler efficiency
Auxiliary electric loads 1 Auxiliary electrical loads, no HVAC effect

Note: HVAC: heating, ventilating, air-conditioning.

Source: Adapted from Reddy, Maor, and Panjapornpon (2007b).

Reddy and Maor (2006) provided little information on 2. Methods

Downloaded by [The University of British Columbia] at 19:00 29 October 2014

the reasons for designating parameters as primary. Because 2.1. Case study building
sensitivity analyses were limited to the primary parameters,
The CDC is a 12,000 m2 four-storey building that con-
it is uncertain whether alternative parameters, such as
tains office and educational spaces (e.g. a children’s day
infiltration, might have offered better results than, for exam-
care on level 1), as well as additional space for an
ple, wall thermal transmittance. Reddy and Maor (2006)
enclosed one storey below grade parkade Calgary. Cal-
provided an extensive literature review that covered sen-
gary is at 51◦ north latitude and “very cold” (ASHRAE
sitivity analysis, and this was used as basis for selecting
2010) with mean annual temperature around 4◦ C.
influential parameters.
The CDC opened in October 2007, although the pho-
Schedules, which prescribe the variation of parame-
tovoltaic (PV) system only began operation in May 2008.
ters over time, are highly influential. Seven schedules were
A consultant was retained to commission the building sys-
included among 1051-RP influential parameters (Table 2).
tems to meet the Leadership in Energy and Environmental
Reddy and Maor (2006) noted that schedules have three
Design (LEED) best practice commissioning requirements.
diurnal degrees of freedom (DOF), which may be described
Extensive efforts were made to achieve lower than typ-
as intensity, duration, and occurrence (affected intervals).
ical heating and cooling loads and to meet the remaining
Seasonal changes introduce an additional DOF. The 1051-
loads in an energy-effective manner. The typical effective
RP case study sensitivity work used maxima and minima
wall and roof thermal transmittances were about 0.43 and
schedules that were large deviations from the starting point
0.17 W/m2 K, respectively. The fixed and operable window
and differed from the refinement process that might be
transmittances were about 1.9 and 2.3 W/m2 K, respec-
preferred if the starting point had been known with a
tively. The solar heat gain coefficient was 0.32. The building
fair degree of certainty, such as schedules developed from
has a low overall window-to-wall ratio of 0.21, intended
empirical data (Abushakra et al. 2001). 1051-RP minimum
to reduce thermal exchanges; the south-facing windows
value schedules for lighting, equipment, and fans matched
are shaded by a 43 kWp PV array, with the shading effect
schedules in ASHRAE 90.1-1989 (ASHRAE 1989).
included in the model. The south perimeter areas have over-
As Lam and Hui (1996) found for a subtropical cli-
head radiant cooling panels. Perimeter heating was provided
mate, “sensitivity tends to follow the end-use components
via below window fin tube radiation, selected to allow low
that consume the most energy” and “input design variables
return water temperatures to enhance the operating effi-
affecting these components will have significant influence
ciency of the two 360 kW condensing boilers. The main
on the annual building energy consumption”. The influence
level of the building houses educational spaces that are
of boiler efficiency in Mottillo (2001) reflected the cold
served by a displacement ventilation system (AHU-1) with
climate.
enthalpy wheel ventilation heat recovery, due to the rela-
It is to be expected that a review of sensitivity analyses
tively high fraction of outdoor air. Levels 2 to 4, which
yields no universal set of influential parameters. General
mainly house offices and related spaces, have a ducted
trends emerged, such as the strong influence of parame-
underfloor air distribution system (AHU-2). Both systems
ters directly affecting the large end uses, as well as regional
were recirculating, with air- and water-side economizers
trends, such as the high impact of internal loads and cool-
and variable speed drives to support variable flow. The
ing parameters in subtropical regions. The importance of
chiller capacity was near 400 kW, about one-third of that
experience and judgement in the selection of parameters
found in typical contemporary buildings. The average con-
was evident in the literature; given limited resources, one
nected lighting power density in the areas fitted out at the
cannot test all parameters and must identify those likely to
time of the calibration study was about 7.5 W/m2 , with
be the most influential.
occupancy sensors in many spaces.
 Journal of Building Performance Simulation 349

Table 3. CDC energy-use monitoring equipment.

Equipment Details Start-up date

Building utility meters Total natural gas, total electricity use (monthly) Elec: December 2007; gas: October 2007
Energy management meter Total electric power at 15 min intervals 15 January 2008
PV transducer logger PV power (hourly) June 2008
Elec. Monitoring Plug, lighting by floor; panels about 90–100% 5 January 2008
“pure”; the large 3000 m2 floor plates made
perfect separation of loads prohibitively expensive;
elevators, 15 min intervals
Building management system Electricity (fan, pump, chiller), gas submeters (each Varied
boiler, service water heater, parkade make-up air
unit), flow meter for hydronic heating loop, node
temperature, flow rate, etc.
Microloggers On–off for lighting Varied
Downloaded by [The University of British Columbia] at 19:00 29 October 2014

The CDC has unusually extensive built-in monitoring Inspection of the wall assembly information suggested
systems (Table 3), selected to meet the then existing LEED that a framing percentage of 0.08% for the Z-bar wall sys-
measurement and verification credit requirements. tems was more realistic than the nominal 0.37% (NRCan
2007); the wall parameters were adjusted accordingly.

2.2. Energy simulation model 2.3. Weather data

The energy simulation model that was calibrated was cre- The University of Calgary (2013c) has a weather station
ated in EE4-DOE2.1E. DOE2 is a widely used whole about 500 m from the CDC. The study year was 2008, dur-
building simulation program (USDOE 2013b). EE4 is a ing which there were some gaps in data for June and July,
GUI developed by Natural Resources Canada (NRCan 99% and 90% complete, respectively. For gaps of 2 h, miss-
2007). EE4-DOE2.1E was selected based on the avail- ing values were determined by linear interpolation between
ability of a federal government peer-review program that values on either side of the gap. There were two July days
restricted reviews to that software. The model that was with longer gaps, and the missing values were determined
calibrated was based on the peer-reviewed model submit- based on linear interpolation between values for the same
ted for LEED certification, which was developed based on hours on preceding and succeeding days.
“issued for construction” drawings and contractor submit-
tals. In another study, EE4-DOE2.1E simulation estimates
2.4. Simulation procedure
were found to closely match those obtained with an Ener-
gyPlus model once similar assumptions (e.g. pump control 2.4.1. Influential parameters
schemes) were used (Tian et al. 2009). Calibration of a sim- A list of parameters, shown in Table 4, was developed and
ulation model created with a GUI was considered a common each parameter classified as (1) D – directly measured with
scenario for reasons given in Section 1.1. little uncertainty; (2) I – indirectly measured or uncertain;
The model was modified (from the version submitted and (3) U – unmeasured. Sensitivity to these parameters
for the LEED review) to reflect conditions in the areas that was tested using (1) input–output analysis (or differential
had yet to be fitted out at the time of the calibration study. sensitivity analysis) and (2) sensitivity coefficient “2a”, as
Loads in those areas were adjusted to actual connected loads suggested by Lam, Wan, and Yang (2008). The parame-
rather than occupied levels as required for the LEED rat- ters were ranked according to relative influence to identify
ing process. Since unoccupied areas had no ventilation, the significant input parameters for which measured data
“fictitious” air handling systems were created (capacities were available (SIPs) and those that were unmeasured or
prorated based on floor area) to disaggregate supply and unknown (USIPs); a parameter classified “I” qualified as a
return fan power among occupied and unoccupied areas. For USIP if there was large uncertainty in its valuation.
unoccupied zones, the schedules for lighting, occupancy, As noted in Section 1.1.1, Reddy and Maor (2006)
service hot water (SHW), and equipment were set to zero. used up to 24 parameters deemed to be most influential
For occupied zones, the fan schedule was changed to 0600 (Table 2). This study sought to be as thorough as possible
to 1800 to match the actual schedule rather than the nom- with respect to including parameters in the analysis. The
inal office schedule for performance rating; other system list in Table 4 was broad and included all the parameters
schedules (e.g. lighting) were adjusted commensurate with identified as potentially having substantial influence and
this. Interior drawings, including mechanical and electri- adjustable through the EE4 interface with reasonable ease.
cal, were available for the fitted out areas. Features, such as Reddy and Maor (2006) varied the maximum and
lighting, were checked by site inspection. minimum values relative to the base value (refer to
 350 M.J. Gestwick and J.A. Love

Table 4. Parameters included in CDC sensitivity analysis (by DOE2 module).

Module Parameter Classification Type

LOADS Occupant density, schedule I Discrete

Solar heat gain coefficient, wall, roof and window U -values, infiltration, wall Continuous
absorptivity
Service water heating schedule U Discrete
DOE2 thermal mass factor (“floor weight” index), infiltration schedules Continuous
SYSTEMS Schedules – fans D Discrete
Supply and return fan motor, power fan motor, lighting power, outdoor air flow, Continuous
temperature set points
Schedules – fans, lighting, equipment I Discrete
Schedule – service water heating U Continuous
AHU-1 energy recovery efficiency, parkade make-up air combustion efficiency
PLANT Heating circulation pump design temperature drop D Continuous
Boiler efficiency, chiller coefficient of performance I Continuous
Downloaded by [The University of British Columbia] at 19:00 29 October 2014

Chiller power, pump efficiencies, tankless service water heater efficiency U Continuous

Section 1.1.1), with a larger range for the “complex” net. EE4 lacks an input mechanism for exterior lighting.
building than the “simple” one for the “synthetic” case stud- Estimates projected that PV would produce about 10% of
ies. For the actual building, the ranges tested were highly annual electricity use (excluding process and plug loads).
variable and at times an order of magnitude different from Exterior lighting cannot be modelled in EE4.
the base value. Based on this, ±30% was used because (a)
the CDC is a complex building and (b) it was believed that
the inputs were known within an order of magnitude due to a 2.4.2.1. Calibration with monthly energy-use data. The
high degree of familiarity with the project. Exceptions were top five SIPs were identified by the sensitivity analysis.
infiltration and outdoor air flow rate. The former is fixed in The lead author of Reddy and Maor (2006) recommended
EE4, while adjustment of the latter is time-consuming for that “less than ten” be used. The number of parameter
a large building (see the first paragraph of Section 2.4.2.1). vectors (M ) for (N ) USIPs is given by M = 3N , result-
Schedule perturbations were the same as in Reddy and ing in 243 and 720 unique parameter vectors with 5 and
Maor (2006). They omitted occupancy and SHW from the 6 USIPs, respectively. Assuming 0.5 h to set up, simulate,
influential parameters, so no alternative schedules were and process the data from each vector, moving from 5 to 6
available for these; default schedules were perturbed ±10% USIPs would have increased the processing time from about
in terms of intensity. The EE4 built-in options were used for 120 to 360 person-hours, or from about 1 person-month to
other discrete parameters (e.g. floor weight). The sensitiv- about 3 person-months. Given the resources available for
ity of these parameters was evaluated to identify those most the project, five USIPs were considered the feasible limit.
influential. The monthly calibration used the base model and the top
five influential input parameters (SIPs) determined from the
sensitivity analysis: boiler efficiency, infiltration, lighting
2.4.2. Calibration approach and equipment schedules, and fixed window U -value. Fan
As noted in Section 1, the batch program used to run the schedules, while identified as highly sensitive, were omitted
sensitivity simulations in Reddy and Maor (2006) would because their operational hours were known with a high
have been difficult to apply. Development of the full SIP degree of certainty, a result of the initial walk-through audit.
selection method defined in 1051-RP was beyond the time Outdoor air fraction was considered in lieu of fixed
and resources available for this study, so an approach was window U -value, because their sensitivities were nearly
developed for this study incorporating some of the pro- identical, but was omitted since the parameter would have
cedures used in 1051-RP. The most influential SIPs were had to be changed zone by zone in the 40 zone EE4 model.
defined heuristically using the findings of past studies and The heat recovery system effectiveness was omitted from
the author’s judgement. the parameters deemed “influential” that were tested in
For research purposes, two stages of model calibrations the sensitivity analysis. Sensitivity to this parameter could
were conducted: (1) one using monthly data and (2) one be assessed to some extent from the sensitivity to out-
using shorter interval subsystem data. The monthly cali- door air rates (see discussion in the second paragraph of
bration was conducted prior to the detailed calibration to Section 2.4.2.1) (comparable to sensitivity to window trans-
avoid modeller bias from analysing the end uses. It was lim- mittance). One of the limitations of the DOE2 variable
ited to monthly data, except for (1) PV production and (2) air volume module is that the preheat coil is hard-coded
exterior lighting. An exception was made for PV end-use as positioned behind (downstream of) the energy recovery
data, because the monthly building energy-use data were system, which is the opposite configuration typical in cold
 Journal of Building Performance Simulation 351

climates, due to the need to raise the temperature of outdoor less than that of the measurements, because, as the
air to control frosting in heat recovery systems when out- number of measurements increases, the CV(RMSE)
door air temperatures are below 0◦ C. Another limitation is decreases. Reddy and Maor (2006) suggested using
that while it has been known since the early 1980s (Besant the Akaike information criterion (1985).
and Bugg, 1983) that heat recovery performance degrades in (3) Direct comparison of CV(RMSE) values based on
cold weather, DOE2 represents heat recovery effectiveness different time scales (i.e. different values for n)
as a single fixed number. These problems impede sensitivity is statistically incorrect. A proper statistical solu-
analysis for ventilation heat recovery. tion to deal with this specific case remains to be
The five SIPs were discretized using the methods identified.
described below. Unique vector parameters and correspond- (4) Uncertainty in measured data is not addressed.
ing versions of the simulation were developed, each with a ASHRAE Guideline 14 (ASHRAE 2002) assigns
unique identifying code. The GOF was determined using (1) zero error to utility data. Reddy and Maor (2006)
normalized mean bias error (NMBE) and (2) the coefficient suggested this is never really the case and that
of variation of the root-mean-square error (CV(RMSE)) the combination of reading the meter at different
(ASHRAE 2002): times on different days of the month (when using
Downloaded by [The University of British Columbia] at 19:00 29 October 2014

n monthly data) and instrument accuracy could pro-

(yi − ŷi ) duce an error of 3% or greater. Errors using field
NMBE = × 100, (1)
(n − p) × ȳ instrumentation would be greater.
 1/2
(yi − ŷi )2 )/(n − p)
CV(RMSE) = × 100, (2) Continuous SIPs were discretized within a prescribed
ȳ
range using a triangular distribution as suggested in Reddy
where yi is the measured utility datum, ŷ is the simula- and Maor (2006). A triangular distribution should only be
tion predicted datum, ȳ is the mean value of observations, used when there is confidence in the “best guess” value for
n is the number of data points, and p is the number of the parameter being discretized. To discretize, variable x
parameters in the model. For calibrated simulation, it was was bounded where the best guess, xbase , was within the
recommended that p = 0 for Equation (1) and p = 1 for range [xmin , xmax ]. The discretization seeks to find xlow ,
Equation (2) (Reddy and Maor 2006). Giving no explicit xmid , and xhigh such that each variable has an equal proba-
reason, ASHRAE Guideline 14 (ASHRAE 2002) states that bility of occurrence. The discretized variables were defined
“The computer model shall have an [maximum] NMBE of as (Reddy and Maor 2006)
5% and a CV(RMSE) of 15% relative to monthly calibra- 
tion data.” The models with the 10 “best” GOF within the (xbase − xmin )(xmax − xmin )
x low
=x min
+ , (3)
accepted range would define the solution set for the monthly 6
calibration. The monthly calibration was representative of 
(xbase − xmin )(xmax − xmin )
the “coarse grid search” in Reddy and Maor (2006) and x mid
=x min
+
2
excluded the two-stage calibration. Reddy and Maor found
that there “does not seem to be much incentive in performing xmax + xmin
if xbase ≥ , (4)
a two-stage calibration” offering the following comments on 2

these guidelines: (xmax − xbase )(xmax − xmin )
xmid =x −
min
2
(1) These limits provide initial guidance, but there is
x max
+x min
no “broad consensus” regarding the definition of the if xbase < , (5)
accuracy limits for calibration. Others think this tol- 2

erance cannot be defined and is subject to the tools (xmax − xbase )(xmax − xmin )
being used (Kaplan et al. 1990 cited in Reddy and xhigh = xmax − . (6)
6
Maor 2006). A calibrated simulation is the net result
of tuning many different input parameters, yet there This straightforward discretization is only applicable
is no guarantee that any individual parameter is when the SIP is continuous; different approaches must be
properly calibrated. When using a model to evaluate used for discrete SIPs, particularly schedules. To discretize
an EEM (energy efficiency measure), it is suggested schedule intensity, the same approach can be used as above
that the influential parameters be known and prop- for continuous parameters, assuming that a base value and
erly calibrated even if it means less influential ones range are known with some certainty. Duration and tran-
remain improperly calibrated. sition, which are temporal changes, require identification
(2) The use of CV(RMSE) is questioned on the grounds of a range in time. Reddy and Maor (2006) used 2 h varia-
that uncertainty in the measurements means the tions for these changes. The combination of changes to all
CV(RMSE) of the model would likely never be three DOF means that a fully discretized schedule parameter
 352 M.J. Gestwick and J.A. Love

results in 27 schedule variations. This is beyond a reason- ASHRAE recommends a “proper uncertainty analysis
able manual input effort, so an alternative approach was evaluating the instrumentation under consideration and data
adopted. Specifically, for lighting and equipment, schedules expected to be gathered should be performed prior to testing
for medium and large buildings were taken from 1093-RP, to refine the measurement system to the level of uncertainty
a research project that used empirical data to develop load desired” (2002). Most of the instrumentation used in this
profiles for office buildings (Abushakra et al. 2001). These study was “built in” as part of construction; an uncertainty
schedules provided adequate variability from the base case analysis was not provided. Therefore, the initial calibration
and, when compared with those used in IPMVP (2006), they and maintenance of the instrumentation had to be taken
provided profiles closer to those of the study building. For in good faith, a practice with precedent in dealing with
all other schedules, this study used those provided in Reddy monitored data from buildings (Abushakra et al. 2001).
and Maor (2006). Nevertheless, this practice introduces undesirable uncer-
tainty in the results. Additionally, the precision of some
instruments installed in the field was unknown and poor
2.4.2.2. Detailed calibration. The goal of the detailed installation (e.g. airflow sensor installed near pipe elbow)
calibration was to use field data to reduce the number of further degraded information quality and there was “no way
Downloaded by [The University of British Columbia] at 19:00 29 October 2014

unknown inputs in the simulation and thus alleviate the to quantify these other effects” (IPMVP 2006). For these
“curse of dimensionality” (Reddy and Maor 2006). Data reasons, quantitative uncertainty analysis was omitted.
relating to parameters with “D” or “I” classifications in The data for individual parameters were reviewed for
Table 4 were analysed. Time constraints precluded con- points that were missing or faulty (e.g. efficiencies greater
sideration of all available data; parameters determined to than 100%). Missing data points were dealt with in one
be most influential were the focus. of two ways. If a large gap existed the data points were
The IPMVP (2006) notes that errors occur in three areas: removed from the sample. If only a few data points were
modelling, sampling, and measurement. All three were rel- missing amid other data showing steady-state behaviour,
evant to varying degrees. Modelling error is a function of the average of the neighbouring data was used as an accept-
the simulation program’s numerical methods and capacity. able replacement (ASHRAE 2002). Faulty data points were
For the initial analysis, based on monthly data, envisioned removed from the sample.
as the most common application case by Reddy, Maor, and
Panjapornpon (2007a, 2007b), the building gas and electric-
ity use was measured with utility grade meters, for which
the error is about 0.5%. The PV logging system had similar 3. Results
accuracy. The exterior lighting electricity use was calcu- 3.1. Calibration with monthly energy-use data
lated based on monthly dusk to dawn hours and installed The initial simulation energy end-use estimates are shown in
exterior lighting power. Since exterior lighting electricity Figure 1. The monthly model was calibrated using monthly
use was only about 1.3% of total annual electricity use, even utility data for gas and electrical energy use; utility demand
a 10% error would amount to only 0.1% of total electricity data were not used, because the PV logging system reported
use. only hourly kWh use. The utility, PV, and estimated exte-
Sampling and measurement errors resulted from the rior illumination energy data are provided in Table 5. Two
CDC monitoring systems (Table 3). For example, the BMS adjustments were made to the utility data so it would more
logged 15 min “point-in-time” values rather than the aver- realistically match the model output: (1) electricity supplied
age value over the interval. Further, some logs (i.e. power by the PV array was added to the utility meter values and (2)
demand with the electrical systems logger) combined end the simulation excluded exterior illumination; the monthly
uses. These records were predominantly used to develop electricity demand for exterior illumination was calculated
custom lighting and equipment schedules for various areas
of the building. A lack of supplemental data precluded sep-
arating the loads. In cases where the additional load was
relatively small (e.g. an exhaust fan included in an entire
floor’s installed lighting capacity), the effects of the addi-
tional load was assumed negligible. In cases where loads of
comparable size were mixed, or where an additional load
was thought to have a large influence on the demand profile,
the parameter being evaluated was reclassified “U” and con-
sidered for inclusion in the input vector component of the
calibration. If selected, the median, 25th percentile, and 75th
percentile defined the base, minimum, and maximum dis-
cretized inputs, respectively. These were evaluated relative
to the installed capacity of the parameter under review. Figure 1. CDC base model annual energy end-use estimate.
 Journal of Building Performance Simulation 353

Table 5. CDC 2008 adjusted monthly utility data (PV production included, exterior lighting excluded).

Electricity Exterior Adjusted

Month (meter) (kWh) illumination (kWh) PV (kWh) electricitya (kWh) PV% of total Adjusted gasb (GJ)

January 35,307 603 0 34,704 0 737

Februarya 29,813 476 0 28,325 0 608
March 29,574 452 0 29,122 0 410
April 28,873 365 0 28,508 0 420
May 29,835 301 896 30,430 3 174
June 29,054 292 4658 33,420 14 85
July 30,579 301 5431 35,709 15 36
August 28,634 301 5465 33,798 16 41
September 27,025 365 5318 31,978 17 97
October 31,271 452 5311 36,130 15 180
November 32,692 510 3265 35,447 9 315
December 33,010 603 2000 34,407 6 739
Downloaded by [The University of British Columbia] at 19:00 29 October 2014

Annual total 365,666 5020 32,345 391,979 8 3840

a PV production included (added), exterior lighting excluded (extracted); total of columns 1–3 multiplied by 28/29, because 2008 was
a leap year and a 365-day year was simulated.
b Gas use with leap-year adjustment – see Footnote a.

Table 6. Monthly GOF indices for base model.

NMBE CV(RMSE)
CDC base ASHRAE CDC base ASHRAE
model criterion model criterion

Electricity 24% 5% 26% 15%

Gas −15% 5% 24% 15%

matching trends, but poor GOF (Table 6). The simulation

underestimate of gas use seems to have been persistent
from June to December (this issue will be revisited in
Figure 2. Comparison of monthly CDC electricity use: base Section 3.2.2).
model versus “utility-adj”.

3.1.1. Sensitivity analysis

3.1.1.1. Load parameters: schedules. Sensitivity coeffi-
cients for schedules cannot take the form 2a (Lam, Wan, and
Yang 2008), because they are not entered as a scalar value.
Fan schedules were identified as a highly sensitive param-
eter for both electricity and gas use. Simulation estimates
were also very sensitive to both equipment and lighting
schedules, particularly with respect to electricity use. As
expected, these parameters had net energy effects that par-
tially offset one another: increased hours of lighting use
increased electricity use (and zone heat generation) which
simultaneously decreased gas use.
Figure 3. Comparison of monthly CDC gas-use data: base model
and utility. 3.1.1.2. Load parameters: other than schedules. With
respect to load parameters other than schedules, the scale of
the SCs for gas was roughly two orders of magnitude greater
using estimated hours of operation and these values were than those for electricity. Infiltration and fixed window
subtracted from the total electrical load. thermal transmittances were the most influential envelope
Figures 2 and 3 show the adjusted (as described in the parameters. A narrower range (±10%) for fixed window
preceding paragraph) electrical and gas use plots with the thermal transmittance was tested to determine whether the
simulation estimates. The monthly values show generally 30% range was appropriate for this parameter. The resulting
 354 M.J. Gestwick and J.A. Love

change in average SC for gas was 3% and negligible for buildings as a basis for annual energy calculations of build-
electricity, confirming the appropriateness of the range ings (Abushakra et al. 2001). These offered a more realistic
and exhibiting the diminishing returns from increasing the load profile without dramatically changing the overall load
thermal performance of this envelope component for this relative to the 1051-RP schedules (Reddy and Maor 2006).
design. The range for infiltration was reasonable given the They are identified in Table 7 by the prefix “RP1093”.
results of past research on infiltration in Canadian office The GOF indices were calculated based on monthly
buildings (Shaw, Reardon, and Cheung 1993). electricity and gas use for each unique input parameter vec-
tor. Of all 243 variations (each assigned a distinct code in
3.1.1.3. System parameters. Make-up air unit (parking the file name), none simultaneously met ASHRAE calibra-
garage) efficiency was highly influential for gas. Other influ- tion criteria for electricity and gas. Certain combinations
ential system parameters included the outdoor air rate (for did result in either gas or electricity meeting GOF criteria.
gas use), and lighting and equipment power densities, for Some of those that most closely met the GOF criteria for
electrical use. gas were outside the NMBE range for electricity by only
about 10% (i.e. 16% versus 15%).
Downloaded by [The University of British Columbia] at 19:00 29 October 2014

3.1.1.4. Plant parameters. Parameters related to pumps

and cooling had no effect on gas use. Conversely, those 3.2. Detailed calibration: high-resolution energy-use
directly related to heating had no influence on electricity data
use. As might be expected, with about 75% of annual energy For the detailed calibration, the base-case model was
use for heating (Figure 1), boiler efficiency was highly updated based on measured energy end-use data. Time and
influential, while cooling equipment performance had little resource limits precluded use of all of the vast amount of
effect. data from the monitoring systems. To focus data use, param-
eters in Table 4 with (1) a “D” or “I” classification and (2)
relatively high sensitivities were selected.
3.1.2. Monthly calibration results This analysis involved additional scrutiny of the base-
The monthly calibration of the base model used the five case model which revealed a number of items that required
SIPs determined from the sensitivity analysis: boiler effi- correction, including:
ciency, infiltration, lighting and equipment schedules, and
fixed window U -value. Fan schedules, while identified as • the addition of lighting power to L2, raising it by 9%
highly sensitive, were omitted, because operational hours (2% increase for the building);
were known with a high degree of certainty from the initial • zeroing of weekend/holiday schedules of the “zero”
walk-through audit. Although outdoor air and fixed win- schedule used in unoccupied areas (the week-
dow U -value had nearly identical effect in the sensitivity end/holiday schedules had been inadvertently set at
analysis, window U -value was chosen due to the relative 5% for all hours);
difficulty of changing the outdoor air parameter in the EE4 • a halving of the parkade system capacity since one of
interface. Make-up air combustion efficiency, while identi- two make-up air units did not run during 2008;
fied as a sensitive parameter, was excluded, because one of • setting the minimum outdoor air requirement in unoc-
the units was not operational during the study year due to cupied zones to 0.1 L/s/occ, the lowest possible in
equipment problems. the EE4 interface, to minimize heating of outdoor air
Table 7 summarizes the input values of parameters for these spaces (the reason this influenced the out-
used for the general calibration. Continuous parameters put when the occupancy schedule was set to zero is
(i.e. infiltration, fixed window thermal transmittance, and unclear);
boiler efficiency) were discretized using Equations (3)– • regeneration of the 2008 weather file, because minor
(6). The sensitivity analysis made use of the schedules discrepancies were found. This had a small effect on
from 1051-RP. The schedules used for the calibration were the original base model annual energy use (0% and
developed from observed data in medium and large office 3% changes in electricity and gas use, respectively).

Table 7. Parameters and values used for monthly calibration.

Parameter ID Base Max. Min. D− low (1) D− mid (2) D− high (3)

Infiltration (L/s/m2 ) a 0.25 0.33 0.18 0.22 0.25 0.28

Lighting schedule b MNECB – – MNECB RP1093− Medium RP1093− Large
Equipment schedule c MNECB – – MNECB RP1093− Medium RP1093− Large
Fixed window U -value (W/m2 C) d 1.94 2.13 1.75 1.86 1.94 2.02
Boiler efficiency (%) e 96 [75] 50 64 74 83

Note: The “D” values are the discretized inputs used in the calibration process.
 Journal of Building Performance Simulation 355

3.2.1. Detailed calibration – stage 1: energy end-use against which Raftery, Keane, and O’Donnell (2011) cau-
analysis tioned. It is also exemplifies the modelling uncertainty that
For the detailed calibration, the base-case model was Reddy, Maor, and Panjapornpon (2007a) sought to address.
updated based on measured energy end-use data. The boiler efficiency increases with cold weather (the DOE2
default condensing boiler part load curves model efficiency
3.2.1.1. Boiler efficiency. Figure 1 illustrates the influ- as declining with increasing load), while the heat recovery
ence of heating (and, hence, heating plant efficiency) on effectiveness (a fixed number in DOE2) actually decreases,
overall energy use. The CDC has two high-efficiency con- and vice versa. If one thought the boiler model were correct,
densing boilers. The overall heat production efficiency one might further derate the heat recovery effectiveness to
(thermal energy supplied versus fuel value of the natural achieve satisfactory GOF, but this would be wrong in terms
gas burned) was much less than the rated efficiency used of equipment performance.
in the base simulation model. Further, the efficiency was
found to be proportional to load, contrary to manufacturer’s 3.2.1.4. Lighting power. Analysis of power panel data
curves based on steady-state lab measurements, which are revealed a 1 h time shift in the first quarter of 2008. While
data before and after the spring daylight savings time (DST)
Downloaded by [The University of British Columbia] at 19:00 29 October 2014

the basis for the boiler performance curves provided in

DOE2. As wrote, ASHRAE’s Standard 155P for boiler test- date of 9th March showed a 1 h offset, it was opposite
ing had been under development for 17 years when they to the expected offset. It appeared the logger clock had
published, and been incorrectly reset, so the timing of data from the elec-
trical monitoring system was shifted to correct for this.
is sorely needed by the HVAC industry because there is no There was no straightforward or reliable way to separate
standard for rating the performance of commercial boilers
at part load conditions (where most boilers operate most of VU-1 power demand (1.5 kW) from level 1 (L1) lighting
the time) or at realistic entering water temperatures. BTS (18 kW), since they were on the same panel and the VU-1
– 2000 is the current standard used by the industry but it (main vestibule air handler) schedule was unknown. The
only rates boilers at full load and only at unrealistically low L1 lighting schedule was revised to the measured pattern.
entering water temperatures (Beliso, Huestis, and D’Alboro The parkade panel included lighting and other loads (e.g.
2012, 10).
exhaust fans and pumps). Short-term field measurements
In the detailed calibration, the load seen by the boilers was with lighting-sensing microloggers revealed that parkade
compared, rather than the fuel use. lights (with occupancy controls) remained on through-
out the night. The lighting schedule was therefore set
3.2.1.2. Service water heating. Because the BMS was to continuous operation. There was no way to discern
not programmed to log SHW gas use, manual gas meter what the demand and load shape was for parkade lighting
readings were made during site visits for approximately one alone. Additional field measurements with lighting-sensing
year. Comparison of the daily average energy use and sim- microloggers revealed that the parkade lights remained on
ulation results showed that the latter were 65% higher. This throughout the night. The lighting schedule for this zone
meant that SWH gas use was a negligible end use relative was set to 100% for all hours.
to space heating, considering the end uses in Figure 1. The data from logs for emergency lighting were suspect,
particularly one that showed a large mid-day 9 kW load.
3.2.1.3. AHU-1 energy recovery wheel effectiveness. This indicated load(s) other than emergency lighting. The
Evaluation of ventilation heat recovery performance was lighting schedule was adjusted to match this load.
problematic. Key sensors (dry bulb temperature and rela-
tive humidity downstream of the heat wheel) were missed in 3.2.1.5. Miscellaneous. The weekday profile for plug
the original construction and were only added to the system loads showed a large spike around 0800 to 1000. A reason-
in the spring of 2009, about 1.5 years after the beginning able explanation is that it reflects food service equipment
of data collection for calibration. Additionally, a compari- (the daycare has a kitchen and the main floor common area
son of air flow rates showed the outdoor air supply reading has a coffee shop). The plug loads were left as in the base
consistently much higher than supply air design flow. The model.
controls contractor confirmed that the outdoor air sensor for The fan schedules and power were adjusted to reflect
this air handler provided faulty readings due to inadequate measured values.
straight duct length upstream of the sensor, which was pre-
cluded by the duct configuration. Because of the lack of
valid data and associated uncertainty, evaluation of the heat 3.2.2. Simulation with detailed calibration adjustments
wheel effectiveness was omitted. The modifications in Section 3.2.1 resulted in a new “base”
The unrepresentative DOE2 ventilation heat recovery model, for which the electrical GOF was summarized in
model (discussed in Section 2.4.2.1) and the lack of mea- Figure 4 and Table 8, which included many parameter
sured performance data combined with the unrepresentative values from the general calibration model. The simulated
boiler model could lead to the type of erroneous “tuning” electrical energy use began to consistently exceed the
 356 M.J. Gestwick and J.A. Love

Figure 4. Comparison of monthly electrical data between

updated base model and utility-adj after detailed calibration Figure 5. CDC measured and simulated heating load for 16
adjustment. January 2009.
Downloaded by [The University of British Columbia] at 19:00 29 October 2014

Table 8. Monthly GOF indices for electricity, updated base the non-ventilation (mainly envelope and internal) gains
model. were plotted. These are closer to expectation, with the load
low near midday due to internal gains and higher at night,
NMBE CV(RMSE)
although it is difficult to understand the total load being
CDC updated ASHRAE CDC updated ASHRAE lower than this during the latter part of the day. The mea-
base model criterion base model criterion sured heating energy use was higher midday, as would be
8.4% 5% 11.5% 15% expected due to ventilation heating.

4. Conclusions
measured values in June, the same month when measured
gas use began to exceed simulated gas use. A likely expla- Reddy, Maor, and Panjapornpon (2007a) wrote that, while
nation was increased occupant numbers (with associated their “analytical optimization” method was “applicable to
lighting equipment heat gains) as the building population any building energy simulation program, the scope [of
increased after the October 2007 official opening. In the very their application examples] . . . was restricted to the DOE-2
cold regional climate (while rare, snowfall can be experi- program”. They created a customized batch processing pro-
enced in July and August), heating requirements are reduced gram to run the many simulation model variations required
by increased internal gains. It was beyond the resources by their method. In this study, the more common case of a
available to the researchers to track occupant numbers on simulation program with a GUI was considered. Outside a
an ongoing basis. research environment with resources to create a batch sys-
Many data from the electrical logging system were not tem for a given GUI, it is unlikely that modellers would have
used in adjusting the model due to uncertainty regarding the time and resources to create a batch system to run sim-
some logged circuits. Despite this, the GOF was about 60% ulations. The work involved in sensitivity analysis could
above the acceptance threshold. There were no relevant be offset to some extent, by the development of libraries
parameters in the model that could be used as a proxy for of “heuristic templates” akin to the building-specific ones
uncertain electrical data, nor was it appropriate to input advocated by Reddy, Maor, and Panjapornpon (2007a) that
the data as a sensible process load, because of uncertainty would provide characterization of parameters by climate
regarding spatial allocation of this load. region and building type. An alternative would be to use a
different calibration approach, such as the signature anal-
ysis approach (Liu et al. 2005). As Raftery, Keane, and
3.3. Evaluation of hourly heating plant load O’Donnell (2011) have argued, the confidence in calibration
The heating plant thermal output metering allowed com- is much higher when hourly data are considered.
parison of simulated and measured hourly values on an The test building was a 12,000 m2 high-performance
hourly basis. This was carried out from 16 January 2009 to “complex” (dual energy) cold climate building. More than
June 2009, after replacement of a return water temperature 200 model variations were created manually to test model
sensor found to be faulty in 2008. The 16 January results variations. This generated 27 models that met the monthly
(Figure 5) were similar to those for other winter days. The GOF criteria for electricity, but had fit values for gas that
simulated total heating load shows a surprising pattern – were about 5–7 times the NMBE acceptance threshold. Five
very high at night and near 0 in the latter half of the day. models met the criteria for natural gas and had acceptable
Fan and outdoor schedules were checked to ensure they CV(RMSE) for electricity, but NMBE was about 100% too
were zeroed overnight. To better understand the estimate, high. Use of finer interval (hourly) monitored data yielded
 Journal of Building Performance Simulation 357

a model with electrical NMBE about 60% above the accep- commercial and institutional buildings. For example, Fisk
tance threshold, and gas use about 200% above. Modelling et al. (2012) conducted a major study of DCV in schools
of the boiler plant was a known and quantified primary with measurement of CO2 levels, but energy effects were
source of problems with GOF. studied by simulation. Even less was found on in situ per-
Standard boiler part load efficiency curves show higher formance of enthalpy recovery systems in commercial and
efficiency with lower load, which is the case in other major institutional buildings. While Shang and Besant (2008) pro-
simulation programs (e.g. in ESP-r as per Cockroft, Samuel, posed methods for assessing heat recovery effectiveness,
and Tuohy 2007). Contrary to this, input–output analysis none of the articles that cited this publication seem to have
that was undertaken for the case study building around the carried out field measurements. Current versions of major
same time as the calibration study (aaa 2013, personal com- simulation programs (e.g. DOE2) treat effectiveness as a
munication) found boiler plant efficiency declining with single value, while, for many systems, it declines at outdoor
load at two buildings. Relatively little has been published temperatures that cause frosting.
on in situ performance of heating plants, but Ratkovich Discrepancies between modelled and actual perfor-
et al. (2011) also found lower than expected efficiency. mance for boilers, ventilation heat recovery systems, and
Orr, Lelyveld, and Burton (2009) found that efficiency DCV systems will result in discrepancies between measured
Downloaded by [The University of British Columbia] at 19:00 29 October 2014

was relatively constant except at very low loads (below and simulated whole building energy use that will, gener-
about 10% of capacity) for about 60 condensing boilers ally, increase with climate severity. More field studies are
with return water temperature in the condensing range in needed to better characterize actual system performance.
single-family dwellings. Butcher’s (2007) lab study of a Methods that rely on monthly utility data (e.g. as opposed
condensing boiler of unstated capacity also found declining to short-term subsystem monitoring) will be particularly
efficiency with declining load, although no explanation was susceptible to errors in calibration. That is, models may be
offered. It would be difficult to accurately model and cali- calibrated but erroneous due to uncertainties in subsystem
brate a simulation model with any method if the algorithm performance, which could result in problems such as error in
representing a major energy using system has large discrep- evaluating the cost-effectiveness of energy efficiency mea-
ancies with respect to measured performance. It was also sures in energy performance contracting. Raftery, Keane,
found that the pattern of whole building hourly heating load and O’Donnell (2011) developed and advocated a method
computed by DOE2 was quite different from that measured based on use of hourly measured subsystem energy-use val-
in the building, although the daily match was much better. ues, commenting that “monthly data analyses can easily
On an annual basis, the measured and simulated total miss significant errors at a daily or hourly resolution”. The
energy use were reasonably close, about 1320 GJ measured results provided in this paper show that higher resolution
electricity use versus 1050 GJ simulated and about 5040 GJ data can be crucial in forming a correct understanding of
measured natural gas use versus 5400 GK simulated for the performance of energy systems. It will likely be many
the simulation model pre-calibration, after adjustment for years before such high-resolution data are commonly avail-
observed operating conditions (e.g. as opposed to perfor- able for buildings, in which case approaches that allow for
mance rating values for lighting in areas yet to be fitted undetermined outcomes, such as Reddy, Maor, and Pan-
out). In term of performance rating, this is reasonably good. japornpon (2007a), could be useful. However, more detailed
However, in terms of systems analysis for purposes of eval- characterization of field performance and modelling of key
uating EEMs and ongoing commissioning, much better fits systems such as boiler plants, ventilation heat recovery, and
would be desirable. DCV will be required to use even such less precise methods
While the model fit was poor, the measured energy with confidence.
use was low, especially considering the climate – about It is recommended that major combustion devices in
500 MJ/m2 , which is around the median of the simula- commercial and institutional buildings over about 2000 m2
tion estimate for LEED platinum buildings (Frankel and floor area (e.g. boilers, parking garage make-up air units) be
Turner 2008). It may be that full occupancy of the build- equipped with instrumentation (e.g. gas meters, flow meters
ing will yield better model fit, since lighting and equipment on hot water loops) to allow assessment of input–output
energy use would increase, making distortions from other efficiency. This is relatively inexpensive if included in con-
systems less significant. It would be useful to redo the struction documents at the time of new construction. Gas
calibration under these conditions. It is expected that heat- meters should be as high in resolution as possible.
ing would decrease, due to these internal gains. It would Torcellini et al. (2006) recommended that “monitoring
also be useful to do the calibration with another simula- systems should be separate” from the BMS. The authors
tion engine, especially with respect to short-term (hourly) feel this has advantages and disadvantages. Installing a sys-
heating performance. tem that would parallel all of the data points covered by the
The CDC was also equipped with ventilation heat BMS would be very expensive, even if a dedicated monitor-
recovery (AHU-1 only) along with room and central CO2 ing system would likely have greater reliability in terms of
demand-controlled ventilation (DCV). Relatively little has data collection. A separate monitoring system also requires
been published on the field performance of these systems in knowledge of operation of an additional system, which may
 358 M.J. Gestwick and J.A. Love

be possible in a research setting, but daunting in routine ConstructionPro Network. n.d. “DOE Unveils New Version of
building operations. The logistics of funding a secondary Building Energy Modeling Software.” Accessed September
monitoring system may also be an issue. Data collection 2013. http://constructionpronet.com/Content_Free/122211
part2.aspx
via the BMS is likely more readily accepted in construction Fisk, W. J., M. J. Mendell, M. Davies, E. Eliseeva, D. Faulkner, T.
documents than adding an additional system. Hong, and D. P. Sullivan. 2012. “Demand Controlled Ven-
As building energy-use intensity declines, systems that tilation and Classroom Ventilation.” Accessed July 2013.
formerly contributed a negligible fraction of annual energy http://homes.lbl.gov/
use (e.g. elevators in a four-storey building in the case of sites/all/files/lbnl-6258e-demand_controlled_ventilation.pdf
Frankel, M., and C. Turner. 2008. “How Accurate Is Energy
the CDC) may account for enough of energy use to affect Modeling in the Market?” In Proceedings of the ACEEE
calibration. Identifying and monitoring such “background” 2008 Summer Study on Energy Efficiency in Buildings.
systems in large, complex buildings can be difficult. Vol. 3, 88–101. Washington, DC. Accessed June 2013. http://
newbuildings.org/sites/default/files/ModelingAccuracy_
Acknowledgements FrankelACEEE2008_0.pdf
IPMVP. 2006. “International Performance Measurement and Ver-
Rick Smith provided high quality weather data from the Univer- ification Protocol: Concepts and Practices for Determining
sity weather station. Chris Lashmar, Wei Tian, Danielle Coolman,
Downloaded by [The University of British Columbia] at 19:00 29 October 2014

Energy Savings in New Construction.” Volume III, Part I.

Anthony Dratnal, and Kim Gould, assisted with data collection. No. EVO30000-1:2006. Efficiency Valuation Organization,
Past President of the University of Calgary Dr. Harvey Weingarten IPMVP New Construction Subcommittee. Accessed June
and Past Dean of the Faculty of Environmental Design Dr. Brian 2013. http://smartenergy.arch.uiuc.edu/pdf/clearinghouse/
Sinclair had the vision to commit the University to a LEED Plat- IPMVPNewConstruction.pdf/
inum certification for the CDC; this created the opportunity to Kaplan, M. B., J. McFerran, J. Jansen, and R. Pratt. 1990. “Rec-
carry out this research. The University of Calgary and the Natural onciliation of a DOE2.1C Model with Monitored End-Use
Sciences and Engineering Research Council of Canada provided Data for a Small Office Building.” ASHRAE Transactions 96
financial support for this work. (1): 981–993.
Lam, J. C., and S. C. M. Hui. 1996. “Sensitivity Analysis of Energy
Performance of Office Buildings.” Building and Environment
References 31 (1): 27–39.
Abushakra, B., A. Sreshthaputra, J. S. Haberl, and D. E. Lam, J. C., K. K. W. Wan, and L. Yang. 2008. “Sensitivity Analysis
Claridge. 2001. “Compilation of Diversity Factors and and Energy Conservation Measures Implications.” Energy
Schedules for Energy and Cooling Load Calculations.” Conversion and Management 49 (11): 3170–3177.
ASHRAE 1093-RP. Texas A&M University Energy Sys- Liu, M., D. E. Claridge, N. Bensouda, K. Heinemeier, S.
tems Laboratory. Accessed June 2013. repository.tamu.edu/ U. Lee, and G. Wei. 2005. “Manual of Procedures for
handle/1969.1/2013/ Calibrating Simulations of Building Systems.” Berkeley,
Akaike, H. 1985. “Prediction and Entropy.” In A Celebration of CA: Lawrence Berkeley Laboratory. Accessed May 2013.
Statistics, edited by A. C. Atkinson and S. E. Fienberg, 1–23. http://btus.lbl.gov/HPCBS/pubs/E5P23T2b.pdf
Berlin: Springer Verlag. Mottillo, M. 2001. “Sensitivity Analysis of Energy Simulation
ASHRAE. 1989. ASHRAE 90.1-1989 Energy Standard for Build- by Building Type.” ASHRAE Transactions 107 (2): 722–
ings Except Low-Rise Residential Buildings. Atlanta, GA: 732.
ASHRAE. NRCan. 2007. “Modeling Guide for EE4 version 1.7.” Office
ASHRAE. 2002. ASHRAE Guideline 14-2002: Measurement of of Energy Efficiency and CANMET Energy Technology
Energy and Demand Savings. Atlanta, GA: ASHRAE. Centre, Natural Resources Canada. Accessed June 2013.
ASHRAE. 2010. ASHRAE 90.1-2010 Energy Standard for Build- http://canmetenergy-canmetenergie.nrcan-rncan.gc.ca/eng/
ings Except Low-Rise Residential Buildings. Atlanta, GA: software_tools/ee4/modelling_guide.html/
ASHRAE. Orr, G., T. Lelyveld, and S. Burton. 2009. “Final Report: In-Situ
Beliso, E. Huestis, and M. D’Alboro. 2012. “Boiler Research Monitoring of Efficiencies of Condensing Boilers and Use of
Project – ASHRAE Standard 155P – Phase II.” Pacific Secondary Heating.” Cheltenham: GASTEC. Accessed June
Gas and Electric, San Ramon, CA. Accessed June 2013. 2013. http://www.energysavingtrust.org.uk/Publications2/
http://www.etcc-ca.com/reports/boiler-research-project- Housing-professionals/Heating-systems/In-situ-monitoring-
ashrae-standard-155p-%E2%80%93-phase-ii of-efficiencies-of-condensing-boilers-and-use-of-secondary
Besant, R. W., and J. D. Bugg. 1983. “The Performance of a -heating-trial-final-report
Counterflow Air to Air Heat Exchanger with Water Vapor Raftery, P., M. Keane, and J. O’Donnell. 2011. “Calibrating Whole
Condensation and Frosting.” ICHMT Digital Library Online Building Energy Models: An Evidence-Based Methodol-
19: 413–422. ogy.” Energy and Buildings 43 (9): 2356–2364.
Butcher, T. A. 2007. Performance of Integrated Hydronic Sys- Ratkovich, B., T. Martin, R. Sawatzky, D. Alfonso Ponte, and
tems. BNL Report BNL-79814-IR. Upton, NY: Brookhaven S. Vukadinov. 2001. “Measurement and Verification of
National Laboratory. Accessed June 18, 2013. http://forum. Hot Water Boiler Plant Thermal Efficiency in Real Oper-
oiltechtalk.com/attachments/fullreportbrookhavenefficiency ating Conditions.” In Proceedings 19th National Confer-
test.pdf ence on Building Commissioning, August 10–12, 2011,
Cockroft, J., A. Samuel, and P. Tuohy. 2007. Development of Cincinnati, OH. Beaverton, OR: Building Commission-
a Methodology for the Evaluation of Domestic Heating ing Association. Accessed June 2013. http://www.cesgroup.
Controls. Phase 2 of a DEFRA Market Transformation Pro- ca/images/industrial-commissioning/presentations/NCBC-
gramme Project. Glasgow: Energy Systems Research Unit – 2011-Paper-CES-Final.pdf
University of Strathclyde. Accessed June 2013. www.esru. Reddy, T. A., and I. Maor. 2006. Procedure for Reconciling
strath.ac.uk/Documents/07/BRE_E302_040707.pdf Computer-Calculated Results with Measured Energy Data:
 Journal of Building Performance Simulation 359

Final Report, submitted to ASHRAE under Research Project Torcellini, P. A., M. Deru, B. Griffith, N. Long, S. Pless,
1051-RP, Drexel University. R. Judkoff, and D. B. Crawley. 2006. “Lessons Learned
Reddy, T. A., I. Maor, and C. Panjapornpon. 2007a. “Calibrat- from Field Evaluation of Six High-Performance Build-
ing Detailed Building Energy Simulation Programs with ings.” National Renewable Energy Laboratory. Accessed
Measured Data – Part I: General Methodology.” HVAC&R June 2013. http://www.nrel.gov/docs/fy04osti/36290.pdf
Research 13 (2): 221–241. University of Calgary. 2013a. “Child Development Centre.”
Reddy, T. A., I. Maor, and C. Panjapornpon. 2007b. “Calibrat- Accessed June 2013. http://www.ucalgary.ca/fmd/buildings/
ing Detailed Building Energy Simulation Programs with child_development_centre/
Measured Data – Part II: Application to Three Case Study University of Calgary, 2013b. “Interactive Room Finder Map.”
Office Buildings.” HVAC&R Research 13 (2): 243–265. Accessed June 2013. http://www.ucalgary.ca/map/interactive
Shang, W., and R. W. Besant. 2008. “Theoretical and Experimental University of Calgary. 2013c. “Weather Research Station.”
Methods for the Sensible Effectiveness of Air-to-Air Accessed June 2013. https://www.ucalgary.ca/fmd/buildings/
Energy Recovery Wheels.” HVAC&R Research 14 (3): weather_station/
373–396. USDOE. 2013a. “United States Department of Energy, Whole
Shaw, C. Y., J. T. Reardon, and M. S. Cheung. 1993. “Changes Building Analysis: Energy Simulation.” Accessed June 2013.
in Air Leakage Levels of Six Canadian Office Buildings.” http://apps1.eere.energy.gov/buildings/tools_directory/subj
ASHRAE Journal 35 (2): 34–36. ects.cfm/pagename=subjects/pagename_menu=whole_buil
Downloaded by [The University of British Columbia] at 19:00 29 October 2014

Tian, Z., J. A. Love, and W. Tian. 2009. “Applying Quality ding_analysis/pagename_submenu=energy_simulation

Control in Building Energy Modeling: Comparative Simu- USDOE. 2013b. “United States Department of Energy, Ener-
lation of a High Performance Building.” Journal of Building gyPlus.” Accessed June 2013. http://apps1.eere.energy.gov/
Performance Simulation 2 (3): 163–168. buildings/tools_directory/software.cfm/ID=34/