Building Performance Simulation Tools:

Selection Criteria and User Survey

Research Based Report

31 January 2010

prepared by

Shady Attia,

Architecture et climat,
Université catholique de Louvain,
Louvain La Neuve,
Belgium
Disclaimer
The contents of this report are based on research conducted by Architecture et climat at
Université catholique de Louvain.
The views, opinions, findings, conclusions and recommendations contained herein are those of
the authors and should not be interpreted as necessarily representing policies or endorsements,
either expressed or implied, of Université catholique de Louvain.

© Université catholique de Louvain 2010

2
Table of Contents

0. Summary............................................................................................................................. 4
1. Introduction ......................................................................................................................... 1
1.1 Problem context ............................................................................................................. 5
1.2 Definition of problem....................................................................................................... 6
1.3 Objective and Methodology............................................................................................ 6
2. Tools Selection Criteria ...................................................................................................... 7
2.1 Major interested bodies................................................................................................. 7
2.2 Previous studies............................................................................................................ 8
2.3 Tools selection criteria................................................................................................... 9
2.3.1 Usability and Information Management of the interface ............................................... 9
2.3.2 Integration of Intelligent design Knowledge-Base......................................................... 9
2.3.3 Accuracy and Ability to simulate Detailed and Complex building Components........... 10
2.3.4 Interoperability of Building Modelling .......................................................................... 10
2.3.5 Integration of tools in Building Design Process........................................................... 11
3. Questionnaire..................................................................................................................... 12
3.1 Survey 1 .......................................................................................................................... 12
3.2 Survey 2........................................................................................................................... 13
4. Analyzing Results............................................................................................................... 13
5. Discussion ......................................................................................................................... 22
5.1 Tools selection criteria ................................................................................................. 22
5.2 The Gap ...................... ................................................................................................. 25
6 Conclusions ....................................................................................................................... 27
7 References ........................................................................................................................ 28

Appendix 1. Final Survey Results ........................................................................................ 31

3
 Summary:
This report reviews potential challenges and opportunities for using Building Performance Simulation
(BPS) tools. After reviewing current trends in building simulation, it outlines major criteria for BPS tools
selection and evaluation based on analyzing user’s needs for tools capabilities and requirement
specifications. The research methodology is based on a literature review and two online surveys. The
findings are based on an inter-group comparison between architects’ vis-à-vis engineers’. The aim is to
rank BPS tools selection criteria and compare ten state-of-the-arts BPS tools in the USA market. Lastly,
the paper presents five criteria composed to stack up against theories and practices of building
performance simulation. Based on the experience gained during the survey, the suggested criteria is
critically reviewed and tested. The final results indicate a wide gap between architects and engineers
priorities and tools ranking. This gap is discussed and suggestions for improvement to current BPS tools
are presented.

4
1. Introduction BPS which resulted in a broader user’s base.
1.1 Problem context Simulation tools moved progressively towards all
Since the inception of the building simulation professions involved in design of buildings
discipline it has been constantly evolving as a including architectural designers. Architects, who
vibrant discipline that produced a variety of have been regularly described in literature as
Building Performance Simulation (BPS) tools that non-specialist, non-professional, non-experts,
are scientifically and internationally validated. The novice or generalist (Morbitzer et al., 2001, Ibarra
foundation work for building simulation was done and Reinhart, 2009, Schlueter and Thesseling,
in the 60s and 70s focusing on building thermal 2009, Augenbroe, 2002, Hand and Crawley,
performance addressing load calculation and 1997, Mahdavi, 2005) became engaged in the
energy analysis (Kusuda, 1999; Clarke, 1985; BPS community. Recognizing the implications of
Kusuda, 1970). In the late 70s, and continued design decisions made by the different team
through the 80s, efforts were invested into members on the energy and environmental
analytically validated and experimental testing performance of the building, engaged all design
methods for codes for simulation tools team members in performing simulations. As a
(Augenbroe 2002). This foundation work was consequence, simulation tools became
developed mainly within the research community recognized as design support tools within the
of the mechanical engineering domain. Architecture-Engineering-Construction (AEC)
Simulation tools were developed by technical industry. In fact, simulation became an integrated
researchers and building scientist aiming to element of the design process (Augenbroe 1992;
address the needs of engineers. During those Mahdavi 1998). This resulted into a diverse
early days, the user base of BPS tools was growing user’s uptake addressing more the whole
mainly limited to researchers and experts who design team.
are concerned with detailed energy analysis The second major change was the trend to
applied during design development phases. For progressively move towards early design phases.
example, simulations were performed to estimate Due to the increasing importance of the decisions
peak hourly loads for heating and cooling made early in the design process and their
seasons or predict the annual consumed energy impact on energy performance and cost, several
in order to size and select mechanical BPS tools have been developed to help
equipments for large buildings. architects perform early energy analysis, and
It was until the 90s that the building simulation create more energy efficient more sustainable
discipline reached a certain level of maturation buildings (Hensen 2004).
offering a range of tools for building performance The third change was the rapid sprawl of BPS
evaluation (Hensen, Lamberts et al. 2002). The tools. Today we have a diverse tool landscape for
beginning of the 90s manifested the shift from an all building design professionals. The U.S.
energy consumption focus to many other building Department of Energy (DOE) maintains an up-to-
performance characteristics (Augenbroe 1992). date listing of BPS tools on the Building Energy
For example, the integrated modelling whereby Software Tools Directory (BESTD) website
the heat and mass transfer, air flow, visual and ranging from research software to commercial
acoustic aspects of performance were products with thousands of users
considered. This shift led to a the development to (http://apps1.eere.energy.gov/buildings/tools_dire
a relatively large range function complete tools ctory/) (Crawley 1997). In 2009, the number of
(Clarke, Hensen et al. 1998). By the end of the tools reached more than 378 tool (US-DOE
90s, a range of simulation applications spinned 2009). Between 1997 and 2009 the number of
out from the research community to professional tools has almost quadrupled. Figure 01
practice allowing a diverse tools landscape for a documents the number of developed tools listed
variety of users. For the first time, analytical on the BESTD DOE for that period.
simulation power became at the finger tips of
building designers (Papamichael, LaPorta et al.
1996; Tianzhen, Jinqian et al. 1997).
This maturation of building simulation had a
major influence on the building design profession
and resulted into four major changes namely:

• Diversifying tools users and addressing more

the whole design team
• Modifying the tools to suite early and late
design phases
• Increasing the number of tools and developing
a large range of function complete tools Figure 01, BPS tools developed between 1997- 2009

• Localizing the tools capabilities The forth major change was the localization of
tools capabilities. With the localization of BPS
The first major change was the trend to tools incorporating local weather data and
encourage the whole design team to use BPS provision of local building materials, construction
tools. The increased complexity of building and codes the number of tools users is growing
delivery process has led to a broader view of

5
enormously. High quality thermal models are selection criteria for BPS tools in the architectural
uploaded on earth viewer software (Google and engineering practice. The overall objective is
Earth) and positioned on 2D and 3D satellite identify the barriers that hinder engineers and
images of terrain and cities. We literally can architects from integrating BPS tools in practice.
simply fly over any location on earth and come to Thus the goals of this study are:
a model and run it using BPS tools. With the
rapid advances of computer technology, internet (a) Ranking and identifying generic BPS tools
and building information technology, building selection criteria
simulation will be more often and more widely (b) To conduct an inter-group comparison
applied in building design and analysis worldwide between architects vis-à-vis engineers
offering design solutions, economic analysis, (c) To compare the potential challenges and
comparing & optimizing designs, computing opportunities of using existing BPS tools
performance and verifying compliance{Ellis, 2002 (d) To compare ten state-of-the-arts BPS tools in
#138}. the USA market

1.2 Definition of problem In order to rank and identify the selection

By examining those four major changes we criteria we conducted a literature review that
generally can observe that BPS tools are and will forms the basis for achieving the other goals (b-d)
continue penetrating the building design practice. through an online survey.
Simulation tools became recognized as design The final goal of this paper is to provide
support tools within the AEC industry. In fact, guidance and feedback to BPS tool developers,
simulation became an integrated element of the with particular focus on the different expectation
design process (Augenbroe 1992; Mahdavi of architects’ vis-à-vis engineers’ needs. This
1998). study will allow user feedback to get main
However, a deeper examination to those knowledge into application. The paper provides
previously mentioned changes, we notice that also an up-to date comparison of ten major BPS
they happened so quickly and resulted into a tools: ECOTECT, HEED, Energy 10, Design
growing landscape of tools that is considered in Builder, eQUEST, DOE-2, Autodesk Green
itself as barrier. Today, the continuing growing Building Studio, IES VE, EnergyPlus and Open
number of BPS tools reflects a broader variety of Studio. This can provide a resource for simulation
tools abilities but it does not necessarily reflect a tools developers about architect and engineers
wider penetration within the building design needs. Comparing tools is performed to generate
community. Already there is replication of many use cases that allow recording and identifying the
tools that have striking similarities with no attempt BPS functions required by both groups, in order
to develop design team friendly, effective and to present a checklist that forms a design basis
efficient design decision support applications. for software developers. Finally, the study is
Most BPS tools are difficult and cumbersome to providing a glimpse into the future, in order to
use, and cater more for engineers. The scope of allow the evolution of architecture and
most existing tools is mainly still oriented towards engineering education and profession aiming to
final design stages. Moreover, most tool bridge the gaps between architects and
developers use engineers’ feedback to develop engineers in the professional design practice.
architect friendly tools. The rapid changes could The paper is organized into six sections. The
not bridge the mono-disciplinary R&D inheritance, first section is positioning the research problem
mostly lacking the architects’ viewpoint, which within the BPS community. The second section
does not make BPS suitable for the design (Attia, screens the selection criteria and requirements
2009). Attempts to address the architects and specifications of tools regarding their usability
engineers use of BPS tools have been proposed and functionality. These criteria are reviewed and
separately by many researchers. Very little effort collected from literature and are classified in the
has been carried out to develop BPS tools with second section forming the basis for the two
adaptive interfaces that cater for both groups, online surveys discussed in the third section. The
namely architects and engineers. analysis of the results and the survey findings are
More importantly, there is no independent presented in chapter four. This includes ranking
evaluation and classification of tool usability and of the ten tools and comparison of different
functionality in practice versus users’ type and priorities and preferences of architects’ vis-à-vis
needs (Clarke, 2009). Tools developers rarely engineers. The final two sections are discussing
state the tools capabilities and limitations the survey findings and providing feedback to
(Reinhart, 2006). Potential user is faced with tools developers and architecture and
difficulty of choosing a suitable program among engineering education community.
this growing BPS tools pool.
In brief, we need to identify clearly the needs 2. Tools Selection Criteria
of different users and identify BPS capabilities
The simulation community does not have a
and limitations in order to cross the previous
clear criteria to classify and evaluate the facilities
mentioned barriers.
offered by tools (Crawley, Hand et al. 2008).
There are not yet uniform definition of tools
1.3 Objective and Methodology requirements and specifications based on formal
Therefore, this study seeks to identify and study consultations with users, practitioners and tool
architects and engineers’ requirements and developers (Clarke 2009). For example, there is

6
no common language to describe what the tools sensitivity, versatility, speed and cost,
could do (Crawley, Hand et al. 2005). We note reproducibility and ease of use(ASHRAE 2009).
there are many nuances of the word ‘criteria’, for Additionally ASHRAE developed the standard
example, capabilities, requirements, functionality, method of test for the evaluation of building
specifications, features, factors etc… Also there energy analysis computer programs(ASHRAE
is no clear methodology to compare BPS tools. 2007). The standard specifies test procedures for
Identifying the basic criteria for BPS tools can evaluating the technical capabilities and ranges
support architects and engineers creating more of applicability of computer programs that
efficient and cost effective sustainable buildings, calculate the thermal performance of buildings
as well as facilitating future innovation and the and their HVAC systems. The standard strength
progress of the AEC industry. In order to provide lies in its diagnostic power of the procedure,
the necessary conditions for a evolutionary cycle which helps program authors debug and correct
of tool development; a critical review of the status software errors. However, by examining ASHRAE
quo and in-depth reflections on the tools must be publications we find that they have a specific
achieved (Lam, Wong et al. 1999).Therefore, as focus on accuracy and validity of tools algorithms.
part of this paper a literature review was carried Thus they do not rise to be a complete set of BPS
out to identify, classify and group requirements tools criteria.
and selection criteria for future development of The third body is the International Energy
BPS tools. The following review forms an entrée Agency (IEA). The IEA has created a number of
into the literature. This review forms the basis tasks for evaluating BPS tools. However, most
that will ensure the clarity and relevance of the IEA tasks were focused only on assessing the
questionnaire content and allow tools comparison accuracy of BPS tools in predicting the
in section three. performance neglecting other important criteria.
Task 12 (Building Energy Analysis and Design
2.1 Major interested bodies Tools for Solar Applications) has created a
There are various bodies that could help with number of procedures for testing and validating
building energy modelling and simulation building energy simulation programs. Task 22
information. On top of those bodies, stands the (Building Energy Analysis Tools: Analyzing Solar
International Building Performance Simulation and Low-Energy Buildings) assessed the
Association IBPSA, is a non-profit international accuracy of BPS tools in predicting the
society of building performance simulation performance of widely used solar and low-energy
researchers, developers and practitioners, concepts. Task 30 (Bringing Simulation to
dedicated to improving the built environment. Application) was aiming to investigate why BPS
IBPSA is founded to advance and promote the tools were not widely used in the design process
science of building performance simulation in and to identify ways of overcoming this problem
order to improve the design, construction, (Warren 2002). Task 34 (Testing and Validation
operation and maintenance of new and existing of Building Energy Simulation Tools) is evaluating
buildings worldwide. IBPSA is not particularly the accuracy and capability of analysis and
busy with developing tools selection criteria; but it design tool algorithms and developed the
provides a framework and medium for R&D of BESTEST procedure (Judkoff and Neymark
BPS industry. The bi-annual conference 1995). However, most the tasks focus on
publications are available online and provide a quantitative evaluation measures and the
source for many topics including: simulation & audience for most IEA Tasks that are concerned
users, software environments & paradigms in with BPS tools is limited developers, and energy
addition to tools and interfaces selection. In fact, standards development organizations (i.e.
many IBPSA paper presented attempts to set ASHRAE, BESTEST and CEN). It is very difficult
selection and evaluation criteria of BPS tools. to estimate the benefit of the IEA tasks on tool
However, most these attempts are individual and users, such as architects, engineers, energy
dispersed. There is no formal attempt within consultants, product manufacturers, and building
IBPSA to define a formal tools requirements owners and managers.
specification for practitioners and tool developers. In brief, and based on the overview of the three
Another important body involved with evaluating major bodies that contribute to shape and
BPS is the American Society of Heating, organize the BPS domain, we do not have a body
Refrigerating and Air-Conditioning Engineers that is working to develop tools selection criteria
(ASHRAE). ASHRAE lists what factors to take and requirement specifications. There is a
into consideration when selecting energy analysis common focus among the bodies to guarantee a
tools. According to ASHRAE Handbook the most minimum level of reliability through simulation
important criterion for selecting a tool is the model validation and accuracy. Similar effort
capability of the tool to deal with the project should be taken to create a framework that
requirements. The second is the complexity of regulates the world of BPS tools development
input and the third is the quality of output. Also and assist users to select and evaluate based on
the availability of weather data is a major feature other important criteria. There are many other
of a tool. The forth concern in selecting a tool is criteria that need to be addressed and
the auxiliary capabilities, such as economic accumulated to form a consistent selection and
analysis. Apart from these four criteria there are evaluation process. The DOE BESTD provides a
general factors that must be embedded in any good start for communicating basic information
energy analysis method namely, accuracy, on different tools including their validation,

7
required expertise, audience, input, output, including the adaptability and integration of the
strength and weaknesses. tool for different design phases, users and
different design iterations. Finally, the author
2.2 Previous studies reviewed a wish list of simulation tools and
A number of studies and surveys have been identified what remains unfulfilled of this list
carried out in the past that were concerned with including the ability run perform rapid evaluation
the criteria and requirements of BPS tools. In of alternatives, support decision-making, support
August 1995 and June 1996 the DOE sponsored incremental design strategies for design
workshops to provide input from developers and refinement cycle, detect when the tool is used
users to future planning efforts on future outside its validity range and the robustness of
generation BPS tools (Crawley and Lawrie 1997). nonlinear, mixed and hybrid simulation solvers
The developer’s workshops focused on {Augenbroe, 2002 #41}.
applications, capabilities and methods and In 1996, Lam et al. carried out a survey on the
structures. The user’s workshops focused on usage of performance-based building simulation
application, capabilities, and interfaces. However, tools in Singapore (Lam, Wong et al. 1999). With
the user’s workshop group included mainly one hundred and sixty four valid responses,
software experts, researchers and engineers. including architects and engineers, the survey
Also the workshops did not address the different was organized around six main questions. The
requirements for different design phases. In fact, questions were simple and direct asking about
the final results focused mainly on identifying the reasons of using or not using the tools and
criteria for the development of a heart or asking for the major limitations and obstructions.
calculation engine of EnergyPLUS. User interface Except the question about the ability of the tools
issues were postponed for the future (Crawley to enhance the design process, no other question
and Lawrie 1997). could be considered as major BPS tools selection
Ten years later, when Crawley et al compared criterion.
the features and capabilities of twenty major BPS Then in 2004, Lam conducted a study that
tools. they grouped the comparison criteria into involves the development of a comprehensive
18 category including: results reporting; classification schema for comparing five tools and
validation; and user interface and links to other running a comparative analysis by graduate
programs (Crawley, Hand et al. 2005; Crawley, students. Lam conducted a literature review on
Hand et al. 2008). The grouping was based on well-known energy modelling tools that exist. A
vendor-supplied information with no uniform and comparison of 22 tools was made based on four
standard selection criteria. major criteria, namely, usability, functionality,
Aiming to identify the vital capabilities of BPS reliability and prevalence. Under the usability
tools, Tianzhen Hong identified five vital criteria. criteria he listed the system requirements,
The first capability is the usability. The second Interoperability, user interface, learning curve,
relates to computing capability. The third is the effort to update model, conducting parametric
data exchange capability. The fourth is the studies and processing time. Then, under the
database support. In the final notes of his functionality criteria he listed the
research, published in the year 2000, the author comprehensiveness of geometric and system
highlighted five additional trends that are on the modelling, types of energy calculations, types of
road ahead. The first is the knowledge-based data analysis and presentation and availability of
systems. The second is the BPS for early design other environmental domain simulations (e.g.,
stages. The third is the information monitoring lighting). The third criteria namely reliability
and diagnostic system. The fourth is the included consistency of results and accuracy of
interested building design system. The fifth is the results. The forth criteria was the prevalence
virtual reality (Hong et al., 2000). including compliance with industry standards,
In 2002, Augenbroe presented an overview of documentation, user support and pricing &
the trends in building simulation. In his paper the licensing{Lam, 2004 #190}.
author highlighted the changing team context of In 2005, Hopfe et al identified the features and
simulation and how BPS tools need to be brought capabilities for six software tools and interviewed
into the nucleus of the design team. Also the designers to screen the limits and opportunities
author addressed the interoperability as an for using BPS tools during early design phases
emerging trend. In his paper, Augenbroe referred (Hopfe, Struck et al. 2005). The tools
to functional criteria including, the usability and classification was based on six criteria namely
friendliness of interfaces, for example GUI, the capabilities, geometric modelling, defaulting,
documentation, output presentation, error calculation process, limitation and optimization.
diagnostics, learning curve, adaptive usability, in However, the authors did not report what
addition to the integration of knowledge base methodology was used to compile these criteria.
within tools. Accountability and confidence in In 2005, Punjabi et al, identified major BPS
tools results was also discussed including issues tools usage problems undertaken an empirical
such as validating the model assumption, using testing. The usability testing was based on
performing sensitivity analysis, uncertainty and usefulness, effectiveness, likeability and learn
risk analysis, methods to assert correct data input ability(Punjabi and Miranda 2005). The research
and post-processing of output data to generate defines six indicators for usability and information
performance indicators. On the other hand, the management including interface design,
paper addressed process related criteria navigation, saving and reviewing, database

8
creation and learnability. However, the research Murray 1993 ). This means express information
was only limited to usability and did not include using presentation techniques and media to
other tools evaluation criteria. achieve communicative purposes and support
Summing up, bodies and previous surveys users performing their task (Maybury and
were capable of identifying general trends and Wahlster 1998). In fact, usability is a broad term
needs in the BPS community. However, all these that incorporates better graphical representation
efforts are dispersed and based on individual of simulation input and output, simple navigation
initiatives without a unified consensus based and flexible control. Users would like to see
framework. There is not yet a uniform and clear results presented in a concise and
methodology or outline to assess and define tools straightforward way, with a visual format or 3D
specifications and criteria for developers, spatial analysis preferred to numerical tabulation
practitioners and tools users. The following (Attia, Beltran et al. 2009). For example, CFD is
section presents the five selection criteria that very appealing to architects, engineers and even
under grid current notion of how we can classify clients because of the tremendous explanatory
and evaluate the facilities offered by BPS tools. power of graphical output. Moreover, usability
entails being adaptive. This means that GUI has
2.3 Tools selection criteria to adapt to certain users and certain design
Summarizing the literature findings we found phases. Also usability entails the ability to learn
that the simulation community at large is thinking easily, quickly and to support the user with
about and discussing at least five major training, online help, look-up tables and error-
challenges. As shown in Figure 02 they are traps.
namely, the (1) Usability and Information More to this criterion emerges information
Management (UIM) of interfaces, (2) Integration management, as a growing concern for tool
of Intelligent design Knowledge-Base (IIKB), (3) users. Information management is responsible for
Accuracy of tools and Ability to simulate Detailed allowing assumptions, using default values and
and Complex and building Components templates to facilitate data entry (Donn 2001).
(AADCC), (4) Interoperability of Building Issues such as simulation input quality control,
Modelling (IBM) and the (5) Integration with comparative reports creation, flexible data
Building Design Process (IBDP) as. Under those storage, user customization, simple input review
five titles we classified the sub criteria and as well as input modification are all considered as
challenges, found in literature, that correspond to part of the information management features of
the five topics. any simulation interface (Crawley, Hand et al.
The goal in examining these challenges and 2005).
criteria is not to conduct an exhaustive analysis.
Instead, it is to tease out broad yet critical 2.3.2 Integration of Intelligent design
underlying premises to see if common ones exist. Knowledge-Base (IIKB)
This paper does this and then applies the results The second criterion that generated several
to the surveys to assess how the criteria debates during the last years is the Integration of
compare. Intelligent design Knowledge-Base (IIKB) within
the tool. The concept of IIKB today trades in other
realms under such names as design decision
support and design optimization. Since the
ultimate wish of BPS users is to have tools that
support the design process, the knowledge-base
(KB) supports the decision making (Yezioro
2008). It should give quantitative answers
regarding the influence of the design decisions
(Ellis and Mathews 2002) (Lam, Wong et al.
1999). A common observation from the literature
is that designers cannot estimate the relative
importance of design criteria. “They feel it
continuously throughout the design process,
reformulating it as they compromise between
what is desired and what is possible”
(Papamichael and Protzen 1993). Therefore, the
importance of integrating KB into tools lies in its
ability to provide the user with valuable insights
Figure 02: The five selection criteria and directions during the design process. With
the complexity of the design, next generation of
2.3.1 Usability and Information Management simulation tools must embrace KB. This will add
(UIM) of the interface real power to BPS tools that will contain
The usability and information management of descriptive explanations, templates, building and
the interface refers to the human-computer components examples and procedural methods
interaction. A fundamental feature of a simulation for determining appropriate installation and
tools is to incorporate adaptive GUI that enhance systems, e.g. guidelines, case studies, strategies
the human-computer interaction and overall etc… KB comprising facts and
system effectiveness of simulation (Hefley and heuristic/prescriptive rules for decision taking at

9
least on the level of compliance with building is concerned with all the aspects connected to
codes (e.g. ASHRAE, IECC) and rating systems empirical validation, analytical verification and
(e.g. LEED®, EnergyStar® and Green Globes®), in calibration of uncertainty, as defined by IEA and
addition to be able to assist in adjusting the BESTEST procedure, in order to provide liability
design parameters to the needs within the and a level of quality assurance to the simulation
framework of existing codes. Despite the criticism results (Judkoff, 1995).
to existing BPS tools, which incorporate expert or Another important feature incorporated under
knowledge-based systems, that they may this criterion is the ability to simulate complex and
mislead designers due to defaulting subjective detailed building components, in other words, the
preferences (Papamichael and Protzen 1993) or ability of BPS tools to perform various and
the limited pre-processed rules of thumb (Donn, specific functions with higher model resolution.
Selkowitz et al. 2009), there is great advantage of With the rapid changing building technologies as
incorporating knowledge-base in simulation tools well as knowledge explosion BPS are providing
as an educational means that help more new features and functions allowing simulating
understanding the complex thermo physical the performance of passive design strategies
processes and interactions within building and (e.g. natural ventilation, shading etc...),
environmental control systems (Hand and renewable energy systems, HVAC systems,
Crawley 1997). A knowledge-base plays the role energy associated emissions, cost analysis, life
of justifier that rationalize and explain the building cycle cost analysis (LCCA) in addition to new
behaviour and in the same time guide the user building elements such as green roofs, double
during the decision making process. skin facades, chilled beams, atria, concrete core
Another very practical ramification of IIKB is the conditioning etc... Therefore, we defined this
intelligence that is namely defined as design criterion (AADCC) as a pervasive and persistent
optimization. The intelligence entails finding criterion for tools selection and evaluation.
quantifiable answers to design questions in order
to create context specific analysis, evaluate 2.3.4 Interoperability of Building Modelling
complex design strategies, optimize design (IBM)
solutions, engage ‘what if’ scenarios, verify Next, we define a criterion that incorporates
compliance and analyze life cycle (LC) and data exchange and the interoperability of building
economical aspects. With the increasing model. The Interoperability of Building Modelling
analytical power of BPS tools we can examine (IBM) responds to the ability to manage and
sensitivity and uncertainty of key parameters in communicate building data between collaborating
relation to design-decisions (Bambardekar and firms and within individual companies design,
Poerschke 2009), compare various concepts, construction and maintenance. The IBM is a
rank, quantify parametric and even generate fundamental criterion for assessing BPS tools
semi-automatically design alternatives (Hensen because it allows multidisciplinary storing of
2004 ). BPS will never replace good design information with one virtual representation. The
judgment, but it will calibrate and inspire it. Thus, need for sharing information and rapid feedback
one of the most important selection and exchange between various design professions
evaluation criteria of BPS in future is the ability to emerged in the 90s (Ellis and Mathews 2002).
assist design teams and answer qualitative and Significant research and development has been
quantitative design questions during the design carried out to integrate simulation tools with
process. computer aided design (CAD) applications.
However, it has been frequently reported that
2.3.3 Accuracy and Ability to simulate software application process the same building in
Detailed and Complex building Components different representations and formats and the
(AADCC) integration of BPS tools with CAD application is
The Accuracy and Ability to simulate Detailed not sufficient (Lam, Wong et al. 1999). Aiming to
and Complex building Components (AADCC) is improve the integration and alliances between
the most popular criterion found in literature for engineers, architects and even constructors to
selecting and evaluating BPS tools. Under this create realistically integrated projects together
criterion we meant to include all aspects and overcome the differences between the logical
regarding the validity and quality of simulation model and the realities of AEC industry practice,
models and their resolution. Since the inception the Industry Foundation Classes (IFC) standard
of BPS discipline, research has been carried out evolved as an international information exchange
to provide analytical verification, empirical standard that allows project participants to work
validation and comparative testing studies across different software applications. It is
(ASHRAE, 2007; Judkoff, 1995). Since, all possible to write IFC interfaces to HVAC design
building models are simplification and abstraction and simulation tools and cost estimation tools. It
of reality therefore there is no such thing as a is also possible to import building geometry data
completely validated BPS tool. However, the from CAD. This allows the exchange of HVAC
importance of this criterion is that it guarantees a data and performance specification, construction
changing common-accepted agreement properties, geometry. Comparison of
representing the state-of-the-art in whole BPS performance and cost (Bazjanac 2003; Bazjanac
programs. BPS tools are eminently challenged to 2004).
represent physics accurately by the mathematical Later, as an application of the IFC formats,
and numerical models. Thus the term ‘accuracy’ emerged the Building Information Modelling

10
(BIM), a model-based technology that is linked to complexity. The aim was to assign due
a project information database (AIA 2007). BIM responsibilities to specialist so that they can
technology involves the creation and use of contribute their specific knowledge. However, this
coordinated, consistent information about a move resulted into the fragmentation and
building. It allows better decision making, compartmentalization of the design process
documentation and accurate prediction of (Mahdavi 1998). As consequence, the
building performance. In the recent five years, fragmentation has been echoed in the BPS
BIM became a comprehensive depository of data domain. Today most BPS tools cater to only one
that are accessible by many software applications discipline or only one design phase. Most BPS
that take part in the AEC industry projects tools are still easier to use in developed design
(Bazjanac and Kiviniemi 2007). Recent market phases. They help designers to improve their
surveys show that 48% of the architectural offices basic concepts, not to create the basic concepts
in the US already use methods of building (Donn, 2001).
information modelling (AIA 2007). Direct links In fact Balcomb (1992), Tianzhen Hong (2000)
between BIM or non-BIM modelling tools, such as and Ellis (2002) classified the BPS tools used
the SketchUpTM plug-in for IES and Energy Plus during the design process mainly into two groups.
or Revit Architecture plug-in IES and ECOTECT The first is the advanced design stages
are an important ramification of BIM technology evaluation tools mainly used by engineers. The
enabling the creation of deliverable that has an second is the guidance tools used by architects.
explicit relationship to each other, resulting in The early design phase tools are called design
better coordinated and seamless data exchange tools (DT) and the late design phase’s tools
that time, resources, effort and assures quality called detailed simulation programs (DSP). DTs
based liability and reduces risk. are more purpose-specific and are often used at
However, the success of BIM is limited to the the early design phases because they require
detailed design phase because it ensures access less and simpler input data. For example, DTs
for the design team to BPS tools, only after the are very useful in the compliance checking of
whole building design has been completed. The prescriptive building standards. Because DTs are
proposition to embrace BIM during early design easy to develop and test they proliferated. On the
phases will result in adding complexity by limiting other hand, DSPs often incorporate
and freezing the design choices during the most computational techniques such as finite
critical design phase (Eisenberg, Done et al. difference, finite elements, state space, and
2002; Donn, Selkowitz et al. 2009). Therefore, transfer function for building load and energy
BPS tools still find a limited application during calculation. Besides design, DSPs are also useful
early design phases. Therefore, we should keep in the compliance checking of performance-
in mind that BIM is an application within the based building energy standards {Hong, 2000
broader definition and objectives of the #37} {Balcomb, 1992 #109} {Ellis, 2002 #138}.
interoperability of building modelling. Thus, the On contrast, BPS tools should be adaptive and
challenge that is facing the IBM is to assure design process centric as proposed by many
utmost interoperability by fluidizing model experts (Hayter, Torcellini et al. 2001; Mendler,
representation, allowing low and high resolution Odell et al. 2006; De Wilde and Prickett 2009).
building models that correspond to all design With the growing importance in bridging this gap
phases and allow a design team based model. and integrating BPS tools during the whole
building design delivery process simulation
2.3.5 Integration of tools in Building Design should be used as an integrated element of the
Process (IBDP) design process (Augenbroe 1992; Mahdavi
The final criterion assesses the tool Integration 1998). Experience has showed that high
in the Building Design Process (IBDP). The performance buildings (e.g. passive houses, low
building design process is a dynamic process of energy and zero energy buildings) cannot depend
creating concepts that involve design strategies on intuitive design and therefore simulation tools
and technologies and then predicting and should be an integral part of the design process
assessing their performance with respect to the (Torcellini, Hayter et al. 1999; Hayter, Torcellini et
various performance considerations within the al. 2001). For example, the integration of BPS
specific design context {Hien, 2003 #105}. tools during early design phases can influence
Within the building design community there is better design to achieve our millennium
constant complains that BPS tools are not objectives (Robinson 1996; Mahdavi, Silvana et
adaptive to the design process and its different al. 2003; Morbitzer 2003). In order to encourage
phases and cannot be integrated into the design designers to use simulation tools, IBDP tools
process (Morbitzer, Strachan et al. 2001; Yezioro should be provided allowing the adaptive use for
2008). There is evidence that existing tools lacks different purposes, by different users and at
the capabilities to deal with the nature of design different design stages (Tianzhen, Jinqian et al.
process. The tools does not match the design 1997). Thus the IBDP became a basic criterion
process (Lam, Wong et al. 1999; Ellis and for BPS tools selection and evaluation.
Mathews 2002). According to Mahdavi (1998), In brief, in this section we defined the five
the increasing complexity involved in the design criteria that reflect the main stream in simulation
process resulted into mono-disciplinary, community. The inherent limits to a synopsis of
specialist-professional approach that emerged as the five influential selection criteria are apparent.
an attempt to address the design process However, these five criteria are more linked than

11
the categorization suggest. In order to guarantee into one question group. They also advised
plausible and persuasive selection criteria, this launching two surveys during different time
categorization form the basis for the surveys periods to guarantee the maximum participation
questionnaire. The surveys provide the of respondents. The final step, prior to launching
opportunity to test and critically judge the the survey, was to include reviewers’ feedback
selection criteria. The next section explores the and conduct several modifications to the format
questionnaire design and execution. and content the two final surveys.
Participants were recruited through email
3. Questionnaire: invitations to the mailing lists and forums of the
The tools selections criteria were used to form ten above mentioned tools, in addition to the AIA
the basis for of the survey questionnaire. The Committee on the Environment (COTE),USGBC,
questionnaire was dedicated to gathering 2030 Challenge, 2007 Solar Decathlon entry
information from beginner simulation tools users teams and the building performance simulation
including architects, engineers, designers and mailing lists (Bldg-SIM, Bldg-RATE, IBPSA-USA).
fresh graduate students who are concerned with Departments and schools of architectural
sustainable building design in the USA. The engineering, environmental design and
survey aimed to probe the users’ perception of architecture in addition to students’ chapters,
the most important criteria on the use and students’ blogs and architecture firms in the USA
function of ten major USA market tools. How were approached including the winning offices of
users utilize, and benefit from the tools during the the Top Ten Green Architecture Award between
design decision process 2005 and 2008.
Prior to launching the survey the authors set up
an online test version. Comments and Survey 1
suggestions were requested from peer reviewers. The first survey was hosted at eSurveyPro.Com
Reviewers were asked to revise the and was launched between mid December 2008
questionnaire and provide critical feedback in and mid February 2009 including 22 questions.
order to optimize the structure, clarity and An invitation letter was included within the email
relevance of the questionnaire before posting the body with a link to the survey web link. The
final version online. Also reviewers were asked to questionnaire’s home page clearly stated that the
screen and list top-ten BPS tools, using the U.S. questionnaire purpose, focus group and duration
BESTD list and the comparison study by Crawley (see Figure 03).
et al in (Crawley, Hand et al. 2005). The selection
had to represent an overview of state of the art
BPS tools used by architects and engineers in
the USA (DOE 2009). The list was narrowed
down to those tools that are used for evaluating
and analyzing the building energy performance.
Lighting and CFD simulations have been
excluded because they are disconnected from
the energy and thermal performance realm. As a
result, eight tools namely, ECOTECT (Autodesk
2008; Autodesk 2009), HEED (UCLA 2008;
UCLA 2009), Energy 10 (E10) (NREL 2005; Fig. 03, Survey 1: (mid December 2008- mid February
NREL 2009), Design Builder (DB) (DesignBuilder 2009)
2008; DesignBuilder 2009), eQUEST (LBNL and
Hirsch 2009; LBNL and Hirsch 2009) , Autodesk As an incentive to complete all the survey
Green Building Studio (GBS) (Autodesk 2008; questions the respondents were promised to
Autodesk 2009), IES Virtual Environment Viewer receive the final survey summary report. The
plug-in (IES VE plug-in) (v.5.8.2) and average duration for taking the survey was
SketchUP/Open Studio (OS) plug-in (Google approximately 8 to 12 minutes. A welcome page
2009; NREL 2009) were selected plus ‘raw’ explained the objective of the survey, informed
DOE-2 (LBNL and Hirsch 2008; LBNL and Hirsch participants of the approximate survey duration,
2009) and Energy Plus (EP) (DOE 2009; DOE and defined the expected target group. Including
2009). Reviewers suggested adding DOE-2 and the above mentioned issues, the page listed the
EP to broaden the range of examined tools. First, tools that will be inquired. The questionnaire was
to allow comparing tools that are capable of structured into three parts. The first part was
making overall energy analysis in the early screening the respondent’s background and
design phase, versus tools capable of making experience with BPS tools. The second and third
detailed analysis in later design phases. parts of the survey focused on the following key
Secondly, to allow comparing the sensible use of criteria:
tools vis-à-vis the amount of knowledge required (1) The usability and information management
for and by each tool. Thirdly, to compare tools (UIM) of interface and (2) the integration of
with developed graphical user interface (GUI) intelligent design knowledge-base (IIKB),
versus tools with text based user interface. including the (3) IBDP. The respondents were
Due to the questionnaire’s density and length, asked not only to judge the relevant importance
reviewers recommended a shorter version. of the above mentioned criteria, but also to share
Questions regarding IIKB and IBDP were merged their experience by comparing longitudinally the

12
ten selected tools. An open question followed questionnaire is based on an open sample and
every part of the questionnaire in order to allow therefore, the four respondents sample groups
respondents to share their thoughts and cannot be proven to be representative for the
comments. At the end of the survey respondents engineering or architecture community. However,
were invited to post their ideas about current the number of respondents of both surveys was
limitations or improvements that should be quite reasonable to allow the identification of
avoided or integrated in the future development patterns and conduct cross-discipline analysis
of BPS tools. (Pilgrim, Bouchlaghem et al. 2003).

Survey 2 4.1 How do you describe your current position?

The second survey was launched between mid
July 2009 and mid October 2009, including 16 This question allowed the respondents to choose
questions. Both surveys were structured to from 12 available categories including architect,
include the same introduction entailing eight engineer, architecture or engineering designer,
questions. The eight questions were addressing interns, educators, graduate student and)
the respondent’s background and experience undergraduate student. Remarkably under the
with BPS tools. The second and third part of the “Other” option a number of construction
survey focused on the following key criteria: (3) management professionals and students
Interoperability of Building Modelling (IBM), (4) provided complete responses. However, in order
Accuracy of tools and Ability to simulate Detailed to conduct the inter-group comparison between
and Complex and building Components (AADCC) architects and engineers all categories were
as shown in Figure 04. binned into two main groups. Participants that did
not fall into any of the 12 categories were
excluded. This step was necessary in order to
detect any disciplinary difference between both
tools users. Figure 05 shows the number of
respondents grouped for each survey. The
sample size of each group in both surveys was
almost equal. Thus, we could say that on average
the magnitude of each group is balanced which
allows us to compare votes.

Figure 04: Survey 2: (mid July mid October 2009)

4. Analyzing Results:
The first survey was closed after two months.
The second survey was closed after three
months to ensure a balanced participation
compared to the first survey sample. The user’s
responses were stored and results summaries
were automatically generated. The first survey
attracted over 800 interested visitors. However, Figure 05: Number of respondents in both surveys
the automatic report filtering generated only 481
eligible respondents. The second survey 4.2 Which of the following affiliations apply to
attracted over 750 interested visitors with 417 you?
eligible respondents. Many respondents opted
not to complete the survey till the end. The The second question revealed the participants’
responses came from various IPs of users that affiliation. In the first survey, 18% of the architects
answered the survey. IP responses from outside were AIA accredited with almost the same
the USA and uncompleted responses were proportion (17%) in the second survey. On the
excluded. Questions 4.1-4.8 are representing the other hand, more than a quarter of engineers
8 introduction questions for both surveys. The (27%) were ASHRAE Professional Engineers
results of both surveys are summarized below. (PE) in the first survey, with a higher
Questions 4.9-4.11 are representing the UIM. representation in the second survey (30%). Next,
Questions 4.12-4.14 are representing the IIKB 21% of the first survey respondents and 19% of
and IBDP. Questions 4.15-4.17 are representing the second survey respondents were LEED
the AASDC. Questions 4.18-4.19 are accredited professional including architects and
representing the IBM. Then question 4.20 ranks engineers. The summary report indicates the
the most important features of BPS from the point participation of 44 LEED AP architects in the first
of view of architects and engineers. Finally, survey and 31 in the second survey. Surprisingly
Figure 21 and 22 compiles the respondents’ LEED AP engineers were more than architects.
choices and ranks the ten tools automatically However, the encouraging finding here is that
according to the two different groups’ preference. both groups are acknowledging LEED, as a
Thus, prior to analyzing the survey results it is common ground, and are seeking for professional
very important to question the statistical accreditation.
significance of the survey. In fact, the

13
 Figure 08: Geometric modelling tools used by both
groups

Figure 06: Respondents’ affiliations and certifications. Figure 08 indicates a remarkable difference
between architects and engineers. Both groups
use different tools for geometric modelling.
4.3 What of the following energy simulation tools
Architects are more in favour of SketchUp while
do you use?
engineers are in favour of CAD applications.
Therefore, finding a common geometrical
As seen in Figure 07, most architects have
modelling medium is still a challenge.
used ECOTECT. eQUEST, DB and IES plug-in
Geometrical modelling should not only cater for
were also commonly used among the architects
the whole design team but also cover aspects of
samples. On the other hand, most engineers
buildings performance (Mahdavi 1998).
have used EP and eQUEST. DB, DOE-2 and IES
plug-in came in the second category of usage. 4.5 How many tools do you use when performing
This question was not aiming to rank the tools. simulations for a project?
The aim was to get a snap shot of the current use
of tools by architects and engineers. Notably, As shown in figure 09, the number of tools used
eQUEST, DB and IES plug-in were used by both per project varies widely. The majority of
groups. This match does not necessary architects use one tool per project (49% of first
demonstrate a preference of usage by each survey and 45% of the second survey). However,
group it rather represents a potential for a large proportion (38% and 43%) uses two tools.
developing tools that suits and incorporates Conversely the majority of engineers, 38% and
architects and engineers. 36% use at least two BPS tools per a project. In
the first survey 29% confirmed the usage of only
one tool and 10% confirmed the usage of three
tools. In the second survey 30% confirmed the
usage of three tools followed by 25% confirming
the usage of only one tool per project.
There is merit among architects to use one
simulation tools. Perhaps this due to that most
architects use BPS tools less frequently and only
Figure 07: Used tools by architects and engineers during early design stages. On the other hands,
building services engineers use tools during
4.4 What CAD/3D modelling software do you different design phases, are more acquainted
use? with BPS tools and rely more on tools for thermal
energy calculation, systems sizing and energy
Due to the advent of BIM and the frequent compliance issues. In fact, simulation tools for
announcements of direct links between BIM building services engineers are almost a
drawing tools and BPS tools, engineers and mandatory instrument assumed to be a ‘sine-qua-
architects have to identify the drawing they use non’ in the engineering practice (De Wilde and
frequently for geometric modelling. The objective Prickett 2009).
of this question was to trace the mismatch
between geometric and thermal modelling.
The majority of architects (36% and 40%), in
TM
both surveys, were using Google SketchUp for
geometric modelling. The second most used
software was AutoCAD with an agreement of
31% and 35% of the respondents of both
surveys. Revit came in the third place being used
by 21% and 15% of respondents of both surveys.
Figure 09: Number of tools used per project
ArchiCAD (10% and 7%) came at the last place.
On the other hand, the majority (48% and 47%) 4.6 What is your primary building type you
of engineers were using AutoCAD followed by model?
Revit (27% and 30%). In fact, the existing
categories of this question did not offer enough The majority of architects are running
choices for engineers. Under the “Other” option building simulations for residential buildings
Revit MEP, DDS-CAD and Bentley Microstation followed by office buildings and educational
products were numerously listed.

14
buildings. For engineers, the most common important parameter as shown in Figure 12. For
building type is office building followed by architects, comfort, shading, passive solar
educational buildings and retail buildings. heating, orientation and natural ventilation filled
Residential buildings came in the last place. the rank from 2 to 6. The three least important
Under the others option engineers listed other parameters were efficient lighting, building
types namely, medical, laboratory and religious tightness and controls. On the other hand,
buildings. engineers ranked HVAC systems, controls,
The results in figure 10 indicate that most comfort, lazing and openings in the top five after
architects’ simulation experience is with the energy consumption. The three least
residential buildings while engineers are larger important parameters were natural ventilation,
buildings. In fact, this is a true reflection to what daylighting and photovoltaic.
happens in practice. Most residential buildings A common observation, that comfort was
are designed by architects who have no budget considered by both groups in the top of both lists.
or time to hire a consultant and therefore rely on But surprisingly the largest difference was
in-house experience. On the other side, large recorded for ranking Controls and HVAC
scale projects such as office, retail and systems. Engineers ranked them in the top of the
educational buildings require simulation experts list and architects suited them at the bottom
HVAC engineers and receive the attention and despite prioritizing the energy consumption
resources to be major player of the design team. parameter. This question indicates a huge gap
between both users’ preferences. Perhaps the
separate building design practice among
engineers and architects is the reason (Deru and
Torcellini 2004). Architects are concerned with
building design issues such as geometry,
orientation, natural ventilation and daylighting
while engineers are concerned with mechanical
systems and controls.

Figure 10: Different simulated building types by

architects and engineers

4.7 For which design phase would you use the

following programs?

In a follow up question, respondents were

asked to justify the design phases for every tool
they use. Figure 11 indicates the typical usage
phases for the ten tools. There was no difference
between architects and engineers classification
and therefore all responses are binned in figure
11. HEED, Energy 10 and GBS were considered
as early design tools that might be used during Figure 12: Ranking the importance of output
the pre-schematic design phases followed by parameters
ECOTECT and eQUEST which are classified to
be used during the schematic design phase. DB, Part I - USABILITY and GRAPHICAL
EP SU, EP and DOE-2 were considered as VISUALIZATION of the interface
extensive tools that are used for detailed analysis
during design development and design 4.9 Indicate how important you think each of
optimization phases. the following objectives is, concerning usability
and graphical visualization of the interface.

Ranking the sub criteria was identical for both

disciplines. As shown in Figure 13, 23% of
architects and 26% engineers agreed that the
graphical representation of the output results is
Figure 11: Ranking the tools according to design the most important feature concerning the
phases usability and graphical visualization of the
interface. Also there was consensus that the
4.8 What are the parameters you focus on the flexible use and navigation (17% architects and
most, when performing simulation? 22% engineers) is the second most important
feature followed by the graphical representation
This question reveals another contradiction of the results in 3D spatial analysis (16%
between architects and engineers priorities. Both architects and 17% engineers). Surprisingly, both
groups were asked to classify and rank 15 design groups agreed that the easy learnability and short
parameters. There was an agreement from both learning curve of simulation tools is the least
groups that the energy consumption is the most important feature. The result of this question

15
indicates the importance and urgency of question. Therefore, both groups’ choices are
representing the simulation results graphically in different due to the different design phase they
a way that can be clearly and easily interpreted. work on and the different type of knowledge they
However, it is important to point to the risk of require and process.
being seduced by the graphic output that impede
the critical examination of outputs results (Donn,
2001 ).

Figure 14: Ranking criteria concerning information

management of BPS interfaces

4.11 . What other features should be

Figure 13: Ranking criteria concerning usability improved in the future development of Building
and graphical visualization of BPS interfaces Energy Simulation Tools concerning friendliness,
usability and information management of the
4.10 . Indicate how important you feel each of Interface? (optional)
the following objectives is, concerning information
management of the interface. The last question for this part was an open
ended question aiming to give participant the
Figure 14 compares the different preferences of opportunity to share or clarify their opinions. The
both groups. Architects first priority (28%), respondents reported a range of comments that
concerning the information management of the were classified for each group as follows.
interface, is the ability to create comparative
reports for multiple alternatives. Quality control of Architects’ comments include:
simulation input comes in the second place with • Regarding skill level of user
approximately 24% of the architects vote.
• Allowing debugging and need wizard like
However, it is very difficult here to draw
assists in data entry: ready examples, etc.
conclusions because the last three features came
• Defaults templates, but also front-and-center
very close comes in the number of votes
delineation and ability to create/modify those
representing 18% for allowing assumptions and
templates
default values to facilitate data entry, 16% for
• Error-checking to ensure models are correct
flexible data storage and user customizable
feature and 14% for flexible data storage and • Mismatch between the common form of
user customizable feature. On the other hand, input/output in most tools and the architects
engineers’ first priority (39%) was the quality expectations
control of simulation input. Flexible data storage • User friendly HVAC templates
and user customizable features came on the • 3D visualization of design strategies, for e.g.
second place with 20% of the engineers votes. daylighting
The creation of comparative reports for multiple • Graphical representation of design parameters
alternatives collected 18% of the votes and 16% (use the language of architects)
for the ability to allow assumptions and default • Easy searchable building inputs database
values to facilitate data entry. • Balance between extensive (deep) and quickly
This question revealed an important finding. (basic) guaranteeing meaningful input data
Engineers clearly identified the quality control of • Modify the floor plan after it is initially drawn,
simulation input as the most important feature ability to add/remove building features with
concerning information management of the ease, ability to make custom reports, ability to
interface. This is not surprising because the issue easily navigate all components with ease.
of attaining quality insurance of simulation input is
repeatedly highlighted in literature (Donn 2001; Engineers’ comments include:
Augenbroe 2002; Pilgrim, Bouchlaghem et al. • Provide echo-print of input in a readable format
2003; Hensen and Radošević 2004; Ibarra and • Mapping data entry trees and limiting access to
Reinhart 2009; Tian, Love et al. 2009). However, relevant paths to objectives
architects prioritized the ability to create • Have a huge amount of customizability in terms
comparative reports for multiple alternatives of output. It would be great to be able to select
above the input quality control. This means that output format, delimiters, etc, so that one could
the issue of assigning meaning and accurate optimize output for input into any visualization
input data is not a priority. An explanation to that package one wished with scripting capabilities
might be that architects are more involved with • Transparent default options, no more black box,
BPS tools during early design tools and use the more background information through links
tools for decision making and design • The number of available input parameters in
optimization. As mentioned by Donn {1987 #192}, many building models is so great that it is
precision is not so important to architects if all
they are looking for is an answer to a ‘what-if’

16
 almost certain that errors (some minor, some criteria support the same thing, which is decision
major) will exist. making, however, architects chose the general
• Convert SI to IP units and vice versa terms and engineers chose a specific issue.
• Supporting database management Perhaps most architects did not know what does
sensitivity and uncertainly mean or perhaps
Part II - Integration of KNOWLEDGE-BASE engineers found the word ‘support the decision
4.12 . Indicate how important you think each making’ too vague.
of the following objectives is, concerning Another remark is related to the last sub
Integration of Knowledge-Base criteria. The number of architects who chose the
ability to embrace overall design during most
Figure 15 shows an agreement among design stages is three times the number of
architects and engineers. Both groups identified engineers. Despite that the total number of
the ability to provide guidelines for buildings architects is 249 versus 232 engineers’
codes and rating systems compliance as the respondents, the comparison is still valid and a
most important feature in BPS tools. The ability to clear difference can be identified. This specific
provide case studies database for decision sub criterion is highlighting a very important issue
making came in second place. The result is not that might be veiled behind the limitation of the
surprising and there is a common ground question type and method. Architects are seeking
between both disciplines concerning the tools that embrace the overall design during early
integration of knowledge-base. and late design phases (Attia, Beltran et al.
2009).

Figure 15: Ranking criteria concerning the integration of

Knowledge-base systems in BPS tools
Figure 16: Ranking criteria concerning knowledge-base
4.13 . Indicate how important you feel each of systems and design process
the following objectives is, concerning Intelligent
Knowledge-base and Design Process 4.14 . What other features should be
improved in the future development of Building
As shown in Figure 16, architects top priority, Energy Simulation Tools concerning Integration
concerning the integration of the intelligent of Intelligent Design Knowledge-Base?
knowledge-base and compatibility with design
process, was the ability to provide quick energy The last question for this part was an open
analysis that supported their decision making ended question aiming to give participant the
(33%). The next priority was the ability to opportunity to share or clarify their opinions. The
examine sensitivity and uncertainty of key design respondents reported a range of comments that
parameters (29%) followed by the ability to were classified for each group as follows.
analyze weather characteristics and suggest
suitable climatic design strategies (20%). The Architects’ comments include:
fourth and last criterion was the overall • Scenario/Alternatives based design approach
embracement of design during most design • Define the most influential design parameters in
stages. However, engineers had a different order early design phases and their sensitivity range
of priorities. The most important feature was the • Assisting decision making process through
examination of sensitive analysis and uncertainty guidance
of key design parameters receiving 55% of the • Cases of low-cost, ultra energy efficient
votes. The ability to provide quick energy analysis buildings and LEED buildings
to support decision making came in second place • Guidance and rules of thumb on passive design
with 23 % of the votes. The last two sub criteria • Passive system – simulation , Simulation of
had the same ranking like architects. traditional passive design strategies and free-
By observing Figure 16 we can find a running buildings
contradictory finding. Architects are favouring the • Guide users into sound building science
ability of to support the decision making while designs the way the Passive House Standard
engineers are favouring the ability to examine the training does.
sensitivity and uncertainly of the design • Assist decision on materials to be used in the
parameters. Despite that both questions seems design
to be different but a closer look to the wording • Conform to codes and rating systems
and semantics reveals that they convey the same • Large design components libraries e.g. double
message. Architects chose the words ‘support façade, green roof
decision making’ and ‘quick analysis’ while • Contextual material property database
engineers chose the word ‘sensitivity and
uncertainly’ and ‘design parameters’. In fact, both

17
• Inclusion of various rates of ventilation based decision. The third most important criterion (21%)
upon latest ASHRAE or IMC standard, and was the ability to calibrate uncertainties. The
ability to compare differences in ventilation ability of BPS tools to create real sustainable
based upon the different codes. Also the ability results (10%) and the high resolution of
to utilize ASHRAE's intermittent occupancy simulation model (9%) were the least important
calculation or air quality calculation based upon criteria.
intermittent occupancy and advanced filtration, Engineers’ answers are not surprising because
respectively. they are in line with many publications and
• Explain what the tool is doing surveys that stress on accuracy, validation and
• Comprehensive HELP menu calibration. However, the architect’s responses
• Developer need to understand the design indicate a problem with confidence in simulation
process through the eyes of non engineers. results. Architects are seeking assurance that
the building model they have simulated with a
Engineers’ comments include: BPS tool represents the real building {Donn, 2001
• Diagnostics to assist with debugging #128}. There is also a lack of knowledge about
benchmarking for comparison of results (e.g. the accuracy requirements. Perhaps it is a
EPA databases) problem of language and nuances. The words
• Default or built in performance comparisons, ‘calibration’, ‘validation’ and ‘model resolution’ are
benchmarking or ratings such as Energy Star or not common words and many architects have
LEED been neither exposed to these realities.
• Multi-objective design optimization
• Assistance on control settings (e.g. air flow and
set point temperatures)
• System recommendations arrived at through an
algorithm of climate and building usage
• Interface with manufacturers' information - e.g.
standard formats for MEP equipment, windows,
etc that can be imported directly. Agencies such
as ORNL for instance could output test results
on materials and assemblies in this format.
Companies could provide product information in Figure 17: Ranking criteria concerning tools accuracy
this format. This would simply make it easier to
incorporate reliable and effective data into 4.16 . Indicate how important you think each
simulation modelling in a similar manner to how of the following objectives is, concerning tools
manufacturers offer DXF plans, models, and ability to simulate complex & specific building
details of products and components. components
• Introducing optimization models to identify
optimal design considering performance and Figure 18 shows another contradictory priority
cost for each group. Almost one third (31%) of
architects chose the ability to allow complex
Part III - ACCURACY of the tools design strategies and elements, as the most
4.15 . Indicate how important you think each important feature of BPS tools. The ability to
of the following objectives is, concerning tools simulate renewable energy system came in
ACCURACY. second place (27%). In the third and fourth place,
selection was made for the ability to evaluate the
Figure 17 shows contradictory priorities for emissions associated with the energy
each group. Architects first preference (40%) is consumption (17%) followed by the ability to
the confidence to create real sustainable design. support various types of HVAC systems (13%).
This choice is in line with Holms and Donn’s The ability to perform cost and LCC analysis (8%)
study in which they confirm that many architects and allow different building types (4%) came in
performing building simulations doubt the liability last place.
of simulation based designs to create real On the other hand, engineers selected the
sustainable designs {Holm, 1993 #132; Donn, ability to support various HVAC systems in the
2001 #128}. The second priority of architects first place (33%). Next, engineers favoured the
(28%) is the ability to provide accurate and reality feature of allowing cost analysis and LCC
like results followed by (18%) the ability to analysis (24%). The ability to simulate complex
provide validated performance measures. The design strategies and construction elements
ability to calibrate the uncertainty (8%) and the collected 22% of the votes. Fewer votes (11%)
high resolution of simulation model (6%) were the went to the ability to simulate renewable energy
least important criteria. systems and the ability to allow emissions
On the other hand, most engineers (31%) associated with the energy consumption (8%).
agreed that accurate and reality like results is the This question highlights the contradiction
most important feature concerning tools between architects and engineers and is in line
accuracy. The second most important sub with the results of question 4.8. Figure 18 reflects
criterion (29%) is the ability to provide validated a gap. Most architects are concerned and looking
performance measures to support design for tools to apply passive design strategies and

18
technologies such as double-skin facades, green • Adapt to the complexities of the real life designs
roofs, heat recovery, thermal storage, atria, and climatic conditions
concrete core conditioning etc..., including • Model thermal mass, air-to-air heat exchangers,
renewable and HVAC types. On the other hand, passive and active solar gains, or the most
engineers are concerned mainly with HVAC efficient lighting and passive drying options,
systems, controls and LCC, issues that architects radiant slabs/beams, ground source HX, heat
have been neither exposed to (Holm 1993). Both recovery chillers etc,
groups identified and showed different interest. • Better analysis for double skin facades, VRV
Perhaps this is due to the different design stages systems, dedicated outdoor air systems, and
each group is concerned with. Architects are natural ventilation
favouring criteria that feed the process of • Indication of the degree of error that could be
energetic concept initiation while engineers are expected in the results
favouring criteria that feed the process of • Error estimate of models for validation and
energetic building optimization. acceptable error range
• Validation and Verification of the simulation
output
• Gather data after implementation and get the
performance data back to into the software
• Be built on an underlying database to aid in
benchmarking
• Perform trade-off analysis and an LCA tool to
compare different options
• Ability to model complex HVAC and lighting
control strategies
• Wider range of HVAC and natural ventilation
Figure 18: Criteria concerning the ability to simulate modelling techniques
complex and specific building components. • Simulate monitor daylighting, displacement
ventilation and chilled beam systems
4.17 . What other features should be • Test cases representing building in reality
improved in the future development of Building • Robustness of models. Features should not be
Energy Simulation Tools concerning the ability to added until they are well-tested features and
simulate complex & specific building well-considered
components? (optional) • Allow more than one system per zone
• Describe uncertainty with the data model
The last question for this part was an open
• Clarity on the algorithms used to perform the
ended question aiming to give participant the
simulations and the limitations of those
opportunity to share or clarify their opinions. The
algorithms
respondents reported a range of comments that
were classified for each group as follows.
Part IV - INTEROPERABILITY of Building
Architects’ comments include: Model
4.18 . Indicate how important you think each
• Renewable energy systems calculators should
of the following objectives is, concerning
be a part of the package and tied into the
interoperability of the building model
overall project's energy performance.
• BESTS" until they can do all this stuff and
Figure 19 shows the major difference between
certify Passive Houses
architects and engineers needs and priorities. In
• Passive strategies such as green roofs and
the first place (39%), architects chose the ability
natural ventilation to exchange models with 3D drawing packages
• Embodied energy calculation such as SketchUp and 3DS Max. The second
• Ability to easily simulate essential elements (i.e. choice was for the exchange of models with CAD
fins overhangs) in sufficient detail programs (25%). The exchange of model for
• Building envelope design optimization multiple simulation domains and the exchange of
• Consider natural ventilation to combine with model with MEP drawing packages came in the
HVAC system design last place with almost no difference (18.3% and
• Integration of daylighting & daylight energy with 17.8%) in preference.
other tools like EP On the other hand, engineers prioritized
• Inform users as to the cost impacts of energy different sub criteria. The most important sub
reduction measures criterion was the ability to exchange model with
MEP drawing packages such as Revit and
Engineers’ comments include: Bentley products (45%). In second place (35%),
• Real-time results, parametric feedback. came the ability to exchange models for multiple
• Collecting realistic data from cases to establish simulation domains. In the third place (18%),
performance based data sets engineers voted for the ability to exchange
• Optimized for small, ultra-efficient buildings models with CAD programs. Not surprising, the
• Data to measure uncertainty last feature was the ability to model with 3D
drawing packages with less than 2% of the votes.

19
 • library of building components and building
assemblies in a common format or formats
(GBXML, IDF)
• Components that include data that describe
how they behave

Part V - Most important features of a

simulation tool

4.20 . What are the most important features

of a simulation tool?

Figure 19: Interoperability sub criteria This is one of the most important questions of
the survey. The question was repeated in both
4.19 . What other features should be surveys aiming to benchmark and rank the
improved in the future development of Building importance of major selection criteria for BPS
Energy Simulation Tools concerning the tools. The question was designed on purpose
interoperability of the building modelling? and positioned at the end of the survey to
guarantee that respondents understand the
The last question for this part was an open meaning of the four compared criteria with
ended question aiming to give participant the minimum confusion. After compiling the answer
opportunity to share or clarify their opinions. The of both samples in one graph, as shown in Figure
respondents reported a range of comments that 20, we can observe a strong cross disciplinary
were classified for each group as follows. difference. Architects in both surveys agreed on
their priorities and ranking of the major criteria.
Architects’ comments include: For architects, the most important criteria (31%
• Allowing organic modelling of curved volumes and 34%) was the ability of the tool to integrate
and non-cubical zones and volumes intelligent design knowledge-base to assist
• Ability to easily model essential elements (i.e. designer in decision making. This was
fins overhangs) in sufficient detail surprisingly, more important (28% and 30%) than
• Ability to directly import .dwg or Revit files. the friendliness of interface concerning usability
• Allowing input from multiple modelling programs and information management. In the third place,
(sketch up, rhino, 3dmax, Revit, etc) easily and selection was made for the IBM. Finally, AASDC
with minimal error. There needs to be clear came in last place (18%). These results reveal a
guidance for how to build models in each of very interesting finding. Respondents prioritize
these interfaces in order to facilitate the use of the IIKB over the UIM of the interface and even
these models quickly and easily. the AASDC. We believe that architects work
• Developing complex geometries more during early design phases and therefore
• Ability to merge architectural CAD drawings into need guidance to answer ‘what if’ scenarios that
respective thermal zones can assist design optimization process. More
• Change building geometry without having to re- importantly, architects lack the knowledge of
enter all data variables from scratch building sciences and building behaviour and
• Importing of detailed geometries with more therefore require constant information and
accuracy and all layers being correctly imported educational knowledge that guide them into
in energy simulation software building science (Attia, Beltran et al. 2009). In this
context, accuracy of simulation results is not of
• Proper translation of the geometry in complex
paramount importance to architects as
models
understanding the relative effect on performance
due to changes in design alternatives. This
Engineers’ comments include:
finding also suggest that the accuracy of the
• One common language like gbXML (but more
simulation model should be adaptive and
robust) to become an open standard, third party
adjustable to the user type and design phase to
organizations need to create a standard
correspond to the different needs of architects as
language.
well as engineers.
• 3D parametric modelling On the other hand, engineers had a different
• Full IFC compliance: Import / Export equally ranking. There was an agreement among both
robust, all elements that can be modelled must engineers’ samples. Engineers ranked the
be able to be exported / imported in IFC with all accuracy of tools and ability to simulate complex
relevant data (at a minimum name, type, size, elements in the first place (42% and 42%). The
material) - this includes MEP as well as second most important criterion (25% and 24%)
Architecture & Structure was the friendliness of interface concerning
• exchange of model needs to be more seamless usability and information management followed
and less frustrating which would greatly by (22% and 24%) the ability of the tool to
facilitate the iterative process of optimizing the integrate intelligent design knowledge-base to
design assist designer in decision making with a very
small difference. The interoperability of building
model came in last place with 11% and 9% in

20
both surveys. In fact, engineers ranking was not comply with the most important architects
surprising compared to what we found in selection criteria. One reason those three tools
literature. However, it was quite interested to find came in the last place not only because they do
that the UIM and IIKB received almost the same not have a friendly GUI but because they request
level of importance. There is no doubt that extensive textual input data.
engineers too require adaptive and friendly
interfaces and are looking for tools that can assist
the decision taking whether for code compliance
or optimization issues.

Figure 20: Ranking the most important features of a

simulation tool

Part VI - Evaluating and ranking ten tools

Architects versus Engineers (primary results)
Besides analyzing the selection criteria that
influence the performance of any tool the study
aimed to compare and evaluate ten existing tools. Figure 21: Ranking the ten tools according to architects
The ranking of the tools is a result of a and engineers
compilation of answers of 7 questions (4.9, 4.10,
4.12, 4.13, 4.15, 4.16 & 4.18). Each question was Engineers ranked the ten tools differently, as
followed by a follow up question asking the shown is Figure 21. DB (85%) and EP (82%)
participants to rank the tools according to the came in the first category. DB and EP are the
question’s sub criteria. Results were classified most accurate state of the art BPS tools that
and binned into two groups. Needless to say, we provide detailed and complex simulation
believe that it is difficult to compare and evaluate capabilities. The strength of DB is that is provides
tools in absolute and valid ways, because each a strong GUI that bridge the difficulties of using
tool keeps improving and has its advantages and the ‘raw’ and text input based EP. However, EP is
limitations. However, the comparison allows considered as the next generation BPS tools. The
identify tendencies. strength of EP lays in its transparency and
Among architects there is a consensus to rank various simulation capabilities including modular
IES VE plug-in (85%), ECOTECT (82%) and DB systems simulation and heat balance-based zone
(72%) on top as shown in figure 21. The three simulation. The tool also allows data exchange
tools provide friendly GUI with various templates and facilitate third party interface development.
allowing an easy use and simple information Therefore DB, which has a user friendly GUI and
management, which are deemed important by support a template driven approach, came on
architects. Also architects believe that these tools top. The second category included eQUEST
have good geometric modelling and data (72%), DOE-2 (70%), IES plug-in (68%) and OS
exchange features. However, it was noted that plug-in (65%). The relation of eQUEST and DOE-
guidance and assistance during decision making 2 is similar to the relation between DB and EP;
is not provided. In the second category, comes however, by announcing EP as the successor,
GBS (62%), E10 (58%), eQUEST (52%) and new users will avoid DOE-2 and eQUEST.
HEED (50%) with less agreement among Surprisingly, IES plug-in did not receive many
respondents. Those tools have less developed votes. Perhaps because the plug-in was
GUI and do not easily exchange data and addressing architects and operates from Sketch-
building models. Despite that HEED and E10 in Up environment. However, the main critique of
particular have an IIKB for decision taking there is engineers for IES is that it does not allow access
a difficulty to integrate the tools with the to source code and documentation. The reason
architectural design process. Except eQUEST, for having OS plug-in in this category was based
which has a poor GUI, the three tools can hardly on its dependence on EP. However, the plug-in in
be used for later design phases or by a itself was not successful and was neither well
multidisciplinary design team. Therefore, perceived by engineers nor architects. In the last
respondents have less trust in those tools. category came GBS (40%), E10 (25%),
Ranking SU OS plug-in, EP and DOE-2 at end of ECOTECT (20%) and HEED (4%). According to
list was not surprising when we refer to architects engineers top BPS tools selection criteria, which
answer of question 4.20. Even that those tools are ranked in Figure 20, the four tools does
representing state of the arts tools, they did not correspond to engineers’ needs and priorities.

21
 In brief the survey findings indicate the • Provide adequate help either at the beginning
significant difference between architects and of the tool or where ever necessary while
engineers in ranking existing BPS tools. The performing simulation/calculation use
preferred tools by engineers are considered as
complex by architects. The reason behind that 5. Discussion:
might be that architects and engineers work This research reviews the current situation of
during different design process and tackle BPS tools among architect and engineers in the
different types of problems and process different US considering the use of simulation in the
type of information. As a result of this comparison building design and design process. The ultimate
of tools ranking we observed a significant finding. objective of this research was to define generic
Despite that architects criticize most tools used tools selection criteria for software developers
by engineers, describing them as cumbersome, and compare the requirements of architects’ vis-
tedious and not user friendly, architects probably à-vis engineers under five classified criteria.
trust those tools used by engineers the most. These five criteria were tested through two online
Therefore, future tools need to develop genuine surveys. The above surveys results confirmed
team simulation toolkit that share the same that there is a large gap between architects and
simulation engine but use different and adaptive engineers and that the classification, of five
interfaces to be used by different users and at selection criteria, inherent apparent limitations.
different design phases requiring different level of Concerning the survey results we should
expertise. Those tools must comply with the five remind ourselves that the survey was dedicated
criteria presented in this paper and above all they toward beginner tools users and that respondent
should educate as well as inform the user. In the samples are not representative because they are
mean time, we see that DB is almost the only tool very small. However, surprisingly to the authors,
that was appreciated by architects and engineers the open end question produced the most
and was ranked in the top. By studying the valuable information. Based on a questionnaire
existing qualities of DB and working on improving and in-depth literature review we can summarize
its capabilities we can reduce the differences and the research findings under two main subjects.
widen the penetration of BPS tools in the AEC The first subject, discussed in section 5.1, is the
industry. tools selection criteria for BPS tools, addressing
mainly software developers. The idea here is to
General Comments present the criteria that can be used to assess
The final survey screen invited participants to the BPS tools as a technology or hardware. The
comment on what should be done to increase the second subject discusses the human factor or the
integration of BPS tools in the design practice. users as the software. There is the gap between
architects and engineers as BPS users.
a. (Architects & Engineers) Integrated Building Therefore, in section 5.2, we try to analyze the
Design Process reason of this gap (academia, lack of code
• Should include building owner, building users, enforcement etc…). Finally, we will discuss
government regulatory and advisory agents, suggestions to overcome this gap in the future.
engineering, construction, facilities
management agents. 5.1 Tools Selection Criteria
• Toolkits for corresponding all design stages The literature review conducted in section 2, is
• Tools are not practical for the design process an example that contains pieces of all five
• Being able to work with the software at perspectives. These five criteria allowed us to
conceptual and DD level that outputs classify and group the user’s wishes and needs.
information that is really useful. The survey generated very comprehensive and
• Integrate different performance domains abundant wish lists as presented in section 4.
• The flexibility to provide basic information Tools developers should tap into those wish lists
during pre-design while more complex and understand the different perspective and
information in later design phases. needs of architects and engineers. Comparing
• Reliable tools address late design phases the ten tools might be viewed as insignificant due
• Integrated tools intended for early phase design to short expiry of any tools comparison study, but
decision making Automatic graphic output (plots relating the questions to real tools allowed
and graphs) of simulation results recording and identifying the BPS functions
• Suitability for the entire design process required by both groups, in order to present this
• Integration of various analysis features in a wish lists. The next section discusses the survey
single software results under the five criteria.

(Architects & Engineers) Tools and training cost, Usability and information management of the
learning curve and future development interface
• Proper training in building science On the level of usability and graphical
• Gentle learning curve representation, the findings of the questionnaire
suggest the users commonly need various and
• Cost of programs for students
customizable graphical representation of input
• Tutorials, help menu, courses
and output results including 3D visualization of
• Video guidance on how to use design strategies and analysis in addition to more
flexible use and navigation of the interface. It is

22
important that simulation result be visualized that over-rely on mechanical systems to achieve
within the 3D model environment. comfort and obscure the passive design
On the level of information strategies.
management, there are many capabilities and On the level of intelligence, the survey findings
needs that are not supported by simulation tools. highlight the importance of providing quick energy
Users need to compare multiple analyses of analysis that support the decision making in
alternatives, easily manage support-databases, addition to conducting quick parametric study and
and ensure quality control of input trough data examine sensitivity and uncertainty of key design
entry mapping and error-checking features. parameters in a simple way. The survey confirms
Beginner users are overwhelmed with complex that architects and engineers generally use
input parameters that require domain expertise different knowledge types. Architects require
with no guidance to assure a minimum assurance tools during early design phases that assist the
for the quality of simulation input. Also there is an decision on designing the building geometry and
emerging call for allowing debugging, transparent envelope in relation to its physical and climatic
and modifiable default templates. context, while engineers require tools that assist
The respondents put forward two missing the decision on design HVAC systems,
features that were not included in the occupancy energy management and control
questionnaire. Both users groups are dissatisfied settings.
with the current inflexibility of data input. Further Respondents suggest many other capabilities
work is required to provide adaptive GUI. An that were not included in the questionnaire but fall
adaptive interface will balance between extensive under the IIKB criteria. For example, the idea of
and basic data input in relation to the user type benchmarking and comparing results features.
and skill level. Tools should allow users to go Additionally, the inclusion of contextual KB for
back and forth moving from simple visual material properties and design components
interfaces t detailed models. Different users need libraries (e.g. double façade, green roof),
to be addressed with different interfaces and occupant behaviour, climatic design
graphics. Specialist will want an in depth characteristics and local codes and standards.
understanding, less experienced users will want Furthermore, innovative strategies for energy
get a quick evaluation. The GUI should be saving such as reflective roof, daylighting, free
adaptive and flexible to improve the usability, cooling, solar hot water heating, heat recovery,
allow simple and basic data entry choices for non and thermal storage can be evaluated before
specialist, and in the same time detailed and implementation {Hong, 2000 #37}. Also many
complex data entry choices for specialist. respondents suggested the introduction of
Also interfaces of most existing tools are optimization models that can identify optimal
designed in an input/output logic that does not design decisions regarding energy performance
correspond to the architects’ expectations. The and cost for architects and engineers. It is clear
idea here is not to support the textual based input that BPS tools of the future must help and inform
text with a graphical icon claiming that this will different users at different design stages to
make it more architects friendly. However, the optimize and identify optimum building design
design of user friendly GUI should correspond to strategies.
the parameters and decisions that the architect is
dealing with. Input and output format should be Accuracy and ability to simulate complex and
user oriented. It is recommended that detailed building components (AADCC)
researchers and developers focus on providing The third selection criteria investigated the
tools interfaces that use a language familiar to accuracy and ability to simulate complex and
architects and explicitly support different user’s detailed building components (AADCC). The
needs. findings of the questionnaire suggest the users
commonly need accurate and validated
Integration of intelligent design knowledge- performance measures and above all the
base (IIKB) confidence that BPS tools can create real
Under the second category, integration of sustainable buildings. The calibration of
intelligent design knowledge-base (IIKB), the uncertainty and fluidity of model resolution must
survey findings suggest the users commonly be supported by the tools. Survey respondents
need KB systems that advice with code, rating are looking forward for simulating the
and certification compliance. The increasing performance of specific design strategies and
complexity in the design and performance building components including complex HVAC
evaluation of buildings has resulted in the need systems. Sizing and estimating renewable
for the use BPS tools. Knowledge-based systems systems, CO2 emissions, energy cost analysis
can provide decision support systems and and LCCA is not commonly supported by
databases. Users repeatedly mentioned that tools simulation tools. Users often employ additional
should embed integrated consideration of calculation tools that require extra time and cost.
passive, ultra low-energy and LEED buildings. The survey proofs that architects define
The survey suggests that users are dissatisfied accuracy different than engineers. Architects who
with tools that do not embrace alternatives based design during early design phases want to have
approach. The performance evaluation process the answer to ‘what-if’ questions and compare
requires the comparison of multiple alternative different design alternatives thus they are looking
design schemes. Users are dissatisfied with tools for tendencies. Engineers on the other side, who

23
design in later design phases, are looking for high merging geometric model in thermal models with
model accuracy with validated and calibrated full zones representation. Most engineers use
performance results. more than one tool according to the survey
Analyzing the answers of the open question, we findings, which implicates managing and
found that users suggested the improvement of exchanging the design for every tool. This
models robustness. Further work is required to process is tedious and creates a barrier in
indicate the degree of error that could be practice. Moreover, IBM is only addressed during
expected in the results, the error estimate of late design phase for large scale and multi-
models for validation and acceptable error range, disciplinary team based projects.
describe uncertainty with the data model and the However, IBM should be also addressed for
algorithms used and the limitations of those small scale projects. IBM in BPS tools should
algorithms. Survey responses suggest the need correspond to the user type and design phase.
for higher model resolution/detail and better Respondents suggest one common language like
model assumptions that allows integrated sub gbXML (CAD) to become an open standard and
systems design. Most of these requirements are full IFC (BIM) compliance. Architects would like to
not new but according to users they are missing see fluent building modelling technologies that
in existing tools. Users suggest also gathering does not cause complexity and allow exporting
measured operating data after implementation back and forth simple geometrical models with
and getting the performance data back to into the little input during early design phases. Engineers
software. prefer seamless model exchange and facilitating
Future tools should adopt post-construction iterative process optimization. . It is clear that
monitoring and verification exercises. These geometric modelling in the future should be
would provide opportunities for the calibration of attuned with early design phases allowing the
models and serve to help understanding the concept development. The tools mechanism
design assumptions. Subsequently, building should encourage the design team approach and
simulation can supplement energy auditing to allow architects and engineers to input building
check the energy performance of the as built data within an integrated central building model.
building. Collecting realistic data from cases to Also, it should emphasize the use of 3D model
establish performance based data sets, aid in from the beginning of the process and the level of
benchmarking, measure uncertainty and geometric details could gradually increase.
generate real-time results is not commonly
supported by BPS tools. The survey identified a Integrated building design process (IBDP)
gap between predicted and real energy use. The integrated building design process, as one
Users suggest more adaption to the complexities of the five selection criteria, was not presented
of the real life designs and climatic conditions. explicitly in the questionnaire. However, the
Features should not be added to tools until they survey findings proved that this criterion is one of
are well-tested features and well-considered. the most important one. Users mentioned that
Another gap was identified spotting the existing tools are not practical for the design
widening discrepancy between tools capabilities process. The open questions provided a rich
and the new technologies. BPS does not meet source that addresses this issue. The two
users’ changing need to new building important findings of the survey concerning the
components and systems application. The survey integrative design process are (1) the integration
suggest that users are dissatisfied with the of BPS in different design phases and (2) the
current obstructions to simulate passive integration of various users in the design process.
technologies and solutions such as thermal Concerning (1) the integration of BPS during all
mass, air-to-air heat exchangers, passive and design phases, users reported that they
active solar gains, efficient lighting, passive commonly need fluid tools that could produce
drying options, double skin facades, VRV initial results from a rough building representation
systems, radiant slabs/beams, ground source during early design phases and in the same time
HX, heat recovery chillers, etc...Future work allow for detailing of building components during
should bridge this gap and oversee the need for later design phases. Users complained that most
more detailed simulation systems and existing BPS tools address late design phases. In
components assuring higher quality and higher this survey, only a few tools had a GUI, which
models resolution. could progressively reveal different levels of
pertinent information input, demands to assist in
Interoperability of building modelling (IBM) decision making at different design stages.
Under the fourth category, interoperability of Interfaces for engineers can look different to
building modelling, the survey findings suggest interfaces for architects. In order to integrate BPS
that users commonly need to exchange the tools in the design process, different user
geometric building model accurately with interfaces must communicate to different users
simulation tools. For architects the priority is for using their familiar language.
drawing and CAD packages while for engineers Users suggested to develop toolkits that
the priority is for MEP drawing packages. The correspond to all design stages allowing the
survey suggests that users are dissatisfied with flexibility to provide basic information during pre-
obscuring organic modeling of curved volumes design while more complex information in later
and detailed elements such as fins overhangs. design phases. Further work is required to better
Another frustrating obstacle is the difficulty of understand the approach of existing building

24
design and delivery process in order to extend stages. New tools have to be developed in close
the application of simulation during all design co-operation with the designers it is intended for.
phases. The building thermal model should The five selection criteria presented in this paper
evolve through the design process as the model are not new. However, it is clear from the survey
resolution becomes more highly specified. As the results that these criteria/objectives have still not
design progresses, the design team can modify been met, for existing tools, to the satisfaction of
the model to higher resolution options BPS user. There is therefore still considerable
incorporating more accurate computational scope for improvement in making BPS tools
algorithms in order to create higher resolution accessible to users.
performance details results.
Concerning (2) the integration of various users 5.2 The Gap
in the design process, users reported that tools Although tools, which are considered as the
should cater more for design teams. The findings ‘hardware’, may be improved based on the
of the questionnaire suggest that architects have criteria discussed previously, they will be wasted
more confident in tools used by engineers while if we ignore the human factor or the user, as the
engineers have more confidence in tools that ‘software’. The survey revealed that some of the
facilitate the multidisciplinary and can be shared barriers to the use of simulation lie outside the
by the design team. Tools that allow the realm of tool development. The common finding
integration and interdisciplinary work were ranked of the survey indicates a wide gap between
higher than tools that focus on individual and architects and engineers as tools users. Out of 9
mono-disciplinary work. The survey respondents questions, architects and engineers agreed only
suggested that BPS should also address building two times. By analyzing the survey results and in
owners, users, facility management agents in particular question 4.8 and 4.20 we discovered
order to include their feedback in the process. that a gap between both groups is evident.
Further work should support multi-disciplinary Architects and engineers had a different ranking
collaborative design as building projects. BPS for the four selection criteria and ten tools.
maybe started discipline oriented however it There are many reasons for this gap that can
should continue to be design oriented in the be traced very early. Since the industrial
future. The mono-disciplinary approach should be revolution with the great development in the field
replaced by a team driven approach. Architects of sciences and materials a clear division
and engineers should be less discipline oriented. between the two professions became more
We consider that BPS tools can play the role of obvious {Larsen, 2003 #193}. Both groups
the vehicle that will get the whole design team on developed within a mono-disciplinary
board. However, the realm of BPS tools environment and catered their services within a
development requires interdisciplinary research linear and fragmented building delivery process
that is based on design teams, design process, {Mahdavi, 1998 #104}. Architects were in charge
design integration. of architectural issues, whereas engineers were
Lastly, the classification of five tools selection concerned with technical issues. Consequently
criteria was composed to stack up against the formation of computational building
theories and practices of building performance performance modelling and simulation as a
simulation. The inherent limits to a synopsis of discipline, developed within the womb of
five BPS tools selection and evaluation criteria engineering, reflecting this mono-disciplinary
are apparent. Several pages are not enough to environment.
convey the nuances of various tools selection Today, we are facing a paradigm shift. With the
criteria. But let us ask a question. Do the five growing trend towards environmental protection
criteria respond to common themes discussed in and achieving sustainable development, the
practice? This is a question of action and further design of green buildings using BPS tools is
research. In fact, each of these criteria has its gaining attention. Simulation tools became a
critiques. Nonetheless, the five criteria presented significant part of the building design {Donn, 2001
in this paper form a basis for tools selection #128}. Together with sustainability BPS tools
criteria. As with many classifications they overlap became a part of the larger trend toward
and are not mutually exclusive. integration in the AEC industry. We are on the
With the sprawl and diversification of BPS tools verge of a major revolution, triggered by
in addition to the increasing growth of number of mandatory codes and standards that will change
users, it will be helpful to find entities (bodies, the way building are designed and constructed.
private magazines, software companies, etc…) With the 2030 objective and international Net
that are only concerned with facilitating the Zero Energy Buildings objective {IEA, 2009 #135;
selection of tools and comparing them among Mazria, 2009 #200; ASHRAE, 2008 #210}, there
different users needs using the five selection is a great effort to work together in a focused
criteria. Despite that new tools must take into effort as a design team including the building
account the five criteria, tools satisfying these owner. Discipline oriented design approaches
criteria are not guaranteed to succeed. Tools can no longer achieve exceptional performance.
developers have to look into the future and think In fact, the typical uni-disciplinary design process
beyond the five selection criteria. To be truly where the architect and engineers work in
effective, tools have to be based on research and separate islands and with no performance goals
adapted to the experience and background of the cannot achieve the new millennium objectives.
different design team members at different design

25
 The solution is to change this approach and Technology (ABET) and the 119 architecture
develop the design goals together as a unified programs accredited by the National Architecture
design team from the beginning during the initial Accreditation Board (NAAB) will likely show the
stages of the design. Performance goals should same mono-disciplinary approach and lack of
be set from the beginning and every team interdisciplinary team work. A deeper review will
member has to be in the service of the objective, show that the word building performance
but where and how to start to bridge this gap? simulation is missing from most undergraduate
The answer to this question works at several course curricula and in particular architecture
levels. programs. In fact, the integration cannot take
place in a single course or during a single project
In Education that is taught only for architects or engineers. It
There is a fundamental need that the must be developed in a team environment over a
architectural and engineering education has to lifetime of education {Geschwindner, 1995 #196}.
evolve. Evolution has to take place in universities We believe that the next generation architects
introducing new classes and updating existing and engineers have to be trained to work in
ones. In most architecture schools, students are teams. Architecture and engineering student
taught within the design studio to start with a have to come together early as possible during
concept and then push their concept towards the the undergraduate educating to work together
design details (Schon 1985). During and design together in teams. In fact, there are
conceptualization, students jump from a concept many successful examples of extensive forms of
to another aiming to compare alternatives and collaboration in educational schools, such as the
optimize their designs (Lawson 1997). During ecoMOD project at the University of Virginia and
this phase the major design characteristic are the Solar Decathlon Entries across 17
determined. Architects often think of design as universities in the USA {Quale, 2005
being the heuristic conception and idealization #194}{Charles, 2009 #195}. Also architecture
process of vision, for the appearance and students should be introduced to scientific and
function of a building. Therefore, they are more technical foundation to the use of BPS tools
interested in simple, visual, straight and intuitive during their education to learn how to integrate
tools. On the other hands, student in most them in their own practice. {Pedrini, 2005 #144}.
engineering schools are taught to follow On the other hand, engineering students will
systematic and methodological and progressive need to study the architectural approach to
steps. Designing and sizing energy systems is design and find ways to integrate that approach
based on building a simulation model step by with a diverse set of engineering goals
step based on the accretion of detail (Holm {Geschwindner, 1995 #196}.
1993). The procedure moves from the basic parts Today, architecture and engineering education
towards the whole, thus in the opposite direction takes place in an increasingly computationally
of the architectural design approach. Moreover, it rich and diverse world. BPS tools are a part of
is true that the language difference between this environment and must be brought to the
engineers and architects is a barrier. Engineers students in the classroom and in the studio.
are more technically oriented and require verified Successful high performance buildings are a
and accurate models that effectively represent logical outcome of an integrated process,
the real world complexity. Thus, engineers are requiring a connection between designs and
typically educated to think of design as being a building performance. Therefore, BPS tools are
systematic methodological process for already a potential medium or vehicle that can
determining the appropriate energy systems. In bridge this gap.
fact, the educational system is nourishing the gap
between architects and engineers. In Practice
Moreover, not all architecture schools provide a In traditional practice the building owner and
good grounding in building physics and even if architect create the building program and the
provided in practice much of this knowledge is architect has to satisfy the requirements of the
quickly lost {Marsh, 2004 #145}. User surveys design brief. Then the building services engineers
indicate that architect lack simulation know-how design the mechanical and other systems. As a
(Mahdavi, Silvana et al. 2003). For example, an result, the decision on mechanical systems and
architect, not aware about building thermal controls is frequently decoupled from the design
characteristics, will find it difficult to specify the (Lam, Wong et al. 1999). Therefore, most BPS
thermodynamic properties of a building. However, tools are used during late phases of design
he can easily define the construction material because they were mainly catering to engineers.
used. In doing so, some of the thermal Thus they help engineers to design and refine the
characteristics are inherently specified {Marsh, basic systems design, not to help architects in
2004 #145}. Therefore, it is necessary that formulating the basic building ideas (Donn, 2001).
architecture student receive a sufficient As a consequence, most major decisions have
knowledge in environmental building design to been taken, by the time BPS are performed,
use the tools for quick evaluation of design making it impossible to go back {Holm, 1993
concepts. #132; Yezioro, 2008 #107}. Holm (1993) point out
A review of the overall curriculum of 17 that “By that time the building owner and/or the
architectural engineering programs accredited by architect may have fallen in love with the design
the Accreditation Board for Engineering and

26
or even be married to it, in which case the cost of solve those problems. Using BPS tools can help
divorce would exceed the cost of hanging on”. in bridging the gap between architects and
Moreover, in traditional practice most engineers. Recent development and application
architecture bureaus pass the simulation work to of information technology in the building industry
engineers and shift the energy issues away. This is changing completely the building design
is probably true in situations where the design philosophy and methodology. The experience of
team comprises a diverse group of specialist the National Renewable Energy Lab (NREL) with
working together right from the beginning. the High Performance Building Initiative (HPBi)
Passing the simulation work to engineers might highlighted the importance collaboration between
be due to time limitation since preparing and all designers as one team (Deru and Torcellini
interpreting data for thermal building simulation 2004). There is evidence in literature that the
tolls can take several days {Ellis, 2002 #138}. Or starting point for the whole team should be the
maybe due to the great expertise requirement for same {Hopfe, 2005 #51; Lam, 1999 #34;
understanding ,analyzing and interpreting Mahdavi, 1998 #104}. Transformative concepts
physical processes involved {Marsh, 2004 #145}. such as integrated delivery and the long-term
The lack of detailed knowledge in building involvement of architects and engineers in
performance might be the fundamental reason. building operations all have a significant role to
What is evident is that the confidence of most play in creating a more sustainable built
architects in practice quickly falls when it comes environment. Future tools need to develop
to thermal building performance analysis, incident genuine team simulation toolkit that share the
solar radiation and regulatory compliance {Marsh, same simulation engine but use different and
2004 #145}{Reinhart, 2006 #48}{Augenbroe, adaptive interfaces to be used by different users
1992 #99} {Donn, 2001 #128}. and at different design phases requiring different
Another inherent problem in the traditional level of expertise. One of the important findings of
practice is related to small projects that have this study is that architects and engineers are
limited budget. During the design of many acknowledging LEED, as a common ground.
medium and small scales, architects are forced to Thus there is a potential to use LEED as a
base their design on intuition. They are obliged to medium in education and practice to bridge the
generate a reasonably cohesive design solution gap.
without using BPS tools or getting general On the other hand, much more effort is needed
directional advice from engineering consultant to to get BPS tools into the architecture main stream
avoid paying significantly high consultancy fees. and to maximize the tools usage in the design
Also the prescriptive nature of many current process {Wong, 2000 #189}.. There are many
codes of practice and design guidelines facilitates qualities of BPS. BPS entails an embedded
this practice. On the other hand, not much tools feature that not many practitioners recognize.
are developed to satisfy the architects’ needs BPS highlights and reinforces the iterative nature
during early design phases. The results of the of design. BPS can capture the complex design
survey confirm that problem and perhaps explain interrelationships between building design and
possible reason behind this gap. Despite that building performance. BPS can bring the whole
future design trend will entail a multidisciplinary design team participants together. A better
design team approach; this approach will be understanding of energy simulation tools, their
limited to large projects with sufficient budgets. A advantages and their limitations, may encourage
huge part of the newly constructed building stock architects to have the confidence to use the
will be small residential units that are, in principal, simulation tools, {Yezioro, 2008 #107}. The
designed by architects only. Tools developers professional experience with leading architecture
have to reach those architects who are not firms urges brining BPS tools into the
energy experts. Instead of focusing on engineers undergraduate studio environment. Therefore,
needs only, developers have to cooperate with BPS offers a common ground, a platform to
architects to create adaptive tools. Adaptive tools support the collaboration between architects’ and
that address architects, who generally use engineers in practice and education.
different types of knowledge required by existing
tools, during early design. The fragmented
building delivery process has resulted in little 6. Conclusions:
progress in the augmentation of simulation tools The AEC disciplines are moving towards
that address architects during conceptual design. convergence. There is evidence that building
Therefore, we have to foster the development of services disciplines are merging (Attia, Beltran et
architects’ in-house simulation capability within al. 2009) Triggered by the mandatory codes and
design practices by developing tools that are rating systems environments the rapid emerging
centered on architects. As mentioned before, confluence of multidisciplinary integrated building
there is no doubt that architects themselves need design process and BPS will accelerate the use
a fundamental understanding of basic building and development of BPS tools within the
physics. However, it is not enough to leave the architectural and engineering practice and
energy issues up to the engineering consultant education.
who will never participate in such projects with a Today architecture and engineering practice
tight budget {Marsh, 2004 #145}. takes place in an increasingly rich BPS tools
Finally, we strongly believe that the building environment. The purpose of this article has been
regulation enforcement is a good entry point to to identify selection criteria for building

27
performance tools. The overview presented in Acknowledgements
this paper aims to introduce criteria for selecting The authors express their thanks to all
and evaluating building simulation and provide respondents who participated in the survey and
information sources to building simulation appreciate their valuable comments and
developers. The five criteria presented in this feedback. This paper is part of an ongoing PhD
paper, namely (1) usability and information research funded by the Université catholique de
management (UIM) of the interface, (2) Louvain.
integration of intelligent design knowledge-base
(IIKB), (3) accuracy and ability to simulate
complex and detailed building components 7. References
1. AIA (2007). Integrated Project Delivery: A Guide.
(AADCC), (4) interoperability of building 2. AIA. (2007). "Interoperability in the Construction Industry ", from
modelling (IBM) and the (5) integration with http://construction.ecnext.com/mcgraw_hill/includes/SMRI.pdf.
3. ASHRAE (2007). Standard method of test for the evaluation of
building design process (IBDP), continue to
building energy analysis computer programs. ASHRAE.
resound and form the basis of much scholarly 4. ASHRAE (2009). ASHRAE handbook. Fundamentals. Atlanta, Ga.,
and professional activity. The survey results American Society of Heating, Refrigerating, and Air-Conditioning
provide an overview of the criteria that need to be Engineers: v.
5. Attia, S., L. Beltran, et al. (2009). "Architect Friendly": A comparison
addressed by developers to improve the uptake of ten different building performance simulation tools. IBPSA, Galsgow,
of simulation practice. Addressing these criteria Scotland.
will require interdisciplinary research in the field 6. Augenbroe, G. (1992). "Integrated Building Performance Evaluation
in the Early Design Stages." Building and Environment 27(2): 149-161.
of building simulation research and development 7. Augenbroe, G. (2002). "Trends in building simulation." Building and
with design process management {De Wilde, Environment 37: 891 – 902.
2009 #120}. In order to improve the uptake of 8. Autodesk (2008). ECOTECT v5.60.
9. Autodesk (2008). Green Building Studio v3.3.
simulation practice, within an accelerating
10. Autodesk. (2009). "ECOTECT." Retrieved November, 2009, from
growing BPS tools environment, we believe that http:..ecotect.com/products/ecotect.
the BPS community has to set a uniform 11. Autodesk (2009). ECOTECT.
definition of tools selection criteria and 12. Autodesk. (2009). "Green Building Studio." Retrieved November,
2009, from https://www.greenbuildingstudio.com/.
specifications. On the other hand, developers 13. Balcomb, J. D. (1992). Passive solar buildings. Cambridge, Mass.,
might create metrics to analyze the costs and MIT Press.
benefits of using BPS tools. This will accelerate 14. Bambardekar, S. and U. Poerschke (2009). The architect as
performer of energy simulation in the early design stage. BS2009,
and improve the BPS practice. Glasgow.
While developers can use the survey results to 15. Bazjanac, V. (2003). Improving Building Energy Performance
improve their tools and create an innovative Simulation With Software Interoperability
16. Interoperability. IBPSA, Eindhoven.
bridge between architecture and engineering both 17. Bazjanac, V. (2004). "Building energy performance simulation as part
groups have to work together to bridge their of interoperable software environments." Building and Environment 39
interdisciplinary gap at several levels. But first, 879 – 883.
18. Bazjanac, V. and A. Kiviniemi (2007). Reduction, simplification,
architecture and engineering education should
translation and interpretation in the exchange of model data. Bringing ITC
enforce transversal team oriented education and knowledge to work, University of Maribor.
support students with necessary skills to use and 19. Clarke, J. (2009). Integrated Building Performance Simulation:
judge BPS tools results. Architects and engineers Trends and Requirements. Glasgow, University of Strathclyde.
20. Clarke, J., J. Hensen, et al. (1998). Integrated Building Simulation:
in practice must broaden their skills to ‘adjacent’ State-of-the-Art. Indoor climate of Buildings, Bratislava, ASHRAE
domains or learn to work with other experts to Transactions.
successfully support integrated design of high 21. Clarke, J. A. (1985). Energy simulation in building design. Bristol
[Avon] ; Boston, A. Hilger.
performance buildings. Clearly for mastering such 22. Crawley, D. (1997). BUILDING ENERGY TOOLS DIRECTORY.
skills, users’ need domain knowledge for quality IBPSA, Prague.
assurance. Improving the use of BPS tools will 23. Crawley, D., J. Hand, et al. (2005). Contrasting the capabilities of
building energy performance simulation programs. Washington DC.
lead to improved building performance. 24. Crawley, D. and L. Lawrie (1997). WHAT NEXT FOR BUILDING
Finally, we believe that the next generation of ENERGY SIMULATION—A GLIMPSE OF THE FUTURE. IBPSA, Prague.
BPS tools has to direct its development within the 25. Crawley, D. B., J. W. Hand, et al. (2005). "Contrasting the
gestalt of multidisciplinary design team and the capabilities of building energy performance simulation programs." Building
and Environment 43(4): 661-673.
gestalt of design process. Quantitative and 26. Crawley, D. B., J. W. Hand, et al. (2008). "Contrasting the
qualitative understanding of building energy capabilities of building energy performance simulation programs." Building
performance should be brought to architects and and Environment 43(4): 661-673.
27. De Wilde, P. and D. Prickett (2009). PRECONDITIONS FOR THE
engineers. Architects and engineers will design USE OF SIMULATION IN M&E ENGINEERING. BS2009, Glasgow.
buildings using BPS tools which are very 28. Deru, M. and P. Torcellini (2004). Improving Sustainability of
adaptive, accurate and can predict, during all Buildings through a performance-based design approach. World
Renewable Energy Congress. Denver, Colorado.
design stages, the energy consumption of high 29. DesignBuilder (2008). Design Builder v.1.8.1.001.
performance buildings. . BPS will be the heart of 30. DesignBuilder. (2009). "DesignBuilder " Retrieved November, 2009,
designing and building high performance from http://www.designbuilder.co.uk/.
31. DOE. (2009). "EnergyPlus." Retrieved November, 2009, from
buildings in order to inform the design process
http://apps1.eere.energy.gov/buildings/energyplus/.
and evaluate the impact of design decisions. We 32. DOE, U. S. (2009). "Building Energy Software Tools Directory."
believe that BPS offers a common ground, a Retrieved 15 November, 2009, from
platform to support the collaboration between http://apps1.eere.energy.gov/buildings/tools_directory/.
33. Donn, M. (2001). "Tools for quality control in simulation." Building
architects’ and engineers. and Environment 36: 673–680.
34. Donn, M., S. Selkowitz, et al. (2009). Simulation in the service of
design - asking the right questions. IBPSA Glasgow, Scotland.
35. Donn, M., S. Selkowitz, et al. (2009). SIMULATION IN THE
SERVICE OF DESIGN – ASKING THE RIGHT QUESTIONS. IBPSA.
Glasgow, Scotland: 1314-1321.

28
36. Eisenberg, D., R. Done, et al. (2002). Breaking down the barriers: 72. Pedrini, A. and S. Szokolay (2005). The architects approach to the
Challenges and solutions to code approval of green buildings. Tucson, AZ. project of energy efficient office building in warm climate and the
37. Ellis, M. and E. Mathews (2002). "Need s and trends building and importance of design methods. IBPSA, Montréal, Canada.
HVAC system design tools." Building and Environment 37: 461-470. 73. Pilgrim, M., N. Bouchlaghem, et al. (2003). "Towards the effecient
38. Ellis, P., P. Torcellini, et al. (2008). ENERGY DESIGN PLUGIN: AN use of simulation in building performance analysis: a user survey."
ENERGYPLUS PLUGIN FOR SKETCHUP. SimBuild 2008, Berkeley, Building Services Engineers 24(3): 149-162.
California. 74. Punjabi, S. and V. Miranda (2005). Development of an integrated
39. Gale (2001c). General Contractors—Non-residential Buildings, Other building design information interface. IBPSA, Montreal, Canada.
Than Industrial Buildings and Warehouses. Encyclopedia of American 75. Robinson, D. (1996). "Energy model usage in building design: A
Industries, 3rd ed., Farmington Hills. qualitative assessment." Building Services Engineering Research and
40. Google. (2009). "SketchUp Pro7." Retrieved November, 2009, from Technology 17(2): 89-95.
http://sketchup.google.com. 76. Schlueter, A. and F. Thesseling (2009). "Building information model
41. Hand, J. and D. Crawley (1997). Forget the tool when training new based energy/exergy performance assessment in early design stages."
simulation users. IBPSA, Prague, Czech Republic. Automation in Construction 18(2): 153-163.
42. Hayter, S., P. Torcellini, et al. (2001). The Energy Design Process for 77. Schon, D. (1985). The Design Studio. London, RIBA.
designing and constructing high-performance buildings. Clima 2000. 78. Tian, Z., J. Love, et al. (2009). "Applying quality control in building
Napoli. energy modelling: comparative simulation of a high performance building "
43. Hefley, W. and D. Murray (1993 ). Intelligent user interfaces. Journal of Building Performance Simulation 2(3): 163 - 178.
International Conference on Intelligent User Interfaces, Orlando, Florida, 79. Tianzhen, H., Z. Jinqian, et al. (1997). "IISABRE: An Integrated
United States. Building Simulation Environment " Building and Environment 32(3): 219-
44. Hensen, J. (2002). Integrated Building (and) airflow simulation: An 224.
overview. Computing in Civil and Building Engineering, Taipei, Taiwan. 80. Torcellini, P., S. Hayter, et al. (1999). "Low Energy Building Design -
45. Hensen, J. (2004 ). Towards more effective use of building The Process and a Case Study." ASHRAE 105(2): 802-810.
performance simulation in design. 7th International Conference on Design 81. UCLA (2008). HEED v.3.0.
& Decision Support Systems in Architecture and Urban Planning, 82. UCLA. (2009). "HEED." Retrieved November, 2009, from
Eindhoven: TU/e. http://mackintosh.aud.ucla.edu/heed/.
46. Hensen, J., R. Lamberts, et al. (2002). "A view of energy and building 83. US-DOE. (2009). "Building Energy Software Tools Directory."
performance simulation at the start of the third millennium." Energy and Retrieved 01 October, 2009, from
Buildings 34(9): 853-855 http://apps1.eere.energy.gov/buildings/tools_directory/alpha_list.cfm.
47. Hensen, J. and M. Radošević (2004). Some quality assurance issues 84. Warren, P. (2002). Bringing simulation to application, IEA ECBCS
and experiences in teaching building performance simulation. IBPSA Annex 30. Brimingham.
2004, Eindhoven. 85. Yezioro, A. (2008). "A knowledge based CAAD system for passive
48. Hien, W., L. Poh, et al. (2003). "Computer-Based Performance solar architecture." Renewable Energy 34: 769-779.
Simulation for Building Design and Evaluation: The Singapore
Perspective." Simulation & Gaming 34(3): 457-477.
49. Holm, D. (1993). "Building Thermal Analyses: What the
Industry.Needs: The Architect's Perspective." Building and Environment
28(4): 405~07.
50. Hopfe, C., C. Struck, et al. (2005). Exploration of using building
performance simulation tools for conceptual building design. IBPSA-NVL
Conference, Delft, The Netherlands, TU-Delft.
51. Ibarra, D. and C. Reinhart (2009). Daylight factor simulations - how
close do simulation beginners 'really' get? BS2009, Galsgow.
52. IEA (2009). IEA Net Zero Energy. Montreal.
53. Judkoff, R. and J. Neymark (1995). International Energy Agency
Building Energy Simulation Test (BESTEST) and Diagnostic method.
Golden, Colorado, NREL.
54. Kusuda, T. (1999). Early history and future prospects of buildings
system simulation. IBPSA, Kyoto Japan.
55. Lam, K., N. Wong, et al. (1999). A study of the use of performance-
based simulation tools for building design and evaluation in Singapore.
IBPSA, Kyoto, Japan.
56. Lawson, B. (1997). How designers think : the design process
demystified. Oxford, U.K. ; Boston, Mass., Architectural Press.
57. LBNL and J. Hirsch (2008). DOE v2.2 Building energy use and cost
analysis program. Berkeley, CA, LBNL.
58. LBNL and J. Hirsch. (2009). "DOE-2." Retrieved November, 2009,
from http://www.doe2.com/.
59. LBNL and J. Hirsch. (2009). "eQUEST." Retrieved November, 2009,
from http://www.doe2.com/eQUEST/.
60. LBNL and J. Hirsch (2009). eQUEST v3.61b. Berkeley, LBNL.
61. Mahdavi, A. (1998). "Computational decision support and the building
delivery process: A necessary dialogue." Automation in Construction 7:
205-211.
62. Mahdavi, A., F. Silvana, et al. (2003). An inquiry into building
performance simulation tools usage by architects in Austria. IBPSA,
Eindhoven.
63. Maybury, M. T. and W. Wahlster (1998). Readings in intelligent user
interfaces. San Francisco, Calif., Morgan Kaufmann Publishers.
64. Mendler, S., W. Odell, et al. (2006). The HOK guidebook to
sustainable design. Hoboken, N.J., John Wiley & Sons.
65. Morbitzer, C. (2003). Towards the Integration of Simulation into the
Building Design Process. Department of Mechanical Engineering.
Glasgow, University of Strathclyde. PhD.
66. Morbitzer, C., P. Strachan, et al. (2001). Integration of building
simulation into the design process of an architecture practice. IBPSA. Rio
de Janeiro, Brazil.
67. NREL (2005). ENERGY-10 v1.8.
68. NREL. (2009). "Energy-10." Retrieved November, 2009, from
http://www.nrel.gov/buildings/energy10.html.
69. NREL (2009). OpenStudio 1.0.2.
70. Papamichael, K., J. LaPorta, et al. (1996). The Building Design
Advisor. ACADIA, Arizona.
71. Papamichael, K. and J. Protzen (1993). The limits of Intelligence in
Design. 4th International Symposium on System Research, Informatics
and Cybernetics, Baden, Germany.

29
 APPENDIX I

Final Survey Results

© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

Setting criteria for the use of building performance simulations in building

design: A user survey

Survey 1: (mid Dec 2008‐ mid Feb 2009) Survey 2: (mid July ‐ mid October 2009)

1. How do you describe your current position?

2. Which of the following affiliations apply to you?

31
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

3. What of the following energy simulation tools do you use?

4. What CAD/3D modeling software do you use?

32
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

5. How many tools do you use when performing simulations for a project?

6/16. What is your primary building type you model?

7. For which design phase would you use the following programs?

Pre‐conceptual Schematic Design Design Design

Development Optimization (CD)
HEED 38 7 0 0
Energy 10 56 6 0 0
GBS 34 18 0 0
ECOTECT 86 107 13 0
eQUEST 56 110 67 9
DB 0 77 84 87
EP SU 0 4 23 27
EP 0 0 43 45
DOE‐2 0 0 15 19

33
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

8. What are the parameters you focus on THE MOST when performing simulation?

34
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

Part I ‐ USABILITY and GRAPHICAL VISUALIZATION of the interface

9. Indicate how important you think each of the following objectives is, concerning USABILITY and
GRAPHICAL VISUALIZATION of the interface

Usability & Graphical Visualization

Architect Engineer

57 60
51
43 39 40 38 35 38 34
27
19

Graphical Flexible use & Graphical Easy follow‐up Graphical Easy learnability
representation navigation representation structure representation and short
of output results of results in 3D of input data learning curve
spatial analysis period

10. Which tool(s) fulfill the following criteria?

35
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

11. Indicate how important you feel each of the following objectives is, concerning INFORMATION
MANAGEMENT of the interface

12. Which tool(s) fulfill the following criteria?

13. What other features should be improved in the future development of Building Energy Simulation
Tools concerning FRIENDLINESS, USABILITY and INFORMATION MANAGEMENT of the Interface?
(optional)

(1) a. (Architects) Usability and information management (UIM) of interface

• 3d visualization of certain sustainable strategies, for e.g. daylighting
• Easy searchable building inputs database
• Graphical output
• Ability to quickly and easily explore multiple design alternatives
• balance between extensive (deep) and quickly meaningful data
• Allowing debugging
• Ability to go back and change your input parameters
• defaults/templates available, but also front-and-center delineation and even creation/modification of those
templates is necessary

36
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

• user friendly HVAC templates and daylighting map

• ability to modify the floor plan after it is initially drawn, ability to add/remove building features with ease,
ability to make custom reports, ability to easily navigate all components with ease. (i.e. edit the schedule you
are assigning to an air handler from the air handler window, vs. having to exit the air handler window,
navigate to the schedules, identify which one you were working on, and then edit it) Need wizard like assists
in data entry: ready examples, etc.
• Daylighting analysis with visual output
• error-checking to ensure models are correct
• regarding skill level of user
• scenario based design approach
• define the most influential design parameters in early design phases and their sensitivity range

b. (Engineers) Usability and information management (UIM) of interface

• mapping data entry trees and limiting access to relevant paths to objectives
• simple interface
• Transparent default options
• Have a huge amount of customizability in terms of output. It would be great to be able to select output
format, delimiters, etc, so that one could optimize output for input into any visualization package one wished
with scripting capabilities 19no more black box, more background information through links
• The number of available input parameters in many building models is so great that it is almost certain that
errors (some minor, some major) will exist.
• convert SI to metric unit and vice versa
• templates for building type and use and occupancy
• provide echo-print of input in a readable format
• supporting database management

37
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

Part II ‐ Integration of KNOWLEDGE‐BASE

14. Indicate how important you think each of the following objectives is, concerning Integration of
Knowledge‐Base

Integration of Knowledge Base

Architect Engineer

98
88
69 62
54
44 37
30

Provide guidelines for Provide case studies Provide weather data and Support online user help &
building codes & rating database for decision extensive libraries of training courses
systems compliance (e.g. making building compenents &
ASHRAE 90.1, LEED, etc.) systems

15. Which tool(s) fulfill the following criteria?

38
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

16. Indicate how important you feel each of the following objectives is, concerning Intelligent
Knowledge‐base and Design Process

Intelligence of Knowledge Base and Design

Process Support

Architect Engineer
128

83
72
53 51
34 43
16

Provide quick energy Allow examining sensitivity Analyze weather Embrace overall design
analysis that supports the and uncertainity of key characterstic and suggest during most design stages
decision making design parameters suitable climatic design
strategies

17. Which tool(s) fulfill the following criteria?

39
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

18. What other features should be improved in the future development of Building Energy Simulation
Tools concerning Integration of Intelligent Design Knowledge‐Base?

(2) a. (Architects) Integration of intelligent design knowledge-base (IIKB)

• Scenario/Alternatives based design approach
• Define the most influential design parameters in early design phases and their sensitivity range
• Assisting decision making process through guidance
• Cases of low-cost, ultra energy efficient buildings and LEED buildings
• Guidance and rules of thumb on passive design
• Passive system – simulation , Simulation of traditional passive design strategies and free-running buildings
• Guide users into sound building science designs the way the Passive House Standard training does.
• Assist decision on materials to be used in the design
• Conform to codes and rating systems
• Large design components libraries e.g. double façade, green roof
• Contextual material property database
• Inclusion of various rates of ventilation based upon latest ASHRAE or IMC standard, and ability to compare
differences in ventilation based upon the different codes. Also the ability to utilize ASHRAE's intermittent
occupancy calculation or air quality calculation based upon intermittent occupancy and advanced filtration,
respectively.
• Explain what the tool is doing
• Comprehensive HELP menu
• Developer need to understand the design process through the eyes of non engineers.

b. (Engineers) Integration of intelligent design knowledge-base (IIKB)

• Diagnostics to assist with debugging benchmarking for comparison of results (e.g. EPA databases)
• Default or built in performance comparisons, benchmarking or ratings such as Energy Star or LEED
• Multi-objective design optimization
• Assistance on control settings (e.g. air flow and set point temperatures)
• System recommendations arrived at through an algorithm of climate and building usage
• Interface with manufacturers' information - e.g. standard formats for MEP equipment, windows, etc that can
be imported directly. Agencies such as ORNL for instance could output test results on materials and
assemblies in this format. Companies could provide product information in this format. This would simply
make it easier to incorporate reliable and effective data into simulation modeling in a similar manner to how
manufacturers offer DXF plans, models, and details of products and components.
• Introducing optimization models to identify optimal design considering performance and cost

40
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

Part III ‐ ACCURACY of the tools

19. Indicate how important you think each of the following objectives is, concerning tools ACCURACY

20. Which tool(s) fulfill the following criteria?

41
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

21. Indicate how important you think each of the following objectives is, concerning tools ABILITY to
SIMULATE COMPLEX & SPECIFIC BUILDING COMPONENTS

2. Which tool(s) fulfill the following criteria?

42
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

23. What other features should be improved in the future development of Building Energy Simulation
Tools concerning the ABILITY to SIMULATE COMPLEX & SPECIFIC BUILDING COMPONENTS? (optional)

(3) a. (Architects) Accuracy & ability to simulate complex & detailed building components
(AASDC)
• Renewable energy systems calculators should be a part of the package and tied into the overall project's
energy performance.
• BESTS" until they can do all this stuff and certify Passive Houses
• Passive strategies such as green roofs and natural ventilation
• Ability to easily simulate essential elements (i.e. fins overhangs) in sufficient detail
• Building envelope design optimization
• Consider natural ventilation to combine with HVAC system design
• Integration of daylighting & daylight energy with other tools like EP
• Inform users as to the cost impacts of energy reduction measures

b. (Engineers) Accuracy & ability to simulate complex & detailed building components
(AASDC)
• Real-time results, parametric feedback.
• Collecting realistic data from cases to establish performance based data sets
• Optimized for small, ultra-efficient buildings
• Data to measure uncertainty
• Adapt to the complexities of the real life designs and climatic conditions
• Model thermal mass, air-to-air heat exchangers, passive and active solar gains, or the most efficient lighting
and passive drying options, radiant slabs/beams, ground source HX, heat recovery chillers etc,
• Better analysis for double skin facades, VRV systems, dedicated outdoor air systems, and natural ventilation
• Indication of the degree of error that could be expected in the results
• Error estimate of models for validation and acceptable error range
• Validation and Verification of the simulation output
• Gather data after implementation and get the performance data back to into the software
• Be built on an underlying database to aid in benchmarking
• Perform trade-off analysis and an LCA tool to compare different options
• Ability to model complex HVAC and lighting control strategies
• Wider range of HVAC and natural ventilation modeling techniques
• Simulate monitor daylighting, displacement ventilation and chilled beam systems
• Test cases representing building in reality
• Robustness of models. Features should not be added until they are well-tested features and well-considered
• Allow more than one system per zone
• Describe uncertainty with the data model
• Clarity on the algorithms used to perform the simulations and the limitations of those algorithms

43
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

Part IV ‐ INTEROPERABILITY of Building Model

24. Indicate how important you think each of the following objectives is, concerning INTEROPERABILITY
OF THE BUILDING MODEL

25. Which tool(s) fulfill the following criteria?

26. What other features should be improved in the future development of Building Energy Simulation
Tools concerning the INTEROPERABILITY OF THE BUILDING MODELING ?

(4) a. (Architects) Interoperability of building modeling (IBM)

• Allowing organic modeling of curved volumes and non-cubical zones and volumes
• Ability to easily model essential elements (i.e. fins overhangs) in sufficient detail
• Ability to directly import .dwg or Revit files.
• Allowing input from multiple modeling programs (sketch up, rhino, 3dmax, Revit, etc) easily and with
minimal error. There needs to be clear guidance for how to build models in each of these interfaces in order
to facilitate the use of these models quickly and easily.
• Developing complex geometries
• Ability to merge architectural CAD drawings into respective thermal zones
• Change building geometry without having to re-enter all data variables from scratch
• Importing of detailed geometries with more accuracy and all layers being correctly imported in energy
simulation software

44
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

• Proper translation of the geometry in complex models

b. (Engineers) Interoperability of building modeling (IBM)

• One common language like gbXML (but more robust) to become an open standard, third party organizations
need to create a standard language.
• 3D parametric modelling
• Full IFC compliance: Import / Export equally robust, all elements that can be modeled must be able to be
exported / imported in IFC with all relevant data (at a minimum name, type, size, material) - this includes
MEP as well as Architecture & Structure
• exchange of model needs to be more seamless and less frustrating which would greatly facilitate the iterative
process of optimizing the design
• library of building components and building assemblies in a common format or formats (GBXML, IDF)
• Components that include data that describe how they behave

Part V ‐ MOST IMPORTANT features of a simulation tool

27. What are the MOST IMPORTANT features of a simulation tool?

Architect &Engineers priorities from BPS tools

Architect S1 Architect S2 Engineer S1 Engineer S2

98
93
77
66 70
58 57 55 58
52 53
40 44
32
25
20

Integration of intelligent Frienliness of the Interoperability of Accuracy and ability to

design knowledge‐base to interface concerning building modeling e.g. simulate complex cases
assist designer in decision usability & informarion exchanging 3D models
making management from other programs

45
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

General Comments

a. (Architects & Engineers) Integrated Building Design Process

• Should include building owner, building users, government regulatory and advisory agents, engineering,
construction, facilities management agents.
• Toolkits for corresponding all design stages
• Tools are not practical for the design process
• Being able to work with the software at conceptual and DD level that outputs information that is really
useful.
• Integrate different performance domains
• The flexibility to provide basic information during pre-design while more complex information in later
design phases.
• Reliable tools address late design phases
• Integrated tools intended for early phase design decision making Automatic graphic output (plots and
graphs) of simulation results
• Suitability for the entire design process
• Integration of various analysis features in a single software

b. (Architects & Engineers) Tools and training cost, learning curve and future
development
• proper training in building science
• easy learning curve
• cost of programs for students
• Tutorials, help menu, courses
• Video guidance on how to use
• provide adequate help either at the beginning of the tool or where ever necessary while performing
simulation/calculation use

46
© Shady Attia, Setting criteria for the use of building performance simulations in building design: A user survey, UCL 2010

Results: Ranking the Ten Tools

Ranking the Ten Tools

Architect Engineer
85 82 82
75 77 72 73
70 68
61 58 58 57 52
42 40
32 36
29
7

Ranking the Ten Tools

Engineers
85 82
72 70 68 65
40
25 20
4

47