Decision Support Systems 135 (2020) 113322

Contents lists available at ScienceDirect

Decision Support Systems

journal homepage: www.elsevier.com/locate/dss

The effect of interactive analytical dashboard features on situation T

awareness and task performance
Mario Nadja,⁎, Alexander Maedchea, Christian Schiederb
a
Karlsruhe Institute of Technology (KIT), Kaiserstraße 89-93, 76133 Karlsruhe, Germany
b
Technical University of Applied Sciences Amberg-Weiden, Hetzenrichter Weg 15, 92637 Weiden, i.d.Opf., Germany

ARTICLE INFO ABSTRACT

Keywords: In recent years, new types of interactive analytical dashboard features have emerged for operational decision
Interactive analytical dashboards support systems (DSS). Analytical components of such features solve optimization problems hidden from the
What-if analysis human eye, whereas interactive components involve the individual in the optimization process via graphical
Situation awareness user interfaces (GUIs). Despite their expected value for organizations, little is known about the effectiveness of
Out-of-the-loop syndrome
interactive analytical dashboards in operational DSS or their influences on human cognitive abilities. This paper
Eye-tracking
Operational decision support systems
contributes to the closing of this gap by exploring and empirically testing the effects of interactive analytical
dashboard features on situation awareness (SA) and task performance in operational DSS. Using the theoretical
lens of SA, we develop hypotheses about the effects of a what-if analysis as an interactive analytical dashboard
feature on operational decision-makers' SA and task performance. The resulting research model is studied with a
laboratory experiment, including eye-tracking data of 83 participants. Our findings show that although a what-if
analysis leads to higher task performance, it may also reduce SA, nourishing a potential out-of-the-loop problem.
Thus, designers and users of interactive analytical dashboards have to carefully mitigate these effects in the
implementation and application of operational DSS. In this article, we translate our findings into implications for
designing dashboards within operational DSS to help practitioners in their efforts to address the danger of the
out-of-the-loop syndrome.

1. Introduction systems (DSS) largely occurs via graphical user interfaces (GUIs) [3],
the design and implemented features of a dashboard are of specific
In the wake of the COVID-19 pandemic, dashboards that summarize importance for the effectiveness of a DSS, particularly at the operational
large amounts of information have become a topic of public interest. In level [4]. Over the years, research has proposed several possible solu-
particular, an interactive web-dashboard from Johns Hopkins tion types. A first research direction focuses on reporting current op-
University has proven useful in monitoring the outbreak of the pan- erations to inform operational decision-makers [3]. Such dashboards
demic and keeping billions of people informed about current develop- are rather static by nature and do not interact with the user. Static
ments [1,2]. Such dashboards help to illustrate key figures on a screen dashboards focus on visual features, which attempt to present in-
full of information [3] and provide visual, functional, and/or inter- formation efficiently and effectively to a user [4]. Few [3] reports 13
active analytical features to handle chunks of information effectively common mistakes in visual dashboard design (e.g., introducing mean-
[2]. Similarly, these tools can also help ensure that operational deci- ingless variety or encoding quantitative data inaccurately). However,
sion-makers in organizations are not overwhelmed by the avalanche of static dashboards appear no longer sufficient to account for the in-
data when trying to use information effectively [4]. The challenge for creased need to analyze complex and multidimensional data at the
contemporary organizations and related research is, therefore, to make operational level. This issue is a key challenge for the visualizations
the collected data more valuable to operational decision-makers. used in static dashboards, which are often labeled “read-only” [6].
Dashboards and their underlying features have become an essential Echoing this concern, a second research direction complements
approach to addressing this challenge [4]. Negash and Gray [5] de- static dashboards with functional features [6]. Functional features refer
scribe dashboards as one of the most effective analysis tools. As the indirectly to visualization but define what the dashboard is capable of
interaction between operational decision-makers and decision support doing [4]. The aim is to create interactive dashboards that involve users

⁎
Corresponding author.
E-mail addresses: [email protected] (M. Nadj), [email protected] (A. Maedche), [email protected] (C. Schieder).

https://doi.org/10.1016/j.dss.2020.113322
Received 15 August 2019; Received in revised form 12 May 2020; Accepted 12 May 2020
Available online 19 May 2020
0167-9236/ © 2020 Elsevier B.V. All rights reserved.
M. Nadj, et al. Decision Support Systems 135 (2020) 113322

during data analysis. Scientific articles in this stream suggest that in- translation of our findings into implications for designing dashboards
dividuals should drill down, roll up, or filter information, or are auto- within operational DSS to help practitioners in their efforts to address
matically alerted to business situations [7]. Such involvement appears the danger of the out-of-the-loop syndrome.
to support users' comprehension of the complex nature of the data. We have chosen production planning as the study context, focusing
However, obtaining this understanding comes at the expense of cog- on advanced planning and scheduling (APS), the synchronization of
nitive effort and time required to process the data manually, which raw materials and production capacity (cf. section 4.2). First, the
might lead to delayed decisions or errors [8]. amount of APS data sets up severe challenges concerning the display
Against this backdrop, interactive analytical dashboards have resolution in dashboards [26]. Second, until now, few publications
emerged in recent years as a third research direction [9]. Such dash- addressed dashboards and their influence in APS and production
boards recognize the limits of low human involvement with static planning in general. For instance, Wu and Acharya [27] outlined a basic
dashboards, the high human involvement with interactive dashboards, GUI design in the context of metal ingot casting. In addition, another
and value the introduction of interactive analytical features [10]. The study by Zhang [28] visualized large amounts of managerial data for
idea is to solve optimization problems automatically via computational evaluating its feasibility. There is a need to understand how planners
approaches in the backend (analytical component) but to (still) involve effectively use dashboards because planning is not a static, one-time
the individual in the optimization process via the GUI (interactive activity but rather requires continuous adaption of vast data amounts.
component) [11]. Implementations range from robust trial-and-error to Third, this challenge is even more daunting because planners must in-
more advanced approaches such as interactive multi-objective optimi- creasingly make business-critical decisions in less time [26].
zation [12]. Yigitbasioglu and Velcu [4] and Pauwels et al. [9] argue The ensuing section introduces the background related to SA theory
that the promise of interactive analytical features such as a what-if and dashboards. The section thereafter develops the study's hypotheses.
analysis is unrealized in current dashboard solutions for operational In the fourth section, we illustrate the research method. The data
DSS. A what-if analysis is a trial-and-error method that manipulates a analysis and results are explained in the fifth section. The sixth section
dashboard's underlying optimization model to approximate a real- discusses the theoretical and practical implications. The final section
world problem [13]. However, by introducing an interactive analytical concludes the study.
feature, operational decision-makers can interact with the optimization
system without knowing the optimization model or understanding how 2. Background
the optimization procedure behind the results operates [14]. Such be-
havior can harm an individual's comprehension of the current situation, 2.1. Theory of situation awareness
nourishing an out-of-the-loop problem—a situation in which a human
being lacks sufficient knowledge of particular issues [15]. The fear is In recent decades, SA has become one of the most discussed con-
“that manual skills will deteriorate with lack of use” and individuals cepts in human factors research [24]. Although the original theoretical
will no longer be able to work manually when needed ([15], p. 381). impetus of SA resided in military aviation, the concept has splashed
Although what-if analyses have been investigated for DSS, empirical over almost any area that addresses individuals executing tasks in
results have shown conflicting results. In particular, opposing patterns complex and dynamic systems. We used the model by Endsley [25] as
for the relationship toward decision performance have been reported in this model is one of the most prominent and widely used perspectives in
different studies. Benbasat and Dexter [16], as well as Sharda et al. the scientific community and debate around SA. Its robust and intuitive
[17], found a positive relationship. However, Fripp [18] and Goslar nature has been confirmed in different domains enabling scholars to
et al. [19] have demonstrated no statistically significant effect. In turn, measure the concept and derive corresponding requirements for DSS
others like Davis et al. [13] found that decision-makers performed [21].
better without a what-if analysis. With these conflicting results, it is SA refers to an individual's knowledge about a specific situation
surprising that academic literature gives relatively little attention to (1) [25]. It arises through his or her interaction with the environment. An
how interactive analytical dashboard features should appear or (2) how adequate level of SA is known to affect subsequent decisions and ac-
they influence human beings' cognitive abilities in operational DSS [4]. tions positively. The resulting activities induce changes to the en-
An adequate level of situation awareness (SA) is emphasized as a pro- vironment. As the environment changes, SA must be updated, which
mising starting point to examine this unfolding research gap [20]. requires a cognitive effort of the respective individual. Due to this in-
Human factors research confirms that human beings' SA represents a teraction, forming and maintaining SA can be difficult to achieve.
key enabler for operational decision-making and task performance Endsley [25] describes these difficulties as obstacles such as the out-of-
[21,22]. The importance of SA raises the question of how dashboard the-loop problem. The resulting degree of SA is thereby often described
features can be designed to influence SA positively [23,24]. The ob- as “high” or “low” [29]. Thus, we define SA as a quality criterion in
jective of this study is, therefore, to examine the relationship between terms of completeness and accuracy of the current state of knowledge,
an interactive analytical dashboard feature, human beings' SA, and task stating whether a human has an appropriate SA level.
performance in operational DSS. We implement a what-if analysis to However, measuring SA refers to a complex proposition. A plethora
study this relationship. Prominent references acknowledge that present of approaches exists and academic discourse continues regarding which
dashboards “just perform status reporting, and development of [a] of those represent the most valid and reliable one to measure SA [29].
what-if capability would strongly enhance their value” ([9], p. 9). We In general, research advocates a mixture of approaches, such as (1)
rely on Endsley's [25] SA model as a theoretical guide for our analysis. freeze-probe or (2) process indices. In addition, most SA studies also
It is established in academia and has shown favorable outcomes in measure (3) task performance indicators [29]. In the following, these
different studies [21]. We formulate the following research question: three measurement approaches will be introduced.
How does a what-if analysis feature in an interactive analytical dashboard
influence operational decision-makers' situation awareness and task per- 2.1.1. Freeze-probe technique
formance in operational DSS? Freeze-probe techniques involve random stops during task simula-
The primary contribution of this work resides in linking the what-if tion and a set of SA queries concerning the current situation [29].
analysis feature to SA and task performance. A second contribution Participants must answer each query based on their knowledge of the
resides in incorporating an analysis of eye-tracking parameters and the current situation at the point of the stop. During each stop, all displays
Situation Awareness Global Assessment Technique (SAGAT) in one of are blanked. The Situation Awareness Global Assessment Technique
the first large-scale lab experiments (n = 83) on this topic to offer in- (SAGAT) is an established freeze-probe procedure that is in line with
sights from a holistic viewpoint. A third contribution resides in the the model by Endsley [25]. The SAGAT approach has been specifically

2
M. Nadj, et al. Decision Support Systems 135 (2020) 113322

Fig. 1. Components of a Dashboard based on Meignan et al. [11].

created for the military and aviation domain. However, over the years, simplicity to collect, relying on task performance indicators appears
different SAGAT versions have been created in other fields, making this beneficial [29].
measurement approach one of the most popular ones to assess SA [29].
The benefits of this approach represent their alleged direct, objective
nature, which eliminates the problems associated with assessing SA 2.2. Dashboard components and types
data post-trial [22]. This approach possesses a high validity, reliability,
and user acceptance [29]. Classical business intelligence and analytics (BIA) refers to a data-
centric approach promoting strategic and tactical decisions based on
(mostly) retrospective analysis connected to a restricted audience of
2.1.2. Process indices
BIA experts and mangers [33]. Nowadays, BIA refers to all methodol-
Process indices assess cognitive processes of human beings to form
ogies, techniques, and technologies that capture, integrate, analyze, and
and maintain SA during task execution [29]. Specifically, eye-tracking
present data to generate actionable insights for timely decision-making
supports the definition of proficient process index measures and
[34]. The transformation of raw data into actionable insights requires
therefore refers to a further technique to measure SA [29]. Such devices
three BIA technologies: data warehouses (DWH), analytical tools, and
capture fixations of human beings on the elements of the screen to in-
reporting tools [35]. A DWH is a “subject-oriented, integrated, time-
dicate the degree to which such elements have been processed. This
variant, nonvolatile collection of data in support of management's de-
operationalization is based on the assumption that a high fixation level
cision-making process” ([36], p. 33). Typically, extract, transform, load
engages the participant in information-relevant thinking. Although this
(ETL) processes load data into a DWH, before analytical tools produce
assumption represents only a proxy for true SA, it follows the same logic
new insights. On this basis, dashboards are created to inform decision-
as attempts in other research communities that leverage gaze patterns
making and next actions [37]. Hereby, dashboards are usually called
to deduce attention or assign elaboration to memory or recall [30,31].
the “tip of the iceberg” of BIA systems. In the backend, complex and
However, whereas freeze-probe techniques abound in academic dis-
(often not-well) integrated ETL infrastructure resides, attempting to
course [29], only few investigations rely on process indices (such as
retrieve data from diverse operational systems. The relevance of these
eye-tracking) or study their reliability and validity to measure SA.
backend systems is indisputable. However, dashboards appear to hold a
significant role in this information processing chain because the inter-
2.1.3. Task performance indicators action between operational decision-makers and DSS primarily occurs
Most SA studies typically include task performance indicators [29]. via GUIs [3]. Thus, the dashboard design is relevant for BIA systems due
Depending upon the task context, different performance measures are to its influence on an individual's efficiency and productivity. However,
defined and collected. For instance, “hits” or “mission success” re- whereas press articles [38,39] and textbooks [3,40] are abundant, few
present suitable measures in the military context. In turn, driving tasks scientific studies have looked at dashboard features and their con-
could involve indicators like hazard detection, blocking car detection, sequences for users; thus, limited guidance for scholars and practi-
or crash avoidance. Task completion, on the other hand, has been used tioners is currently offered [4,9]. The handful of dashboard articles
as a measure in the study of user performance under the influence of have studied intentions for their usage [41], case studies of instantia-
instant messenger technologies [32]. Due to their non-invasiveness and tions used in organizations [39], different implementation stages [42],

3
M. Nadj, et al. Decision Support Systems 135 (2020) 113322

adoption rates [43], and metrics selection [44]. In the following, we refers to information obtained during this optimization process. Can-
describe the components and types of a dashboard (cf. Fig. 1). didate solutions are final solutions to an optimization problem [11].
Interactive optimization systems involve the user at some point in time,
2.2.1. Visual features allowing him or her to modify the outcomes of the optimization system
In any dashboard type, different visual features are used to present significantly. These systems leverage user feedback from the presented
data in a graphical form to reduce the time spent on understanding and candidate solutions and intermediate results to adjust either the opti-
perceiving them. Visual features relate to the efficiency and effective- mization model or the optimization procedures via a preference model.
ness of information presentation to users [4]. A good balance between The preference model contains the preference information obtained
information utility and visual complexity is therefore needed. Visual from the user feedback [12]. Common examples of a preference model
complexity refers to the level of difficulty in offering a verbal descrip- represent weight values in an objective function or heuristic informa-
tion of an image [45]. Studies show that varying surface styles and tion for the optimization procedure.
increasing numbers and ranges of objects increase visual complexity, The user can refer to different interactive analytical features to offer
whereas repetitive and uniform patterns decrease it [46]. Usually, various forms of feedback to the produced candidate solution or in-
dashboards leverage colors to differentiate objects from one another. termediate results. The user feedback can consist of adjustments of
The colors red and yellow, which are more likely to attract attention, parameter values leveraging interactive multi-objective optimization.
are examples for signaling distinction [25]. Using such properties too In an interactive evolutionary algorithm, the user can choose the most
often or inappropriately may lead to confusion or errors because the promising solution from a set of alternatives. Trial-and-error features
user would no longer be able to identify the critical information. Si- are used for both modifying the optimization model and optimization
milarly, superfluous information within charts, so-called “chart junk” procedures. The first approach corresponds to a what-if analysis in
such as overwhelming 3D objects or non-value-adding frames, can ap- which the user modifies the data, constraints, or objectives of the op-
pear impressive but severely distract users and induce undesired “visual timization problem, whereas the optimization system offers a solution
clutter”. According to Tufte [47], a high data-ink ratio can reduce this for evaluating these adjustments. In the second case, users modify some
problem. Data-ink ratio is a parameter that defines the relationship of parameter values in an optimization procedure.
ink used to illustrate data to the overall ink leveraged to represent the Although not exhaustively implemented, preference learning gen-
chart. A potential strategy refers to the erasing method by eliminating eralizes the feedback of a user to develop a model of his or her pre-
all components in charts with non-data ink. Some chart types promote ferences. Such a model can be used to extend the optimization model
visual illusions, which can potentially bias decision-making. Gridlines (e.g., by adding new constraints). However, it is also possible to in-
within charts are suggested as a useful technique to overcome such tegrate user feedback into the preference model without preference
pitfalls [48]. A detailed discussion on visual features can be found in learning. User values can be directly linked to the parameters of the
Few [3] and Tufte [47]. optimization model. A detailed discussion of interactive analytical
features can be found in Meignan et al. [11].
2.2.2. Functional features
Functional features indirectly link to visual features and comprise 2.2.4. Types of dashboards
what a dashboard can do [4]. Functional features in the form of point By now, numerous types of dashboards have been suggested not
and click interactivity enable the user to drill-down and roll-up. Tra- only for performance monitoring but also for advanced analytical
ditional approaches to point and click interaction refer to merely purposes, incorporating consistency, planning, and communication
clicking the mouse on the favored part of a list or chart to receive de- activities [9]. Next, we consider the dashboard concept as a whole and
tailed information. Cleveland [49] initiates brushing, an approach that describe it in terms of three different types.
illustrates details of the data, by placing a mouse pointer over the data Static dashboards are reporting tools used to summarize informa-
display. Such details might show information ranging in size from tion into digestible graphical forms that offer at-a-glance visibility into
single values to information from related data points. Filtering is an- business performance. The value resides in their capability to illustrate
other valuable functional feature. It enables users to not only sort for progressive performance improvements via fitting visual features to
relevant information but also identify hidden data relationships [7]. users. Depending upon the purpose and context, data updates can occur
Alerts and notifications also describe functional features [7]. They can once a month, week, day, or even in real-time [3]. Static dashboards
recommend corrective actions or make the user aware of issues as soon can fulfill both the urgency of fast-paced environments, offering real-
as measures no longer meet critical thresholds. Common influencing time data support, and tracking of performance metrics against en-
variables address type (e.g., warning or announcement), modality (e.g., terprise-wide strategic objectives based on historical data. However,
visual, audio, or haptic), rate of frequency (i.e., how often to alert the such dashboards do not involve the user in the data visualization pro-
user), and timing (i.e., when to induce the alert). cess and have issues with handling complex and multidimensional data
[6]. Interactive dashboards can be considered a step toward directly
2.2.3. Interactive analytical features involving the user in the analysis process. Interactive dashboards con-
Recent developments have emphasized the introduction of inter- sider not only visual features, but also introduce functional features
active analytical features within dashboards [10]. Such features (still) such as point and click interactivity [4]. These capabilities allow op-
involve the operational decision-maker in the optimization process via erational decision-makers to enable more elaborate analyses [6]. Ad-
the GUI (interactive component). Computational approaches and soft- dressing the impediments of static dashboards, they are used to estab-
ware tools solve the optimization problem in the backend (analytical lish a better understanding of the complex nature of data, which can
component). Any optimization system contains an optimization model also foster decision-making. However, the benefits of interactivity could
and optimization procedures to solve an optimization problem auto- increase the users' cognitive effort and required manual analysis time,
matically. An optimization model describes an approximation of a real- increasing the probability of delayed decisions or (even) errors [8].
world problem. Usually, it includes the definition of the objectives, In more recent years, research has called for more interactive ana-
decision variables, and constraints of the optimization problem. The lytical dashboards to address the limits of low human involvement
problem data define the values of the parameter of the optimization within static and high human involvement within interactive dash-
model when the optimization problem is triggered. In turn, the opti- boards [10]. Such dashboard types rely on interactive analytical fea-
mization procedure solves the problem instances directly linked to the tures [50]. The dashboard literature has confirmed and highlighted the
optimization problem by issuing either candidate solutions or inter- potential value of a what-if analysis feature to quickly access and assess
mediate results. An intermediate result is not necessarily a solution and different aspects of the data in current dashboards [9]. Despite the good

4
M. Nadj, et al. Decision Support Systems 135 (2020) 113322

prospects, these new types of features have (thus far) not materialized trust the DSS implicitly or even “overtrust” the system. “As an opera-
[4]. Potential obstacles concerning users trusting the interactive ana- tor's attention is limited, this is an effective coping strategy for dealing
lytical functionality implicitly appear to remain [14]. Such behavior with excess demands” ([53], p. 3). The result, however, would be that
can induce adverse effects on SA that, in turn, can evolve into an out-of- such human-machine entanglement can lead to intricate experiences
the-loop problem [15]. such as a loss of SA or (even) deteriorating skills [15].
Research shows that humans who rely on system support have dif-
3. Hypotheses development ficulties in comprehending a situation once they recognize that a pro-
blem exists. Even when system support can operate correctly most of
This section describes the research model and the hypothesized ef- the time, when it does fail, the ability to restore manual control is
fects. We expect different effects of the what-if analysis (in an inter- crucial [54]. Likewise, a planner can rely on a what-if analysis without
active analytical dashboard) on an individual's SA and task perfor- understanding the limits of the optimization model or the performance
mance. Furthermore, we study the effect of an interactive dashboard inadequacy behind the results, nourishing an out-of-the-loop problem.
(without what-if analysis) to introduce a baseline for the interpretation Many situations might exist for which neither the optimization model
of our empirical results. Finally, we study the effect of one control nor the optimization procedures have been implemented, trained, or
variable, an individual's mental workload because it has been ac- tested [11]. Such situations require manual intervention. Interactive
knowledged as a bias in DSS development [51]. dashboards offer the possibility for manual data analysis (e.g., via point
and click interactivity) to scrutinize all object-relevant information [9].
3.1. Impact of dashboard design on situation awareness (H1) Typically, such actions are known to keep the human in the loop and
aware of the situation [24].
Dashboard design can induce significant effects on operational de- In conclusion, a what-if analysis within an interactive analytical
cision-makers' SA because the set of dashboard features available de- dashboard is an effective coping strategy to quickly show operational
fines the quantity and accuracy of information obtained by a human decision-makers potential areas of improvement. However, it simulta-
being [4]. In turn, SA defines the state of knowledge for perceiving and neously increases the possibility of getting out-of-the-loop. Thus, we
comprehending the current situation and projecting future events [25]. theorize that the impact of an interactive dashboard on SA is higher
Such information appears to be important because it defines the basis to than the impact of the interactive analytical counterpart.
engage in effortful processing. Thus, the selection of dashboard features
H1. : An interactive analytical dashboard design leads to lower
should be offered in a way that transmits the information required to
situation awareness than an interactive dashboard design without
the human being in the most effective manner, which (ideally) in-
analytical features.
creases SA. We theorize that such transmission can be affected and
optimized by the respective dashboard design and set of features em-
ployed. An interactive dashboard design assists operational decision- 3.2. Impact of dashboard design on task performance (H2)
makers with manual data analysis [4]. The value of such designs stems
from their capability to offer means for comparisons, reviewing ex- The value of dashboards resides in their capability to increase
tensive histories, and assessing performance. In contrast to static control and handling by the user and thus foster task performance [3].
dashboards, interactive dashboards can go beyond “what is going on Task performance relies on the degree to which an operational decision-
right now,” enable filtering options, and drill down into causes of maker scrutinizes the set of available information within a dashboard.
bottlenecks. Such dashboards allow operational decision-makers to This assumption is based on the importance of human beings' SA when
move from general-abstract to more concrete-detailed information and assessing information [24]. However, when the universe of potential
thereby identify hidden data relationships from different perspectives solutions is large or time constraints exist, operational decision-makers
[9]. Such data activities enable a detailed review of the current situa- cannot reasonably assess every possible alternative [31]. They rather
tional context. create a consideration set, a subgroup of alternatives, which the in-
Interactive analytical dashboards introduce interactive analytical dividual is prepared for and aware of to assess further [55]. Interactive
functionality that facilitates operational decision-makers to simulate dashboards can guide the planner through the current planning in-
trends automatically. What-if analyses allow users to quickly receive formation. Based on the cognitive effort and time constraints of the
system support (even) without the need to review information in a more planner, he or she will create a consideration set from the presented
detailed manner or to understand the overall business situation [4]. view. In case of breakdowns, interactive dashboards enable human
Such automated analysis can represent a useful feature to enable in- beings to manually review detailed information and analyze their
dividuals immediate assessment of potential areas of improvement, causes (if necessary) [9]. In production planning, these derived insights
specifically when the dashboard is used as a planning tool [52]. Op- can be used to adjust the production plan.
erational decision-makers can leverage such interactive analytical fea- Operational decision-makers can also rely on a what-if analysis to
tures to obtain quick feedback on how specific changes in a variable adjust their consideration set [10]. They can leverage this functionality
(e.g., order fill rates) influence other values (e.g., profits). This way, to obtain quick feedback for specific relationships of variables (e.g., the
they can emphasize the significance of bottleneck components [9]. effect of order fill rates on profit) [9]. Studies have acknowledged the
We expect that the interactive dashboard design will outperform the potential value of such features to outline critical information that il-
interactive analytical one in terms of SA. Concerning the interactive lustrates most of the difference [52].
analytical dashboard, exceptional planning situations remain. These However, we expect that the interactive analytical design will out-
cannot be handled properly by the optimization model. The real-world perform the interactive one in terms of task performance. Interactive
problem can involve constraints that are hard to quantify. Furthermore, dashboards can represent a less effective means to promote task per-
the operational decision-maker must know several limitations of the formance. The underlying reasoning is that manual analysis activities
quality and efficiency of the optimization procedure [11]. Thus, op- (as offered by interactive dashboards) are cognitively demanding and
erational decision-makers (still) play a critical role in the optimization require time. Operational decision-makers must consider information
process and are required to be aware of the specifics of the planning critically and question its relevance before generating a carefully in-
situation [53]. However, by increasingly relying on a what-if analysis, formed (although not necessarily unbiased) judgment [56]. However,
planners can be deluded into no longer scrutinizing all objective and due to the limits of the human mind with storing, assembling, or or-
relevant information. Shibl et al. [14] showed that trusting DSS re- ganizing units of information, operational decision-makers might not be
presented a success factor; however, the majority of users appeared to able to transfer an (even) increased level of SA into their planning

5
M. Nadj, et al. Decision Support Systems 135 (2020) 113322

activities and thus might experience lower task performance [54]. causes of problems, whereas the interactive analytical one offers a
Common obstacles arise from restricted space or natural dissolution of what-if analysis on top. We conducted a laboratory experiment using a
information over time in human beings' working memory. Given ab- single-factor within-subject design to enhance statistical power for each
stract information, such dissolution can occur (even) in seconds. Con- setup and minimize error variance induced by individual differences
versely, with interactive analytical dashboards, the cognitive effort and [58]. We used a common technique to minimize the bias from potential
time required are lower because planners can rely on interactive ana- carryover and order effects. We randomized the order of experimental
lytical features. For example, a what-if analysis can help to identify sequences. First, we calculated the number of possible experimental
defective components rather than scrutinizing all of the information sequences (i.e., 2). Second, we randomly allocated each participant to
presented on its own [56]. Hence, we expect the following. one of the two sequences. Randomization is a well-known technique to
prevent unintentionally confounding the experimental design [59].
H2. : An interactive analytical dashboard design leads to higher task
Before the data collection, we tested the setup to check on the in-
performance than an interactive dashboard design without analytical
telligibility of the experimental tasks and to evaluate the measures. We
features.
included feedback rounds with practitioners to ensure closeness to
reality.
3.3. Association between situation awareness on task performance (H3)
4.1. Participants
SA is an important antecedent to improving the likelihood to
achieve higher task performance. Effective decisions require a good
We conducted our laboratory experiment with graduate students
state of knowledge in terms of perception, understanding, and projec-
who were enrolled in an advanced IS course. In total, 83 graduate IS
tion of the situation at hand [24]. Individuals experience more suc-
students (48 males, 35 females) took part. Their ages ranged from 20 to
cessful performance outcomes when they obtain a complete overview
34 years, with an average age of 24.1 years. We used students as a
and knowledge of the current situation [54]. When issues emerge, they
proxy for dashboard users in APS. We relied on students because similar
typically do so because some considerations of this overview are in-
laboratory experiments have shown that students are an appropriate
complete or incorrect [24]. The underlying reasoning is that a better SA
population not normally biased by real-world experiences [60]. Second,
can increase the control and handling of the system by the individual
the costs (e.g., financial compensation, physically presence at the
and thus contribute to task performance. Studies confirmed this re-
campus) to incentivize students to take part in the experiment are re-
lationship in different domains (e.g., automotive, military, or aviation).
latively low. In turn, one of the main limitations of relying on profes-
A decrease of SA is often correlated with reduced task performance,
sionals in experiments is that it is problematic to incentivize them [59].
representing the most critical cause of aviation disasters in a review of
Third, APS represents a challenging, complex, and cognitive-de-
over 200 incidents [57]. Hence, we hypothesize the following.
manding task. IS students have learned the abilities or knowledge to
H3. : Situation awareness is associated with higher task performance. apply it in complex contexts. Hence, these students seem to fulfill the
necessary prerequisite for conducting the experimental task. In this line,
Fig. 2 summarizes the overall research model.
we concur with the review by Katok [59] that students can perform as
good as professionals do. Bolton et al. [61] confirmed this conclusion in
4. Research method a carefully executed experiment. Fourth, to account for the lack of
system experience, we applied different techniques to prepare partici-
We developed two dashboards. The interactive design assists the pants for using the dashboard (e.g., personal support, training tasks,
user with manual data analysis to filter, roll up, or drill down into the small groups, help button). Because both dashboards participants

Fig. 2. Research model.

6
M. Nadj, et al. Decision Support Systems 135 (2020) 113322

committed 2.5 errors on average, it appears that they had no difficulties situation, the planner obtains access to supply and retail data within
in using them. Lastly, we discussed our study with two production sortable tables and bar charts. The sortable tables and bar charts are
planners and followed their experienced advice regarding several as- combined with drill-down, roll-up, and filter functionality to explore
pects, that is, how to assess the degree of SA and performance or how to the underlying sets of data deeply.
design both dashboard types and determine appropriate areas of in- Concerning the what-if analysis, the interactive analytical dash-
terest (AoI). board design introduces a simulation area capable of visualizing bot-
tleneck components and their criticality for the production plan (cf.
4.2. Experimental task Fig. 4). The bar chart uses colorful indications (i.e., green/ red) to assess
the effects and criticality of bottleneck components. In this way, it
The problem type at hand was an APS task. Participants had to plan emphasizes immediately whether the current data entries form a valid
the production of bicycles in a fictional manufacturing company. production plan. In the event of a component shortage or surplus, the
Different components (e.g., bicycle handlebar, bell, lamp, reflectors, chart is colored in red. The difference between needed and possible
tire, chair) were available for each bicycle. Hereby, the amount of orders is visualized as a number within the bar chart.
available components or which component was required for which bi-
cycle could differ. Per planning week, only a limited amount of com- 4.4. Experimental procedure
ponents was available. In addition, for each product, the minimal
amount of planned customer demands of all planning weeks was of- The study lasted for approximately one hour and three minutes and
fered. Customer demands represented the basis for the calculation of comprised three sections: an introduction (10 min), training (25 min),
needed components per planning week. In total, four planning weeks and the actual testing section (28 min). Upon arrival, participants
had to be planned by the participants so that all constraints were met started by signing the information consent form. The computer lab was
and the total revenue was maximized. Three parameters were critical to prepared with running Tobii pro X2–30 eye-tracker equipment. The
generate an effective production plan: (i) the number of available eye-tracker recorded the participant's eye movements as X and Y co-
components, (ii) the number of products to be produced (demand), and ordinates (in pixels) at intervals of 30 ms, applying dark-pupil infrared
(iii) information about which component is linked to which product. technology and video. Participants were introduced to the production-
Both designs shared the same APS process and objective to create an planning problem and the dashboard design by the experiment in-
optimal production plan solving a planning problem within the given structor. They had the opportunity to go through the problem for some
constraints. Both plans comprised initial states and planning restric- minutes to become familiar with the topic. The training section fol-
tions. The initial state was characterized by a set of raw data that did lowed the introduction, aiming to practice the generation of a pro-
not comply with the planning restrictions. For each plan, all decision duction plan with both alternatives. The training sections comprised
packages shared the same complexity. The initial raw data for both small groups of three to five participants to offer participants the pos-
plans only differed in their conditions, such as component numbers or sibility to ask questions individually. The instructor and a help menu
weekly demand. We introduced constraints to ensure that users could offered by the application provided guidance and support. Participants
execute the tasks within an appropriate amount of time: component could use as much time as they needed; they required approximately
shortage represented the only restriction for planning, whereas human 25 min to finish both training activities (one for each design). In the
and machine resources were assumed unrestricted. The participants testing section, each participant finished one session for each design.
were required to create a production plan by modifying the changeable Each session lasted a maximum of 14 min.
raw data with (w/) and without (w/o) a what-if analysis.
4.5. Measurement
4.3. Treatment conditions
4.5.1. Situation awareness
Fig. 3 shows a screenshot of the interactive dashboard design. The We employed the SAGAT freeze-probe technique to measure SA. In
design comprises three elements: i) a toolbar to navigate through dif- our study, we halted the task four times within each alternative and
ferent areas of interest at the very top, ii) a navigation bar top right to blanked out the screen. A message on the screen asked the participants
perform different means of analysis and iii) a content area showing the to respond to questions related to their perceptions of the situation in
respective results. To support a holistic view of the as-is planning order to obtain a periodic SA snapshot. Participants answered three

Fig. 3. Screenshot of the interactive dashboard design.

7
M. Nadj, et al. Decision Support Systems 135 (2020) 113322

Fig. 4. Screenshot of the interactive analytical dashboard design.

Table 1 Participants rated each subscale immediately after completing a

SAGAT questionnaire. production plan with the respective dashboard type. We used a
1. What is the current quantity of P1 in the stock?
Wilcoxon signed-rank test to identify whether there is a significant
2. What is the current size of the purchase order for K26? difference in mental workload for both alternatives. This non-para-
3. What is the current size of the purchase order for K9? metric test compares two related samples on non-normally-distributed
4. The production of which product is in danger? data (cf. Table 3).
5. Select the machine(s) which have problems:
The test showed no statistical significance (p > .1), confirming that
6. Enter the current Delivery Reliability:
7. Enter the supplier which does deliver purchased part K13: the perceived mental workload between both designs was comparable.
8. In comparison to the current state, what would happen to Delivery Table 4 summarizes the results.
Reliability if machine 1 broke down?
9. What is the needed amount of purchased part E9 to fulfill all production
orders? 5. Data analyses and results
10. What would you do concerning the production orders?
11. What would you do concerning the purchased parts? 5.1. Normality assumption
12. What would you do concerning the workplace capacity?

We used the Shapiro-Wilk statistic to test the data for assumptions

of normality. As the p-values were less than the depicted alpha level
questions at each stop (cf. Table 1).
(alpha = .1), the null hypothesis was rejected (cf. Table 5). Hence, we
To evaluate whether participants' replies were correct, we con-
applied the Wilcoxon signed-rank test statistic. This test is capable of
trasted them with logged data at the moment of the freeze. We analyzed
obtaining the non-normally-distributed data of two related samples.
the SA scores as the percentage of correct answers for each dashboard.
This way, we could test our hypothesis in terms of participants' SA
Eye-Tracking. Two AoIs for each dashboard feature were defined.
(measured by SAGAT score, fixation duration, and fixation count) and
An AoI refers to the “area of a display or visual environment that is of
task performance (measured by errors committed).
interest to the research or design team” ([62], p. 584). We oper-
ationalized SA via two eye-tracking measures: (i) participant's fixation
duration and (ii) fixation count spent scrutinizing the information of the 5.2. Impact of dashboard design on situation awareness (H1)
respective AoI. A fixation occurs if an eye concentrates on a particular
point for a specific period in time. Any eye-movement around 2° of In the following, we assess the descriptive statistics for the con-
visual arc for at least 60 ms in time is called a fixation [62]. Fixation structs employed, compare their statistical significance and calculate
count refers to the total number of fixations counted in an AoI. Fixation the corresponding effect sizes. Our results indicate that the interactive
duration is defined as the average fixation time spent on an AoI. By analytical dashboard was able to outperform the interactive design in
studying the fixation duration and fixation count with both designs, we terms of task performance. Furthermore, the participants' fixation
can conclude which alternative drew more attention. count, duration, and SA scores were significantly higher for the inter-
Task Performance. We measured task performance by the number of active alternative. All participants answered more than 50% of the
errors committed for each dashboard alternative. Participants could SAGAT questions correctly (compared with their maximum values) for
save their production plan when they felt they had arrived at a suitable both alternatives. Thus, both designs appeared to achieve sufficient SA
solution. If a participant did not successfully manage to save the plan, levels. We confirmed hypothesis H1 because the interactive design
we took the last version. (Mean = 0.53, Standard Deviation = 0.18) outperformed the coun-
Mental Workload. We controlled for individuals' perceived mental terpart (M = 0.39, SD = 0.16) on the SA scores. The Wilcoxon signed-
workload because it induces a direct effect on their ability to execute rank test revealed a significant difference (p = .000) with a medium
tasks. Hence it can influence the effectiveness and efficiency of inter- effect size r of 0.306.
actions with computers or GUIs [51]. We have employed the NASA-TLX The collected fixation duration and fixation count for each alter-
based on Hart and Staveland [63] to measure perceived mental work- native were subjected to a Wilcoxon signed-rank test. Participants fix-
load. This index possesses high validity and refers to “a multi- ated more on the interactive design (M = 716.34, SD = 314.59) as
dimensional, self-reported assessment technique that provides an esti- compared to the interactive analytical screen (M = 335.70;
mation of the overall workload associated with task performance” SD = 185.12). The difference between both designs was significant,
([51], p. 72). Table 2 summarizes the corresponding items. with a p-value less than .001 with a large effect size r of 0.549.

8
M. Nadj, et al. Decision Support Systems 135 (2020) 113322

Table 2
NASA-TLX questionnaire.
Mental demand How mentally demanding was the task?
Temporal demand How hurried or rushed was the pace of the task?
Performance How successful were you in accomplishing what you were asked to do?
Effort How hard did you have to work to accomplish your level of performance?
Frustration How insecure, discouraged, irritated, stressed, and annoyed were you?

Table 3 Table 5
Testing distributions of normality by treatment for mental workload. Testing distributions of normality of constructs by treatment.
Term Skewness Kurtosis Shapiro-Wilk test Term Skewness Kurtosis Shapiro-Wilk test

statistic p-value Statistic p-value

Interactive (w/o what-if analysis) Interactive (w/o what-if analysis)

Mental demand −0.653 −0.015 0.953 0.004⁎⁎ Fixation duration −0.263 −0.823 0.971 0.058†
Temporal demand −0.276 −0.933 0.954 0.005⁎⁎ Fixation count −0.508 −0.349 0.964 0.020⁎
Performance −0.479 −0.685 0.951 0.003⁎⁎ SAGAT 0.369 −1.383 0.821 0.000⁎⁎⁎
Effort −0.157 −0.862 0.960 0.011⁎ Errors committed −1.103 1.101 0.821 0.000⁎⁎⁎
Frustration −0.605 −0.155 0.953 0.004⁎⁎
Interactive-analytical (w/ what-if analysis)
Interactive-analytical (w/ what-if analysis) Fixation duration 1.212 2.050 0.920 0.000⁎⁎⁎
Mental demand −0.671 0.939 0.939 0.001 ⁎⁎
Fixation count 0.770 1.079 0.959 0.010⁎
Temporal demand −0.167 −0.979 0.958 0.008⁎⁎ SAGAT 0.569 −0.991 0.831 0.000⁎⁎⁎
Performance −0.613 −0.184 0.946 0.002⁎⁎ Errors committed −0.444 −0.885 0.876 0.000⁎⁎⁎
Effort −0.046 −1.231 0.938 0.001⁎⁎
Frustration −0.579 −0.586 0.931 0.000⁎⁎⁎ *** p < 0.001, ** p < 0.01, * p < 0.05, † p < 0.1, ns = not significant.

*** p < 0.001, ** p < 0.01, * p < 0.05, † p < 0.1, ns = not significant.
more time participants spent on the screen, the higher their SAGAT
scores became. The effect size of the interactive design (r = 0.313) was
Similarly, the interactive design (M = 208.26; SD = 102.29) out- larger than for interactive analytical design (r = 0.296). Concerning the
performed the interactive analytical design (M = 75.97; SD = 49.88) participant's fixation count, the correlation analysis by Spearman's Rho
on average in terms of fixation duration. Analyses indicated that this confirmed a significantly positive correlation of the SAGAT scores for
difference was significant (p = .000) with a large effect size r of 0.584. each design alternative, constituting a larger effect size within the in-
Thus, eye-fixation data also confirmed that the interactive design sup- teractive analytical design (r = 0.305). The interactive design showed
ported higher levels of participants' SA, compared to the interactive only weak significance (p < .1). In summary, the correlation analysis
analytical design (H1). for both alternatives indicated that a participant's higher fixation
duration (count) led to a greater ability to answer the SAGAT ques-
5.3. Impact of dashboard design on task performance (H2) tionnaire correctly (cf. Table 7).
In terms of the relationship between fixation duration (count) and
In turn, participants produced more errors (M = 2.82; SD = 1.08) the performance indicator errors committed, we found a significant
within the interactive design when trying to create a production plan. negative correlation for the interactive analytical design. The inter-
They committed fewer errors (M = 2.18; SD = 1.31) within the in- active analytical design shared a high effect size for both fixation
teractive analytical design. In line with our assumptions, this difference duration (r = −0.435) and fixation count (r = −0.366). In other
was significant between both dashboards (p = .000; H2), however only words, participants made fewer errors during their planning efforts, the
with a small effect size r of 0.295. This seems in line with the de- longer they looked at and the more they fixated on the interactive
scriptive statistics as, on average, both dashboard alternatives produced analytical dashboard screen. For the interactive design, participants'
approximately two to three errors per plan, indicating a low degree of fixation duration (count) did not significantly correlate with errors
errors committed during planning in general. Both dashboards ap- committed.
peared to achieve appropriate scores of task performance (cf. Table 6). Concerning SAGAT scores and errors committed, the correlation
analysis confirmed the expected negative correlation for each alter-
5.4. Association between situation awareness and task performance (H3) native, constituting a larger effect for the interactive analytical design
(r = −0.421; p = .004). In summary, for the SAGAT scores, the results
On average, the correlation analysis by Spearman's Rho showed that confirmed that a participant's higher SA led to higher task performance
participants' fixation duration shared a significantly positive effect on with both designs (H3). However, the eye-tracking data confirmed this
the associated SAGAT scores for both alternatives (p < .05). Thus, the relationship only for the interactive analytical design (cf. Table 7).

Table 4
Wilcoxon signed-rank test by treatment for mental workload.
NASA-TLX index Experiment treatment Wilcoxon signed-rank test

Interactive (Mean, SD) Interactive & analytical (Mean, SD) U r p-value

Mental demand 62.3 (21.7) 59.0 (23.3) −1.025 0.0796 0.305 (ns)
Temporal demand 51.6 (28.3) 49.1 (27.2) −0.945 0.0733 0.345 (ns)
Performance 58.4 (26.6) 61.9 (25.5) −1.248 0.0969 0.212 (ns)
Effort 47.7 (26.9) 43.1 (28.0) −1.211 0.0940 0.226 (ns)
Frustration 60.4 (22.1) 56.4 (23.8) −1.485 0.1153 0.137 (ns)

*** p < 0.001, ** p < 0.01, * p < 0.05, † p < 0.1, ns = not significant.

9
M. Nadj, et al. Decision Support Systems 135 (2020) 113322

Table 6
Descriptive statistics of constructs by treatment.
Term Experiment treatment Wilcoxon signed-rank test

Interactive (Mean, SD) Interactive & analytical (Mean, SD) Z r p-value

SAGAT (in %) 0.53 (0.18) Min. 0.17 0.39 (0.16) Min. 0.13 −3.946 0.306 0.000⁎⁎⁎
Max. 0.92 Max. 0.75
Fixation duration (in msec) 208.26 (102.29) 75.97 (49.88) −7.528 0.584 0.000⁎⁎⁎
Fixation count (in no.) 716.34 (314.59) 335.70 (185.12) −7.074 0.549 0.000⁎⁎⁎
Errors committed (in no.) 2.82 (1.08) 2.18 (1.31) −3.804 0.295 0.000⁎⁎⁎

*** p < 0.001, ** p < 0.01, * p < 0.05, † p < 0.1, ns = not significant.

Table 7 situation.
Construct correlations by treatment. Second, our analyses revealed that the introduction of interactive
Term [1] [2] [3] [4] analytical features is not cost-free. Participants who used the interactive
dashboard maintained a significantly higher SA when creating a pro-
Interactive (w/o what-if analysis) duction plan, whereas the interactive analytical design led to a lower
[1] Fixation duration –
degree of SA. Hence, participants who used the interactive analytical
[2] Fixation count 0.812⁎⁎⁎ –
[3] SAGAT 0.313⁎ 0.253† –
variant appeared to follow the system support (through the results
[4] Errors committed −0.125 (ns) −0.062 (ns) −0.311⁎ – proposed by the what-if analysis). Subsequently, they reduced their
Interactive & analytical (w/ what-if analysis)
effort and SA to scrutinize all object-relevant information. These find-
[1] Fixation duration – ings illustrate the danger for operational decision-makers of trusting the
[2] Fixation count 0.935⁎⁎⁎ – system results implicitly or (even) of “overtrust”, leading to potentially
[3] SAGAT 0.296⁎ 0.305⁎ – adverse effects such as an out-of-the-loop problem [53]. With in-
[4] Errors committed −0.435 ⁎⁎⁎
−0.366⁎⁎ −0.421⁎⁎ –
creasing simulation abilities, operational decision-makers can become
*** p < 0.001, ** p < 0.01, * p < 0.05, † p < 0.1, ns = not significant.
less aware and slower acting in recognizing problems on their own.
Thus, if problems require manual intervention, further time is necessary
Table 8 summarizes the results of the hypotheses. to comprehend “what is going on”, which sets boundaries on a quick
problem resolution. Following this argumentation, eye-tracking studies
[64] confirmed that continuous use of simulation reduces individuals'
6. Discussion cognitive skill set because such skills rapidly deteriorate in the absence
of practice. These effects can lead to an increasing tool dependence and
6.1. Theoretical implications problems when the underlying optimization model or procedures are
not implemented, trained, or tested for the current situation [11].
There are four central theoretical implications we want to empha- Third, our correlation analysis showed the link of SA to task per-
size. First, our analyses indicated that the interactive analytical dash- formance concerning the SAGAT scores. This finding is in line with
board showed significantly higher task performance. This result con- extant conceptual arguments as task performance is expected to cor-
firms the value of interactive analytical dashboards and encourages the relate with SA [54]. Still, in comparison to our Wilcoxon signed-rank
trend toward introducing interactive analytical features for operational tests for both dashboards, these findings might suggest that tension
decision-making. We showed that a what-if analysis has the potential to between maintaining high SA and accomplishing high task performance
direct participants to the most critical pieces of information such as exists. We conclude that high levels of SA are necessary in general but
bottleneck components that require immediate attention. Obstacles to do not suffice for high task performance. Although SA is relevant for
the what-if analysis included a lack of knowledge of operational deci- decision-making, different influencing variables might be involved in
sion-makers of the current planning situation, which, in turn, resulted converting SA into successful performance while “it is possible to make
in a lower degree of SA (compared with the interactive design). wrong decisions with good SA and good decisions with poor SA” ([22],
However, our results indicated that if the underlying optimization p. 498). For instance, a human with high SA of an error-prone system
model and optimization procedure can account for the planning situa- might not be sufficiently knowledgeable to correct the error or lack the
tion (as in our experiment), high-performance outcomes can be possible skills needed to trigger that remedy. Due to the good task performance
even in the absence of high participant SA levels. When a planner scores for both dashboards, our results, however, do not indicate a
searches for optimization potential by identifying bottleneck compo- problem with regards to the knowledge or skill set of the participants.
nents within production data, the optimization system leads the parti- Thus, future work could delve deeper into other influencing variables
cipant to certain areas of interest. Such focusing, by definition, removes that might be involved in converting SA into task performance.
awareness of the ongoing production context, decreasing the overall Finally, assessing the SA of operational decision-makers describes a
SA. In conclusion, these cases appear beneficial for task performance as key component to design and evaluate DSS [24]. Comparable with most
long as the employed optimization model and optimization procedures other ergonomic constructs (e.g., mental workload and human error), a
behind the what-if analysis are capable of addressing the occurring plethora of methods for measurement exists [22]. However, studies

Table 8
Summary of hypotheses.
Hypothesis Result

H1: An interactive analytical dashboard design leads to lower situation awareness than an interactive dashboard design without analytical features. Supported
H2: An interactive analytical dashboard design leads to higher task performance than an interactive dashboard design without analytical features. Supported
H3: Situation awareness is associated with higher task performance. Supporteda

a
The SAGAT confirmed the relationship for both designs, whereas the eye-tracking data confirmed this only for the interactive analytical design.

10
M. Nadj, et al. Decision Support Systems 135 (2020) 113322

have not agreed upon which of the techniques available represents the cognitive engagement of an operational decision-maker. Gaze-aware
most appropriate for assessing SA [65]. This study compared two dif- feedback could implement such an audit. This technique has already
ferent measures to assess participants' SA: (i) the established freeze- shown promising improvements in operational decision-makers' visual
probe approach (leveraging SAGAT) and (ii) the rather scarcely ad- scanning behavior. Sharma [67] revealed that gaze-aware feedback
dressed assessment of SA via eye-tracking [22]. The results indicated showed beneficial effects for both a teacher's visual attention allocation
that there was a significant correlation between the constructs em- and students' gain in knowledge in an e-learning context. However,
ployed for both methods (i.e., eye-tracking and SAGAT scores). This despite the promising results using eye-tracking data to assess partici-
finding suggests that those approaches were measuring similar aspects pants' SA, our analysis raised (at least) partial concerns concerning the
of participants' SA during task execution. More interestingly, the ana- validity of eye-tracking data to predict the number of errors.
lyses showed that only the participants' SAGAT scores derived via the An alternative design attempt is to make information more trans-
freeze-probe method generated significant correlations with task per- parent to operational decision-makers. This information should concern
formance for both dashboard designs. Our eye-tracking data only con- the underlying optimization model (e.g., used problem specifications)
firmed the relationship of fixation duration (count) toward errors and optimization procedures (e.g., parameter setting) behind the in-
committed for the interactive analytical dashboard. Hence, the freeze- teractive analytical dashboard. Research reports that the effect on SA
probe technique seems to represent a stronger predictor of task per- was improved to a substantial degree by the transparency of the opti-
formance compared with the eye-tracking assessment of SA. This sug- mization system, thus offering understandability of its actions [54].
gests that the rating type in use appears to be a determinant of the Such transparency can help individuals to understand the cause of ac-
respective measure's predictability. In closing, our analyses supported tions and reduce the downside effects of the out-of-the-loop syndrome.
the usage of the freeze-probe technique to assess SA during planning Finally, training should highlight the significance of frequent visual
tasks. The findings can be contrasted with previous research that con- attention scanning to prevent a loss of SA or skills. In other fields, in-
firmed that SAGAT is one of the most appropriate methods to use when dividuals are required to periodically perform in a manual mode to
the experimenter knows what SA should comprise beforehand [22]. maintain their skills and appropriate SA levels [68]. The measurement
of SA is thus key not only to improve SA-related theory but also to
6.2. Practical implications advance the design of training and evaluation efforts for practitioners.
Hence, scholars demand valid and reliable methods to verify and en-
Next, we translate our findings into implications for designing hance SA theory. System designers require a means to ensure that SA is
dashboards to help practitioners in their efforts to address the danger of increased by new features, GUIs, or training programs [54].
the out-of-the-loop syndrome. First, our analyses showed evidence to
support design efforts that advocate a higher degree of automation 7. Conclusion
within analytical tools. The underlying reasoning is to provide opera-
tional decision-makers relief from their cognitively demanding task by Dashboards are important for DSS as they have a significant impact
introducing more realistic optimization models and more efficient op- on their effectiveness, particularly at the operational level. Our study
timization procedures. This design trend accounts for the challenges objective was to examine the effects of an interactive analytical dash-
decision-makers are facing at the operational level. For instance, they board feature, a what-if analysis. To date, little is understood about its
must make increasingly business-critical decisions in a shorter period influence on human cognitive abilities. We argued that designing such a
with exponentially growing amounts of data. In such situations, op- dashboard feature requires a profound understanding of SA because a
erational decision-makers experience increasingly high mental work- lack of awareness is known to interfere with human information pro-
load in their attempt to process data with their own working memory cessing. Further, it entails downstream effects on the human ability to
[8] – a rather limited, laggard, and error-prone resource [66]. Over the make informed decisions. We created a model that relates a what-if
long term, this design trend drives continuously reducing the human analysis to support SA that, in turn, would positively affect task per-
level of authority in the optimization process. The ultimate goal is to formance. We conducted one large-scale laboratory experiment, in-
diminish the role of human interaction and allow most decision-making cluding eye-tracking and SAGAT data, to study this model from a hol-
procedures to occur automatically in the backend of the system. istic viewpoint.
However, inherent limitations remain on the integration of opti- The significance of our study contributes to DSS literature in several
mization systems into DSS such as dashboards. One issue refers to the ways. Our results indicate that interactive analytical features induce
difficulty of constructing an optimization model that accounts for all significant effects on task performance. We showed that the introduc-
aspects of the real problem (i.e., diversity/ number of constraints). In tion of a what-if analysis could reduce the role of required SA for the
other cases, the problem is not completely specified due to a lack of operational decision-maker. However, such cases appear only beneficial
context knowledge or must be (over-)simplified to be appropriate for for task performance as long as the underlying optimization model and
the computational optimization approach. Other issues relate to the optimization procedures behind the dashboard address the occurring
optimization procedure, which might not be appropriately para- situation. We also reported that the implementation of a what-if ana-
meterized before leveraging the system under real circumstances. lysis is not cost-free and can trigger adverse effects such as a loss of SA.
Furthermore, the performance of the optimization method might not fit This finding confirms the notion of the out-of-the-loop syndrome ap-
the real user requirements. Thus, the importance of having more rea- plying in the APS context and extends it to the dashboard feature level.
listic optimization models and more efficient optimization procedures is In addition, our correlation analyses indicated the expected link of SA
indisputable; however, neither of these currently appears to be suffi- to task performance. Although multiple studies have shown the re-
cient [11]. levance of SA for decision-making, we illustrated that low task perfor-
Second, an alternative approach to interpreting our results might be mance could result from good SA and vice versa. Thus, we believe that
to increase the design efforts to keep the human in the loop of the other influencing variables could be involved in converting SA into
current situation. Such a trend endorses a higher interaction of opera- successful performance that require further assessment. However, how
tional decision-makers with interactive analytical dashboards. The un- SA is assessed is also fundamental because correlations within both
derlying reasoning is to realize an effective human-computer interac- dashboard alternatives differed according to the measurements con-
tion but simultaneously consider the cognitive abilities of individuals. cerning the link of SA to task performance. The SAGAT scores con-
Our analyses showed that the degree of SA human beings possess de- firmed this relationship for both alternatives, whereas the eye-tracking
fines a relevant factor in such a system design. One promising design data showed significant correlations only for the interactive analytical
attempt might be to introduce a periodic SA audit to increase the design. We, therefore, provided evidence that using SAGAT seems most

11
M. Nadj, et al. Decision Support Systems 135 (2020) 113322

appropriate. [21] D. Golightly, J.R. Wilson, E. Lowe, S. Sharples, The role of situation awareness for
Although our results appear promising, some limitations require understanding signalling and control in rail operations, Theor. Issues Ergon. Sci. 11
(2010) 84–98.
discussion. We used graduate students as proxies for planners. Thus, [22] P.M. Salmon, N.A. Stanton, W.G.H. D, D. Ladva Jenkins, L. Rafferty, M. Young,
other studies could verify how professionals interact with the proposed Measuring Situation Awareness in complex systems: Comparison of measures study,
designs. We studied the effects of a what-if analysis, which we expected Int. J. Ind. Ergon. 39 (2009) 490–500.
[23] M.L. Resnick, Situation awareness applications to executive dashboard design,
to play a key role in the creation and reduction of SA. Future work Hum. Factors Ergon. Soc. Annu. Meet. Proc. 47 (2003) 449–453.
might consider further interactive analytical features, such as inter- [24] C.D. Wickens, Situation awareness: review of Mica Endsley’s 1995 articles on si-
active multi-objective optimization [12]. Other approaches involve the tuation awareness theory and measurement, Hum. Factors 50 (2008) 397–403.
[25] M.R. Endsley, Toward a theory of situation awareness in dynamic systems, Hum.
user at different points in time (i.e., user feedback on candidate solu- Factors 37 (1995) 32–64.
tions vs. on intermediate results), leverage the feedback loops with the [26] H. Stadtler, Production planning and scheduling, in: H. Stadtler, C. Kilger, H. Meyr
user for different parts of the preference model (i.e., adjustments of the (Eds.), Supply Chain Manag. Adv. Plan, Springer, Berlin, Heidelberg, 2015, pp.
195–211.
optimization model vs. optimization procedures), and occasionally in-
[27] P.Y. Wu, S. Acharya, Visualizing Capacity and Load: A Production Planning
clude preference learning. It might be interesting to compare in- Information System for Metal Ingot Casting, in: Proc, Conf. Inf. Syst. Appl. Res,
dividuals' SA and task performance scores obtained with those features. CONISAR Proceedings, Wilmington, 2011, pp. 1–6.
[28] P. Zhang, Visualizing production planning data, IEEE Comput. Graph. Appl. 16
(1996) 7–10.
Declarations of Competing Interest [29] P. Salmon, N. Stanton, G. Walker, D. Green, Situation awareness measurement: a
review of applicability for C4i environments, Appl. Ergon. 37 (2006) 225–238.
None. [30] M.G. Calvo, Working memory and inferences: evidence from eye fixations during
reading, Memory. 9 (2001) 365–381.
[31] T.O. Meservy, M.L. Jensen, K.J. Fadel, Evaluation of competing candidate solutions
Funding in electronic networks of practice, Inf. Syst. Res. 25 (2014) 15–34.
[32] A. Gupta, H. Li, R. Shardac, Should I send this message? Understanding the impact
of interruptions, social hierarchy and perceived task complexity on user perfor-
This research did not receive any specific grant from funding mance and perceived workload, Decis. Support. Syst. 55 (2013) 135–145.
agencies in the public, commercial, or not-for-profit sectors. [33] T. Bucher, A. Gericke, S. Sigg, Process-centric business intelligence, Bus. Process.
Manag. J. 15 (2009) 408–429.
[34] H. Chen, R.H.L. Chiang, V.C. Storey, Business intelligence and analytics: from big
Acknowledgement data to big impact, MIS Q. 36 (2012) 1165–1188.
[35] H.J. Watson, Tutorial: business intelligence - past, present, and future, Commun.
We thank Tobias Hochstatter for his support. Assoc. Inf. Syst. 25 (2009) 487–510.
[36] W.H. Inmon, Building the Data Warehouse, John Wiley & Sons, 1996.
[37] A. Trigo, F. Belfo, R.P. Estébanez, Accounting information systems: the challenge of
References the real-time reporting, Procedia Technol. 16 (2014) 118–127.
[38] T. Kawamoto, B. Mathers, Key success factors for a performance dashboard, DM
[1] E. Dong, H. Du, L. Gardner, An interactive web-based dashboard to track COVID-19 Rev. 17 (2007) 20–21.
in real time, Correspondence. 20 (2020) 533–534. [39] A. Miller, J. Cioffi, Measuring marketing effectiveness and value: the Unisys mar-
[2] K. Gander, Here Are 6 Coronavirus Dashboards Where You Can Track the Spread of keting dashboard, J. Advert. Res. 44 (2004) 237–243.
COVID-19 Live Online, Newsweek, https://www.newsweek.com/coronavirus- [40] N.H. Rasmussen, M. Bansal, C.Y. Chen, Business Dashboards: A Visual Catalog for
tracking-maps-1491705, (2020) (accessed May 9, 2020). Design and Deployment, John Wiley & Sons, Hoboke, 2009.
[3] S. Few, Information Dashboard Design, O’Reilly Media, Inc., The Effective Visual [41] Y. Wind, Marketing as an engine of business growth: a cross-functional perspective,
Communication of Data, 2006. J. Bus. Res. 58 (2005) 863–873.
[4] O.M. Yigitbasioglu, O. Velcu, A review of dashboards in performance management: [42] D. Reibstein, D. Norton, Y. Joshi, P. Farris, Marketing Dashboards: A Decision
implications for design and research, Int. J. Account. Inf. Syst. 13 (2012) 41–59. Support System for Assessing Marketing Productivity, in: INFORMS Mark, Sci. Conf,
[5] S. Negash, P. Gray, Business Intelligence, F. Burstein (Ed.), Int. Handbooks Inf. Syst. INFORMS, 2005.
Handb. Decis. Support Syst, 2 Heidelberg, Springer, Berlin, 2008, pp. 175–193. [43] B.H. Clark, A.V. Abela, T. Ambler, Behind the wheel, Mark. Manag. 15 (2006)
[6] J. Kohlhammer, D.U. Proff, A. Wiener, Visual Business Analytics - Effektiver Zugang 18–23.
zu Daten und Informationen, dpunkt Verlag GmbH, Heidelberg, (2013). [44] G.K. DeBusk, R.M. Brown, L.N. Killough, Components and relative weights in uti-
[7] W.W. Eckerson, Performance Dashboards: Measuring, Monitoring and Managing lization of dashboard measurement systems like the balanced scorecard, Br.
Your Business, John Wiley & Sons, Hoboke, 2006. Account. Rev. 35 (2003) 215–231.
[8] F.J. Lerch, D. Harter, Cognitive support for real-time dynamic decision making, Inf. [45] C. Heaps, S. Handel, Similarity and features of natural textures, J. Exp. Psychol.
Syst. Res. 12 (2001) 63–82. Hum. Percept. Perform. 25 (1999) 299–320.
[9] K. Pauwels, T. Ambler, B.H. Clark, P. LaPointe, D. Reibstein, B. Skiera, B. Wierenga, [46] F. Heylighen, The growth of structural and functional complexity during evolution,
T. Wiesel, Dashboards as a service: why, what, how, and what research is needed? J. in: F. Heylighen, D. Aerts (Eds.), Evol, Complex, Dordrecht, 1997, pp. 1–18.
Serv. Res. 12 (2009) 175–189. [47] E.R. Tufte, The Visual Display of Quantitative Information, Graphics press,
[10] V.A. Zeithaml, R.N. Bolton, J. Deighton, T.L. Keiningham, K.N. Lemon, Cheshire, 2006.
J.A. Petersen, Forward-looking focus: can firms have adaptive foresight? J. Serv. [48] T.S. Amer, S. Ravindran, The effect of visual illusions on the graphical display of
Res. 9 (2006) 168–183. information, J. Inf. Syst. 24 (2010) 23–42.
[11] D. Meignan, S. Knust, J.-M. Frayret, G. Pesant, N. Gaud, A review and taxonomy of [49] W. Cleveland, Visualizing Data, Hobart Press, New Jersey, 1993.
interactive optimization methods in operations research, ACM Trans. Interact. [50] R. Benayoun, J. de Montgolfier, J. Tergny, O. Laritchev, Linear programming with
Intell. Syst. 5 (2015) 1–46. multiple objective functions: step method (stem), Math. Program. 1 (1971)
[12] K. Miettinen, F. Ruiz, A.P. Wierzbicki, Introduction to multiobjective optimization: 366–375.
Interactive approaches, in: J. Branke, K. Deb, K. Miettinen, R. Słowiński (Eds.), [51] H.S. Vitense, J.A. Jacko, V.K. Emery, Multimodal feedback: an assessment of per-
Multiobjective Optim. Lect. Notes Comput. Sci. Springer, Berlin, Heidelberg, 2008, formance and mental workload, Ergonomics. 46 (2003) 68–87.
pp. 27–58. [52] A. Olivia, Gist of the scene, in: L. Itti, G. Rees, J.K. Tsotsos (Eds.), Encycl. Neurobiol.
[13] F.D. Davis, J.E. Kottemann, W.E. Remus, What-if analysis and the illusion of con- Atten, Elsevier Science, San Diego, 2005, pp. 251–256.
trol, Proc. Hawaii Int. Conf. Syst. Sci., IEEE Comput. Soc. Press, 1991, pp. 452–460. [53] M.R. Endsley, Automation and situation awareness, in: R. Parasuraman,
[14] R. Shibl, M. Lawley, J. Debuse, Factors influencing decision support system ac- M. Mouloua (Eds.), Autom. Hum. Perform. Theory Appl, Lawrence Erlbaum
ceptance, Decis. Support. Syst. 54 (2013) 953–961. Associates, Inc, New Jersey, 1996, pp. 163–181.
[15] M.R. Endsley, E.O. Kiris, The out-of-the-loop performance problem and level of [54] M.R. Endsley, From here to autonomy: lessons learned from human–automation
control in automation, Hum. Factors 37 (1995) 381–394. research, Hum. Factors 59 (2017) 5–27.
[16] I. Benbasat, A.S. Dexter, Individual differences in the use of decision support aids, J. [55] J.H. Roberts, J.M. Lattin, Development and testing of a model of consideration set
Account. Res. 20 (1982) 1–11. composition, J. Mark. Res. 28 (1991) 429–440.
[17] R. Sharda, S.H. Barr, J.C. McDonnell, Decision support system effectiveness: a re- [56] B. Gawronski, L. Creighton, Dual process theories, in: D.E. Carlston (Ed.), Oxford
view and an empirical test, Manag. Sci. 34 (1988) 139–159. Handb. Soc. Cogn. Oxford University Press, New York, 2013, pp. 282–312.
[18] J. Fripp, How effective are models? Omega. 13 (1985) 19–28. [57] C.E.J. Hartel, K. Smith, C. Prince, Defining Aircrew Coordination: Searching
[19] M.D. Goslar, G.I. Green, T.H. Hughes, Decision support systems: an empirical as- Mishaps for Meaning, in: Proc, Int. Symp. Aviat. Psychol, Columbus, 1991.
sessment for decision making, Decis. Sci. 17 (1986) 79–91. [58] T. Hill, P. Lewicki, Statistics: Methods and Applications: A Comprehensive
[20] D. Manzey, J. Reichenbach, L. Onnasch, Human performance consequences of au- Reference for Science, Industry, and Data Mining, StatSoft, Tulsa, 2006.
tomated decision aids: the impact of degree of automation and system experience, J. [59] E. Katok, Using laboratory experiments to build better operations management
Cogn. Eng. Decis. Mak. 6 (2012) 57–87. models, Technol. Inf. Oper. Manag. 5 (2011) 1–86.
[60] J. David Xu, I. Benbasat, R.T. Cenfetelli, The nature and consequences of trade-off

12
M. Nadj, et al. Decision Support Systems 135 (2020) 113322

transparency in the context of recommendation agents, MIS Q. 38 (2014) 379–406. Interaction. He has three years of industry experience in BIA and Business Process
[61] G.E. Bolton, A. Ockenfels, U.W. Thonemann, Managers and students as news- Management solutions from SAP SE. His research has been published in leading inter-
vendors, Manag. Sci. 58 (2012) 2225–2233. national journals and conferences, such as the Journal of Cognitive Engineering and
[62] R.J. Jacob, K. Karn, Eye tracking in human-computer interaction and usability re- Decision Making, ACM Conference on Human Factors in Computing Systems,
search: Ready to deliver the promises, in: J. Hyona, R. Radach, H. Deubel (Eds.), International Conference on Information Systems, European Conference on Information
Mind’s Eye Cogn. Appl. Asp. Eye Mov. Res, Elsevier Science, Oxford, 2003, pp. Systems, and International Conference on Information Systems and Neuroscience.
573–605.
[63] S.G. Hart, L.E. Staveland, Development of NASA-TLX (task load index): results of Alexander Maedche is a professor at the Karlsruhe Institute of Technology (KIT) in
empirical and theoretical research, in: P.A. Hancock, N.B.T. A, P. Meshkati (Eds.), Germany. His research work is positioned at the intersection of Information Systems (IS)
Hum. Ment. Workload, North Holland Press, Amsterdam, 1988, pp. 139–183. and Human-Computer Interaction. Specifically, he investigates novel concepts of inter-
[64] N.B. Sarter, R.J. Mumaw, C.D. Wickens, Pilots’ monitoring strategies and perfor- active intelligent systems for enterprises and the society. He publishes in leading IS and
mance on automated flight decks: an empirical study combining behavioral and Computer Science journals such as the Journal of the Association of Information Systems,
eye-tracking data, Hum. Factors 49 (2007) 347–357. Decision Support Systems, Business & Information Systems Engineering, Information &
[65] J. Patrick, N. James, A. Ahmed, P. Halliday, Observational assessment of situation Software Technology, Computers & Human Behavior, International Journal of Human-
awareness, team differences and training implications, Ergonomics. 49 (2006) Computer Studies, IEEE Intelligent Systems, VLDB Journal, and AI Magazine.
393–417.
[66] D.D. Woods, E.S. Patterson, E.M. Roth, Can we ever escape from data overload? A Christian Schieder is a professor at the Technical University of Applied Sciences
cognitive systems diagnosis, Cogn. Tech. Work 4 (2002) 22–36. Amberg-Weiden in Germany. His research focuses on Business Intelligence & Analytics
[67] K. Sharma, H.S. Alavi, P. Jermann, P. Dillenbourg, A Gaze-Based Learning Analytics (BIA), Enterprise Decision Management and Business Rules Technologies. His work has
Model: In-Video Visual Feedback to Improve learner’s Attention in MOOCS, in: appeared in several conferences and journals including the HMD - Praxis der
Proc, ACM Press, New York, Int. Conf. Learn. Anal. Knowl, 2016, pp. 417–421. Wirtschaftsinformatik, Hawaii International Conference on System Sciences, and
[68] E.L. Wiener, Cockpit automation, in: E.L. Wiener, D.C.B.T. HF, A. Nagel (Eds.), European Conference on Information Systems among others. Models and techniques de-
Hum. Factors Aviat, Academic, San Diego, 1988, pp. 433–461. veloped by him have been incorporated in several dynamic pricings systems. For several
years he held the position of a Chief Digital Officer at a large multi-national machinery
Mario Nadj is a post-doctoral researcher at the Karlsruhe Institute of Technology (KIT) in company where he was responsible for the digitalization strategy and the data-driven
Germany. His research interests include the fields of Business Intelligence & Analytics solution business.
(BIA), Machine Learning, Physiological Computing, as well as Human-Computer

13