HYPOTHESIS AND THEORY

published: 10 September 2019

doi: 10.3389/frobt.2019.00082

Immersive Analytics: Theory and

Research Agenda
Richard Skarbez*† , Nicholas F. Polys, J. Todd Ogle, Chris North and Doug A. Bowman
Center for Human-Computer Interaction, Virginia Tech, Blacksburg, VA, United States

Advances in a variety of computing fields, including “big data,” machine learning,

visualization, and augmented/mixed/virtual reality, have combined to give rise to the
emerging field of immersive analytics, which investigates how these new technologies
support analysis and decision making. Thus far, we feel that immersive analytics research
has been somewhat ad hoc, possibly owing to the fact that there is not yet an organizing
framework for immersive analytics research. In this paper, we address this lack by
Edited by: proposing a definition for immersive analytics and identifying some general research
Mark Billinghurst, areas and specific research questions that will be important for the development of
University of South Australia, Australia
this field. We also present three case studies that, while all being examples of what
Reviewed by:
Bruce Hunter Thomas,
we would consider immersive analytics, present different challenges, and opportunities.
University of South Australia, Australia These serve to demonstrate the breadth of immersive analytics and illustrate how the
Victoria Interrante, framework proposed in this paper applies to practical research.
University of Minnesota Twin Cities,
United States Keywords: immersive analytics, visual analytics, immersion, virtual reality, visualization, sensemaking, knowledge
Carlos Andújar, generation
Universitat Politecnica de
Catalunya, Spain

*Correspondence: 1. INTRODUCTION
Richard Skarbez
[email protected] We are living and working in the era of “big data,” according to Kurose and Marzullo (2016).
† Present
Information such as online activity, news media, health records, social media posts, geolocations,
address:
Richard Skarbez,
and networks of authors are all tracked, collected, aggregated, and stored. But it is not enough
Department of Computer Science and to have the data; the data must be analyzable to make it actionable. This paper explores ways
Information Technology, La Trobe that information visualization, machine learning, and virtual environments can come together to
University, Melbourne, VIC, Australia support analysis of big data. Specifically, we address the multiplicity of ways these fields combine
to support immersive analytics.
Specialty section: There are two distinctly different—but complementary—approaches to big data analytics
This article was submitted to (Bertini and Lalanne, 2009). First, human analysts can sift through the data. Based on expertise,
Virtual Environments,
experience, and intuition, the best analysts can synthesize disparate information into cohesive
a section of the journal
hypotheses. Interactive visualization helps analysts view, organize, and synthesize the data
Frontiers in Robotics and AI
(Van Wijk, 2005). But limitations in human capacity, plus the sheer volume of data, make human-
Received: 18 May 2018
only analysis intractable for many problems at scale. The second approach is to make use of
Accepted: 19 August 2019
machine intelligence, through data mining and machine learning algorithms, to forage for patterns
Published: 10 September 2019
and insights in huge datasets that would be overwhelming for human analysts. This approach has
Citation:
been very successful, but primarily for well-defined problems (Lazer et al., 2009). When it comes
Skarbez R, Polys NF, Ogle JT, North C
and Bowman DA (2019) Immersive
to sensemaking tasks requiring human intuition and pattern recognition (or a deep understanding
Analytics: Theory and Research of semantics), a combined approach is needed (Crouser and Chang, 2012; Counts et al., 2014). The
Agenda. Front. Robot. AI 6:82. varying ways these approaches can be combined are discussed at greater length in section 3 and
doi: 10.3389/frobt.2019.00082 in Figure 2.

Frontiers in Robotics and AI | www.frontiersin.org 1 September 2019 | Volume 6 | Article 82

Skarbez et al. Immersive Analytics: Theory and Research Agenda

The knowledge generation process of Pirolli and Card (2005) Interactive visual analytics recognizes the limitations of these
conceptually models how raw data is converted into a theory two approaches, and proposes a hybrid approach in which
by human analysts via a series of analytical steps. These steps human analysts are assisted by machine intelligence (Cook and
form two bidirectional loops: a foraging loop for uncovering new Thomas, 2005; Keim et al., 2008). However, it is an open
data, and a sensemaking loop for synthesizing that information question as to what form this human-machine collaboration
into structured hypotheses (see Figure 1). By iterating the loops, is to take. Immersive analytics situates this collaboration
analysts “incrementally formalize” their understanding of the in an immersive virtual reality (VR) or augmented reality
data (Shipman et al., 1995). A goal of many visual analytics (AR) context.
applications is to support user sensemaking processes with as With the advent of low-cost, high-quality VR/AR systems,
unobtrusive a design as possible. Note that Pirolli and Card many researchers are thinking of using such systems for visual
use the term “sense making” to refer to both the overall analytics (dubbed immersive analytics by Chandler et al., 2015).
process, as well as just the latter part of the process. To However, what is usually meant by “immersive analytics” is
avoid confusion, we use the term knowledge generation to refer the visualization of abstract datasets in an immersive 3D
to the overall process, and reserve the term sensemaking for environment, as in Bowman et al. (2003), Henry and Polys
the subprocess. (2010), Hossain et al. (2012), Bacim et al. (2013), Radics et al.
Recent trends have turned toward “human in the loop” (2015), Kwon et al. (2016), and Cordeil et al. (2017a). In this sort
analytics, a paradigm by which analysts provide feedback to of immersive analytics, visualizations are designed in advance,
analytical models in order to steer the computational path and users primarily have the goal of foraging: examining the
of those models (Andrews et al., 2010; Endert et al., 2012a, dataset visually to find items, clusters, or trends of interest.
2014). Leman et al. (2013) proposed a more user-centered In this paper, we suggest a definition of immersive
variant, “human is the loop,” in which the focus lies on analytics that goes beyond traditional visualization and
models assisting users in context of their sensemaking process, is more consistent with the approaches used in visual
rather than users assisting models. While the foraging portion analytics. Based on this definition, we propose a research
of the sensemaking process has gained much support from agenda for this emerging field. Finally, we illustrate the
computational algorithms, the synthesis portion of the process challenges and opportunities of immersive analytics with three
has remained largely the domain of human abductive reasoning. case studies.

FIGURE 1 | Knowledge generation loop (inspired by Pirolli and Card, 2005). The knowledge generation process can be organized into two main subprocesses: a
foraging loop that seeks and organizes evidence, and a sensemaking loop that develops a mental model that fits the evidence.

Frontiers in Robotics and AI | www.frontiersin.org 2 September 2019 | Volume 6 | Article 82

Skarbez et al. Immersive Analytics: Theory and Research Agenda

2. WHAT IS IMMERSIVE ANALYTICS? • Capability to render existing channels of sensory output with
greater fidelity (as in Polys et al., 2016)
In Chandler et al.’s foundational paper, they describe immersive
analytics as follows: − “Large” visual display (substantially more than 30◦ FoV)
− “Surrounding” visual display (i.e., head-worn displays,
“Immersive Analytics investigates how new interaction and CAVE-like (Cruz-Neira et al., 1993) displays)
display technologies can be used to support analytical reasoning − Higher-resolution visual display (e.g., “retina” resolution)
and decision making. The aim is to provide multi-sensory − Stereoscopic visual display
interfaces for analytics approaches that support collaboration − Spatialized sound display
and allow users to immerse themselves in their data. Immersive
Analytics builds on technologies such as large touch surfaces, • Support for additional types or channels of user input
immersive virtual and augmented reality environments, − Voice input
haptic and audio displays, and modern fabrication techniques − Touch input
(Chandler et al., 2015).”
− Physiological sensing
− Tracking
This provides a clear view of the field of study and its major
components, but is somewhat resistant to further analysis. In ◦ Head tracking
the interest of informing our research agenda for immersive ◦ Hand tracking
analytics, we propose an alternative definition. Specifically, we ◦ Full-body tracking
define immersive analytics as the science of analytical reasoning ◦ Eye tracking
facilitated by immersive human-computer interfaces. In this, we
• Support for more natural interaction techniques
follow Cook and Thomas (2005), who defined visual analytics
as “the science of analytical reasoning facilitated by interactive − Body-based navigation (i.e., physical head turning,
visual interfaces.” That said, what is meant by the “science of walking, crouching)
analytical reasoning?” And which human-computer interfaces − Natural manipulation (e.g., directly
are considered “immersive?” touching/grasping/moving data with the hand)
We first consider analytical reasoning. This term is quite − Conversational speech interfaces
broad; one can easily imagine analytical reasoning that consists − Gestural interfaces
of pure thought, without the aid of any technology at all. For our
purposes, we will restrict our consideration to computer-aided Having now explicated both the “immersive” and “analytics”
analytical reasoning. Regarding analytical reasoning itself, we parts of immersive analytics, we now revisit our definitions:
consider it to be practically equivalent to knowledge generation, • Immersive analytics is the science of analytical reasoning
as formulated by Pirolli and Card (2005) and addressed in facilitated by immersive human-computer interfaces.
section 1. We therefore propose that computer-aided analytical • By analytical reasoning, we specifically refer to computer-aided
reasoning involves computer assistance in either or both of the analytical reasoning as a partner with the human; that is,
foraging and sensemaking processes. a process of foraging and sensemaking where part or all of
We now turn our attention to immersive human-computer the foraging and/or sensemaking processes are performed in
interfaces. We follow Slater and others in defining immersion as cooperation with a computer.
an objective characteristic of a computer system, and specifically, • By immersive human-computer interfaces, we specifically mean
as the set of valid actions supported by a the system, as seen those interfaces which enable a user to interact with a
in Slater (1999) and further explicated in Skarbez et al. (2017). system using additional or more-immersive displays and user
Here, a valid action is defined as an any action a user can take interface techniques.
that causes in a change to the internal state of the system (an
effective valid action) or to the presentation of the system state
(a sensorimotor valid action). 3. RESEARCH AGENDA
From here, one can see that when we refer to immersive
human-computer interfaces, what we really mean is more In section 1, we commented on the technological trends that
immersive human-computer interfaces, as all computing systems are combining to give rise to immersive analytics at this
that enable user input and output have some degree of point in time. However, it can be reasonably argued that
immersion. There are many ways in which a system can be the fact that immersive analytics is now practical does not
more immersive than such a typical workstation, as discussed in automatically make it a good idea or a productive research
Bowman and McMahan (2007). Some specific technologies that direction. The onus falls on those of us in this field to
would make a system more immersive include: justify the utility of immersive analytics to analysts with
problems to solve. To do otherwise risks putting the cart before
• Support for additional channels of sensory output
the horse.
− Haptic display There are some good reasons to believe that immersive
− Olfactory display analytics can indeed offer significant benefits to data analysts,
− Other sensory displays beyond those offered by more traditional analytics applications.

Frontiers in Robotics and AI | www.frontiersin.org 3 September 2019 | Volume 6 | Article 82

Skarbez et al. Immersive Analytics: Theory and Research Agenda

Bowman and McMahan (2007) identified several potential • One attempt at such a taxonomy appears in Figure 2.
benefits of increased immersion, including increased spatial This taxonomy is based on the definition of immersive
understanding due to integration of more depth cues, decreased analytics put forward in section 2; a different definition
information clutter due to increased effective display size, would likely result in a different taxonomy. The space
peripheral awareness, and information bandwidth. The results we choose to partition is the space of computer-aided
of the study performed by Gruchalla (2004) provide evidence in analytical reasoning; the dimensions are foraging (from
support of these hypotheses. In his study, sixteen participants entirely computer-driven foraging to entirely human-
performed a complicated spatial task–oil well path planning– driven foraging) and hypothesis generation (from entirely
using both a desktop workstation with a stereoscopic display computer-driven to entirely human-driven). Notably,
and a CAVE-like (Cruz-Neira et al., 1993) immersive virtual this partitioning does not specifically take into account
environment. Fifteen of the sixteen participants performed immersion, and as such, applies to visual analytics more
the task faster in the more-immersive condition; this was broadly. (One could consider the immersion of data
accompanied by a statistically significant increase in correct presentation as a third dimension).
solutions identified in the more-immersive condition. We have
• Given such a taxonomy, can it be productively applied to
also suggested (section 4.2) that immersive analytics applications
the existing literature? Some research questions that might
can provide extensive “space to think,” mimicking the ways in
follow are: Are there categories that are over- (or under-)
which analysts naturally use physical space to organize their
represented? Are there categories which are particularly
analysis. Evidence indicates that performance on analytics tasks
effective for specific problem domains?
improves both quantitatively and qualitatively when physically
• One of the questions facing any analytics application is how
larger displays are used (Andrews et al., 2010). This extends the
to distribute the subtasks of analysis among machine(s) and
results from Ball et al. (2007), where the authors observed several
their human user(s). Can an effective partitioning be predicted
benefits of using physical locomotion (such as walking toward or
based on characteristics of the specific task or dataset? For
away from the screen) rather than virtual locomotion (zooming
example, can cognitive task analysis be productively applied
using abstract user interface techniques) with large displays.
to distribute those tasks?
Finally, we hypothesize that immersive analytics systems can
• Can machine algorithms learn useful behavior from observing
take advantage of rich semantic interaction (Endert et al., 2012a)
human user interactions with the system? Can they learn
to infer analyst intent and help guide intelligent algorithms
semantic relationships (e.g., spatial grouping)? Can this be
that support the analysis. While these results do not prove
enhanced by enabling more immersive interactions (e.g.,
that immersive analytics is the preferred solution for every
positional and/or eye tracking)? It would seem that the more
analysis application, they at least suggest that it is worthy of
user interactions that can be captured, the more that could be
further investigation.
learned from them.
To aid in this investigation, this section outlines a research
agenda for immersive analytics, based on the definition and
3.2. Evaluating the Utility of Immersion
discussion in section 2. We generalize and describe five
Beyond an appropriate combination of human and machine
areas of research:
intelligences, by our definition, immersive analytics involves
• Combining Human and Computer Intelligence the use of immersive technologies. This naturally leads to
• The Utility of Immersion the question of benefits of immersion: Why should immersive
• Designing Immersive Analytics Systems technologies be used for visual analytics, and in what ways might
• Facilitating Collaboration through Immersion they be superior to or preferred over other systems?
• Changing the Process of Analysis with Immersion. In prior work, Bowman and McMahan (2007) have argued
that immersion is not an all-or-nothing concept—that is,
there is no sharp distinction between “immersive” and “non-
immersive,” but rather a continuum of levels of immersion.
3.1. Combining Human and Computer In addition, immersion is multi-dimensional—many objective
Intelligence characteristics of a system contribute to an increased level of
In section 1, we suggested that a core premise of visual (and immersion. We have divided these into factors related to the
immersive) analytics is that a combination of human and fidelity of displays (e.g., visual field of view, haptic resolution),
machine intelligence may be able to achieve insights that could the fidelity of interaction (e.g., biomechanical similarity of the
not be reached by either one alone. Arguably, then, the central interaction technique to the corresponding real-world action),
question for immersive analytics is: How can machine intelligence and the fidelity of the scenario (e.g., the realism of the physics
and human intelligence be most productively combined to address engine’s output).
analytics tasks? With these concepts in mind, several key questions can be
Given that question, we propose specific research questions to asked about immersive analytics:
support that line of inquiry:
• When does increased immersion lead to greater effectiveness,
• How can we systematically describe and classify use cases for where effectiveness can be defined in terms of measurable
Immersive Analytics? user experience outcomes like task performance, learning,

Frontiers in Robotics and AI | www.frontiersin.org 4 September 2019 | Volume 6 | Article 82

Skarbez et al. Immersive Analytics: Theory and Research Agenda

FIGURE 2 | (a) The “space” of (immersive) analytics, inspired by the definition proposed in section 2. (b) One possible partitioning of the space, based on the amount
of human processing involved in its collection. (c) Another possible partitioning of the space, based on whether the knowledge generation is being done primarily by
humans or algorithms. (d) A partitioning of the space resulting from the combination of (b,c), populated with analytic processes that illustrate the different categories.

engagement, adoption, or satisfaction? Which components of process in useful ways? See section 4.2 for more detailed
immersion are key predictors of effectiveness? discussion of this idea.
• For a given task, how much immersion is needed to realize • In what ways do “natural” interaction techniques (those
measurable benefits over traditional systems? most similar to real-world interaction) contribute to
• How are the effects of immersion (and its components) improved effectiveness? Is cognitive load reduced by using
moderated or mediated by the analysis task, the characteristics such techniques?
of the user, or the characteristics of the dataset?
In addition, the concept of immersion is related to many other
important research questions: 3.3. Designing Immersive Analytics
• What effect does multi-modal output have? Can other (non- Systems
visual) sensory modalities be effectively used to “display” Section 3.2 considered immersion largely at the level of
additional channels of information? Alternatively, can other technology choice. For example, does a display with a greater
(non-visual) channels be productively used to redundantly field of regard result in improved outcomes? Here we consider
code information, enabling, for example, improved situational how more immersive technologies affect the design of analytics
awareness? Are there other techniques for multi-modal applications. For example, given a display with a large field of
data display that have not yet been considered? Can regard, how should interfaces and interactions be designed to
limitations in the fidelity of one display modality be maximize its utility? Or, more broadly: How should we design
compensated by another? immersive user interfaces (UIs) and user experience (UX) to
• What sorts of immersive technologies are most maximize the potential of immersive analytics?
appropriate/effective for immersive analytics? For example, is Some analytics applications require the integration of multiple
AR preferred to VR? Are head-worn or world-fixed displays types of data in the same workspace to be maximally productive.
superior? Do we need room-scale tracking, or is it sufficient to For the case of intelligence analysis, consider text documents,
work in a desk-scale space? building plans (whether maps or 3D models), still photographs,
• Does “space to think” (Andrews et al., 2010) still work security cam footage, recordings of telephone conversations—all
in immersive virtual environments? In other words, does of which might be presented as geolocated data. Questions central
increased virtual space—provided by large-scale tracking, to this research agenda include:
high-resolution displays, and usable spatial interaction • How can we define and determine appropriate spatial
techniques—enable analysts to externalize their cognitive metaphors for different types of data?

Frontiers in Robotics and AI | www.frontiersin.org 5 September 2019 | Volume 6 | Article 82

Skarbez et al. Immersive Analytics: Theory and Research Agenda

− How can all this data be organized so as to be “readily − How best to notify the user that the machine has provided
at hand?” new information?
− Can characteristics of the data be effectively mapped onto
characteristics of the environment itself? 3.4. Facilitating Collaboration Through
− How best to support spatial references for users, so they Immersion
don’t “get lost” in the space? Much of the preceding discussion has focussed on the potential
• Which components of immersion can best be leveraged benefits of immersion for a single user of an analytics system.
for analytics tasks? Some investigations along these lines However, many analysis tasks involve—or could potentially
include Laha et al. (2012) (head tracking, field of regard, involve—multiple users. In Chandler et al’s description of
and stereoscopic rendering), Bacim et al. (2013) (stereoscopy, immersive analytics, presented at the start of section 2, they
head-based rendering, and display area), Laha et al. (2014) specifically chose to mention as an aim of such technologies
(field of regard, stereoscopy, and head tracking), and Iyer et al. “support[ing] collaboration.” Following their example, we ask:
(2017) (stereoscopy and field of regard). How can immersive technologies facilitate new or improved modes
• Given knowledge about human perception and ergonomics, of collaboration?
how can we improve data representation and composition in
• Certainly when compared to a typical workstation, physically
immersive analytic environments? Some relevant work can be
found in Bowman et al. (2003), Polys and Bowman (2004), large displays such as the CAVE and CAVE2 can better
support multiple simultaneous users. However, such displays
Polys et al. (2007), and Polys et al. (2011).
are generally only capable of generating perspectively correct
− How should one present documents/data suggested imagery for a single user. Some early results from Cordeil et al.
by machine intelligence (rather than resulting from (2017b) suggest that networked HWDs—which do not share
user action) into the user’s workspace? Should there this limitation—may enable equally effective collaboration, at
be a distinction? least under some circumstances.
− How can we design selection and manipulation techniques
− How should user interfaces be designed to display
to work within and across multiple media types inside a
the needed information in multiple-HWD collaborative
virtual environment?
scenarios? In Cordeil et al., they visualized each user’s view
• What are appropriate travel and navigation techniques for frustum and fingertip in the other user’s display. Is this
abstract data spaces? sufficient for other tasks?
− Cordeil et al. explored a two-user collaborative
− If one assumes that at least some analysts will be scenario in which users were tasked with evaluating
using immersive technologies in “deskVR” configurations network connectivity. Do the techniques they
(discussed in Zielasko et al., 2017), what are appropriate describe generalize to more users or to other types
travel/navigation techniques that require minimal or no of collaborative work?
additional tracking/user exertion?
− Alternatively, if one assumes that at least some analysts • How can immersive technologies better enable physically
will have access to tracked spaces, is real walking still the distributed collaboration? For example, it seems that a system
best travel technique for abstract spaces? Do its benefits that was able to track and log the focus of a user’s attention
outweigh its drawbacks? and their interactions with a virtual environment could
reproduce many of the benefits of colocated collaboration for
• What new collaboration models are enabled and/or required users that are physically distributed but working in the same
by the use of immersive technologies? virtual space.
• Similarly, can immersive technologies better enable temporally
− Should computational models and algorithms be
distributed (asynchronous) collaboration? If one can track,
represented to the user as assistants/partners/collaborators,
log, and transmit information about a user’s attention
as tools, or as something else entirely? (Consider Polys
and actions, it seems that one would also be able to
et al., 2015).
replay that data.
− Is there any benefit to having them “reside” in the same
• How can we enable collaboration among devices with
space as the user?
heterogeneous capabilities? For example, immersive devices
− Should the human user(s) explicitly interact
may include large, installation-type devices such as the CAVE
with machine intelligence? What modalities of
and CAVE2, single-user VR HWDs connected to powerful
interaction are most natural and effective (e.g., voice,
PCS such as the HTC Vive and Oculus Rift, and single-user
gestures, text)?
standalone HWDs such as the Oculus Quest. While we would
− Furthermore, is there any benefit to having machine
consider all of these to be “immersive,” they have widely
intelligence be represented by social actors? What effects
varying capabilities.
will avatar representation, character, and trust have on the
analytic outcome? − Are there applications for which heterogeneous
− How does immersion better enable interaction with collaboration is advantageous? Should different user
machine assistance/intelligence (semantic interaction)? roles be mapped onto devices with different capabilities?

Frontiers in Robotics and AI | www.frontiersin.org 6 September 2019 | Volume 6 | Article 82

Skarbez et al. Immersive Analytics: Theory and Research Agenda

Consider the possibility of an immersive analytics the value of this research agenda by exploring how it
“operating theater,” where novice analysts can watch how applies to applications from the past, present, and future of
an expert interacts with the data. immersive analytics. We begin by setting out three examples
of such applications: Information-Rich Virtual Environments,
3.5. Changing the Analytical Process With Immersive Space to Think, and Immersive Archaeology. We
Immersion continue by exploring how these examples have generated
Highly immersive systems (which employ technologies such results that inform this research agenda, or highlighting how
as those listed in section 2) enable the presentation of and these projects will seek to answer questions proposed in
interaction with information in ways that have never before the research agenda.
been possible. It stands to reason, then, that we should carefully Note that we would consider all of these examples of
consider the new ways that users might be affected by the use of “computer-assists-human analytics” (Figure 2D, above
such systems. For example, Milk (2015) referred to virtual reality center). This is not entirely unreasonable, given both our
as “the ultimate empathy machine.” While one can rightly regard research expertise and the current focus of the immersive
this claim with skepticism, it is clear that immersive analytics analytics literature, which is heavily informed by visualization.
offer a user experience that is substantially different from that Nevertheless, the class of “human-assists-computer analytics” is
of traditional analytics systems. Therefore, one might ask: How likely to only grow in prevalence and importance going forward,
might the use of immersive analytics systems (as opposed to other and we consider this a very important area for research in
media) affect the procedures, presentations, or products of analysis this field.
itself? There are at least two fertile areas for investigation:
• What are the effects of immersion on the analytic process? 4.1. Immersive Analytics and
Information-Rich Virtual Environments
− How is decision making affected by the medium of
presentation or the design of the immersive representation? (IRVEs)
(see Tversky and Kahneman, 1974; Evans, 1989; Evans and An essential property of immersive analytics is that information
Stanovich, 2013). and interactions are spatialized in 3D—they exist inside a live,
− Is presentation/dissemination in immersive displays “more interactive 3D environment. Like our “old-school” paper file
powerful” than traditional methods? If so, what are the cabinets or piled desk, or the data mountains of Robertson et al.
associated practical or ethical concerns? (see Polys et al., (1998), we use space to store our data and organize our cognition.
2017, 2018). Since the oral tradition began, spatialization has aided human
− Is it appropriate to use VR as an “empathy machine” in the memory and recall. This also seems to be the case in virtual
context of immersive analytics? reality, as observed by Ragan et al. (2012) and Mann et al. (2017).
So we return to our research agenda asking, “How do
• How can immersive analytics be used to improve rigor and we spatialize information to improve the design of immersive
rationality of analysis? analytics systems?” (section 3.3) One possible combination of
visualization and virtual reality research is Information-Rich
− How can we incorporate formal methods into immersive
Virtual Environments (IRVEs), which studies how perceptual
analytics interfaces? For example, can we explicitly
environments are augmented, or enhanced, with abstract
require analysis of competing hypotheses, as described in
information, including text, visualizations, and multimedia. The
Heuer (1999)?
nuances of the problem were originally identified by Bowman
− Can immersive analytics provide methods and training that
et al. (1999) and Bowman et al. (2003) also articulated a research
improve critical thinking skills?
agenda that identified IRVEs as fruitful direction of study. Chen
− How do we design immersive analytics systems that help
et al. (2004) examined text labels specifically. As the variety of
to identify and mitigate bias and prejudice in analysis and
cases began to multiply, a typology of IRVE presentation was
decision-making?
proposed in Polys (2006) and Polys et al. (2011) that investigated
− Can immersive analytics be used to generate a verifiable
the tradeoff between association and occlusion across different
(or at least externally reviewable) trace of an analyst’s
layout techniques.
thought processes?
Thus, IRVEs are a kind of immersive analytics. They enable
− Can intermediate products of analysis be
users to access and apprehend additional information about the
meaningfully preserved and annotated for
environment and its objects. Properties and attributes can be
educational/training/museum applications?
represented in the space and interactivity can support search,
comparison, and pattern recognition tasks across information
4. IMMERSIVE ANALYTICS RESEARCH: types, scales, and distances. Figure 3 shows a variety of layout
APPLICATIONS AND EXAMPLES spaces (coordinate systems) where interactive information might
live: object, viewport, and display for example. This work
In the previous sections, we have proposed a definition illustrates that we can (and perhaps should) rethink our ways of
for immersive analytics and set out a research agenda designing immersive analytics workspaces and media as we move
for this field. In this section, we hope to make concrete beyond the WIMP paradigm as Behr and Reiners (2008) suggest.

Frontiers in Robotics and AI | www.frontiersin.org 7 September 2019 | Volume 6 | Article 82

Skarbez et al. Immersive Analytics: Theory and Research Agenda

FIGURE 3 | IRVE layout spaces provide support for immersive analytics through displays and interactions in (a) object space, (b) viewport space, and
(c) display space.

Indeed, many of the UI and UX research questions listed in space and time. This project enabled the discovery of the fact
section 3.3 above apply to the visual representations and semantic that persistence of the Epstein-Barr virus—over 30 years—
interaction in an IRVE (discussed in detail by Polys and Bowman, is likely due to its “hiding” in the bloodstream, not the
2004; Bowman et al., 2006; Polys et al., 2007, 2011). lymphatic tissue.
By spatializing visualization and interaction in a virtual
environment, IRVEs support immersive analytics. The PathSim 4.1.1. IRVE Research Questions
(short for Pathogen Simulation) application (described by Polys The information visualization question of “How much
et al., 2004; Duca et al., 2007; Shapiro et al., 2008 and shown information can you pack into a view?” led to a related
in Figure 4) is an example of how immersive analytics can IRVE question, “How much information can be packed into
be used to explore and understand the dynamics of agent- an immersive display?” Early work by Ni et al. (2006) showed
based simulations. Scientists can use PathSim to examine the promising results for larger display sizes, and work by Bacim
types and numbers of agents (such as T-cells, B-cells, and et al. (2013) showed benefits resulting from increased immersion.
virus particles), as well as their states and interactions across Polys et al. (2016) proposed an analytic model to assess the ideal

Frontiers in Robotics and AI | www.frontiersin.org 8 September 2019 | Volume 6 | Article 82

Skarbez et al. Immersive Analytics: Theory and Research Agenda

FIGURE 4 | Spatialized information representations across scales in PathSim.

capacity (bandwidth) of a display that uses visual angle and driven by the goal of improving bandwidth and throughput
pixels per angle to put mobiles, desktops, HMDs, tiled displays, between human and computer. There are still many open
and CAVE-like displays onto the same scale. Focusing on the research questions for immersive analytics IRVEs. For example,
criterion of legibility, Iyer et al. (2017) then tested our model we expect future IRVE experiments to explore the themes
by investigating how text can scale between tiled HD displays identified in section 3, such as the effective spatialization of
and Head-Worn Displays. The analytic model based on pixels multimedia and linked information, the effect of information
suggested a 7x advantage of a projection-based CAVE over an layouts (perception and association) in recall and inference, and
HMD. Experimental results revealed that for text, there was a the role of display and interaction fidelity in task performance.
4x advantage. These experiments highlight our user-centered
approach to quantifying the benefits of immersive analytics 4.2. Immersive Analytics and “Space to
platforms and techniques, and as such, provide substantial Think”
insight regarding the questions raised in sections 3.2 and 3.3. Prior studies, including those by Andrews et al. (2011), have
While it is clear from other VR research that immersive demonstrated the important roles that physical space plays in
context can change a person’s cognition—from memory (Mann human knowledge generation. For example, studies of the use of
et al., 2017) to spatial judgements (Laha and Bowman, 2012) large high-resolution displays to support knowledge generation
to procedural training (Sowndararajan et al., 2008) to racial from large collections of textual documents found that the
bias (Groom et al., 2009)—the IRVE research we describe physical space afforded a form of distributed cognition. Analysts
here explores the specific contributions of pre-attentive cues naturally externalized their cognitive processes into the space,
(for example the depth and Gestalt cues presented by an offloading cognition by spatially organizing the textual materials
IRVE layout) and how the user perceives these given the into visual hypothesis structures via embodied interactions, with
display. Such perceptual cues drive our pre-attentive judgements less reliance on summarization methods such as note taking.
and associations between informational elements—both objects The ability to physically navigate the space afforded efficient
and annotations. information access and recall, leading to increased information
We advocate a deliberate approach to the design of integrated synthesis. Furthermore, these behaviors were hampered when
information spaces that is based on empirical evidence and the physical navigation in a large display space was replaced by

Frontiers in Robotics and AI | www.frontiersin.org 9 September 2019 | Volume 6 | Article 82

Skarbez et al. Immersive Analytics: Theory and Research Agenda

FIGURE 5 | “Space to think” on a large 2D display (Andrews and North, 2012).

FIGURE 6 | In StarSpire, a cluster created by the analyst (Left) is filled with more relevant documents by the system (Right) (Bradel et al., 2014).

virtual navigation in a small display space (i.e., zoom and pan, be developed that observe this cognitive activity, learn from it,
controlled by a mouse). Andrews et al. (2010) referred to this and respond to it to help the analyst more efficiently conduct
phenomenon as “space to think”; this was previously discussed by the activity. Thus, the “space to think” becomes an efficient
Kirsh (1995), and explored further by Andrews and North (2013). medium of communication between human cognition and
While these examples of “space to think” are 2D, one can analytical algorithm.
imagine that an immersive 3D space in which to think and Semantic interaction exploits this fact to directly support
work might be an interesting approach to immersive analytics. In human sensemakers by recasting the analyst’s interactions into
section 3.2, we asked whether the “space to think” approach still analytical model parameter updates. Our work on ForceSpire
works in immersive virtual environments. This question hints and StarSpire (Figure 6) supports text analysis scenarios on large
at the larger idea that the existing forms of immersive analytics displays: as users organize documents in the space, the system
(immersive data visualizations and IRVEs) may not represent the learns a term importance model, enabling algorithms to forage
only ways, or even the best ways, to take advantage of immersive for additional relevant documents and synthesize them on the
technologies for data analytics. screen within the user’s organized structure. We refer to this as
An important result observed by Andrews and North (2012) “synthesis driven foraging.” The system learns from the user’s
regarding the “space to think” phenomenon is that the human activity in document positioning, highlighting, annotating, and
analyst’s cognition becomes partially visible (Figure 5) (Earlier searching. This approach shields the analysts from the usability
work by Zhang and Norman, 1994 discussed this effect in problems associated with directly manipulating algorithmic
distributed cognitive tasks). That is, as the analyst organizes models, such as the problem of premature formality. It exploits
information in the space, it reveals clues about the synthesis interactions that the analyst would perform anyway, even without
activities occurring within the analyst’s cognitive process. Thus, the presence of the algorithms, to help the analyst’s knowledge
machine learning algorithms and intelligent user interfaces can generation. As a result, human and machine learn together,

Frontiers in Robotics and AI | www.frontiersin.org 10 September 2019 | Volume 6 | Article 82

Skarbez et al. Immersive Analytics: Theory and Research Agenda

FIGURE 7 | Conceptual illustration of Immersive Space to Think (IST). A human analyst uses immersive space to organize her thinking about a complex set of
documents and data. The system observes interaction to recognize clusters created by the user, understand what is relevant to the user, and infer the user’s
hypotheses. Based on these semantic interactions, underlying machine intelligence algorithms suggest new data to consider, summarize data, and label data (e.g.,
with estimated credibility).

FIGURE 8 | Screenshots of a prototype IST application that allows users to organize and annotate documents and regions of space in immersive VR.

supporting the process of incremental formalism in knowledge opportunities for semantic interaction to guide machine learning
generation. Results from a series of studies showed that the user algorithms and improve analytic outcomes. We call this approach
and system reached a common mental/machine model, and that Immersive Space to Think (IST). A conceptual illustration of IST
algorithmic support freed up the analyst to focus on higher- is shown in Figure 7, and an early prototype is shown in Figure 8.
level knowledge generating activity (Endert et al., 2012a,b, 2013;
Bradel et al., 2014). 4.2.1. IST Research Questions
In our current work, we hypothesize that immersive VR can Through the development and evaluation of IST, we hope to
be used to provide a more expressive, expansive space to think contribute knowledge that is relevant to our broader research
during analytic synthesis, and that immersive systems offer new agenda. The research plan for IST substantially overlaps with

Frontiers in Robotics and AI | www.frontiersin.org 11 September 2019 | Volume 6 | Article 82

Skarbez et al. Immersive Analytics: Theory and Research Agenda

all four research areas described in section 3. As mentioned by Benko, Ishak, and Feiner facilitated certain analyses “in
above, work by Endert et al. (2012b) suggests that the rich seconds, when normally [they] can take hours to complete”
user interaction data available in immersive systems can be (Benko et al., 2004).
used to drive algorithmic analysis in a richer way than non- We envision several ways that immersive analytics research
immersive input an; we intend to investigate how these data can could actually change the analytical process for archaeologists
be used to enhance machine understanding of human analytic (section 3.5). First, 3D scans and models can be experienced at
processes. Meanwhile, the primary goal of these project is to their true size, with a “God’s-eye” view, or with a zoomed-in
investigate whether the “space to think” concept of Andrews et al. detail view; the relationships of rooms and artifacts and models
(2010) still applies in immersive workspaces; we also hope to can become an embodied perception. Also analytics can be
generate results regarding whether increased immersion leads situated in the present time where augmented or virtual reality
to greater effectiveness more generally, and how these effects technology could overlay historical findings on the landscape as
might be moderated by the specific characteristics of the task, it appears today. This would enable researchers to revisit past
user, and data. Other questions from the research agenda that excavations and data at the human scale with the benefit of
might be informed by this research project include: What are the new knowledge and new technology. Another research thrust in
appropriate spatial metaphors for different types of data? (section immersive analytics for archaeology would be to integrate search
3.3). How can immersive analytics be used to improve the rigor and/or machine assistance tools, so that researchers could access
and rationality of analysis? (section 3.5). external data from within the immersive analysis environment.
For example, an archaeologist discovers a sword that they believe
4.3. Immersive Analytics for Archaeology to be evidence that a site traded with a particular culture.
Archaeology is sometimes called the “science of destruction,” They could then search for—or have automatically provided—
because through the very process of excavation, dig sites are other examples of swords from that culture, in order to test
disturbed to the point that reconstruction is impossible. The their hypothesis.
archaeologist in the field is privileged to the moment of discovery, Figure 9 presents several stages of the proposed work process,
and although detailed documentation of the process preserves culminating in a shared IVE used for education. The figures
much data that can inform interpretation, the site in its original also illustrate the machine intelligence software that can be used
context is forever lost as a result of the excavation process. to view additional details and corroborating evidence regarding
Perhaps because of this, digital documentation of excavations, in some aspect of the dig. Providing that structure data is captured
the form of 3D scanning, is becoming increasingly commonplace, in the field (via laser scan or photogrammetry) at a typical dig site
and even presenting these digital dig sites using immersive (Figures 9A,B), the opportunity exists for rapid processing such
technology has been explored (Acevedo et al., 2001; Benko et al., that field scientists could review daily point cloud records with a
2004; Forte and Kurillo, 2010; Petrovic et al., 2011). Despite this, scientist in the lab or otherwise off-site (Figure 9C). The product
“making full use of [large digital datasets for archaeology] remains of this work would then be made available as an immersive
a challenge” [emphasis added] (Petrovic et al., 2011). learning environment for varying levels of students as well as
We believe that one of the reasons for this has been the informal learning in museums (Figure 9D).
unavailability of suitable hardware. As these datasets are generally
3D, large, and detailed, they are perhaps ideally suited for analysis 4.3.1. Immersive Archaeology Research Questions
using immersive display and interaction technologies, such as Immersive Archaeology has the potential to change the way
VR. With the advent of high-quality consumer-level hardware archaeology research is conducted, or, at least, to change the way
such as the HTC Vive and Oculus series of head-worn displays, archaeological data is recorded and reported. Making effective
this may no longer be a significant barrier. use of existing and yet-to-be-captured spatial and temporal
So on the one hand, both technology improvements and datasets, though, presents a significant UI/UX design challenge;
updated work practices are combining to drive a “data avalanche” new interaction techniques and interfaces will need to be
Petrovic et al. (2011) of high-quality 3D datasets. On the developed, which will certainly inform and be informed by
other hand, technology improvements and market pressures are the research questions proposed in section 3.3. Observations
combining to drive a rapid increase in quality and availability from the development and use of immersive analytical tools for
(with a simultaneous decrease in cost) of immersive displays. archaeological research will certainly inform the questions in
It seems to us that there is a natural symbiosis here, such that section 3.5; we will have the opportunity to observe in real time
immersive VR is uniquely well-suited to serve the needs of the impact of immersive analytics tools on a field of research.
domain scientists in archaeology. One specific question that will Immersive Archaeology will
That said, much archaeological evidence remains 2D, and have to address is whether and how to visualize data uncertainty.
often analog: Photographs, Harris matrices, sketches, field notes, Digital representations of archaeological sites and artifacts, while
etc. There is reason to believe that situating these data in the constantly improving, inherently introduce some measurement
spatial context of a 3D reconstruction of the dig site can be uncertainty. Concern over the presentation of such data is not
a significant aid to analysis: Post-excavation analysis, typically new, and was formally addressed in the London Charter for
the most time-consuming phase of the archaeology process, the Computer-based Visualization of Cultural Heritage (Denard,
can consume many months and sometimes years of work, 2012). It will also be important to quickly determine the
as discussed by Forte et al. (2012), but the system described provenance of any given digital object. For example, a scene may

Frontiers in Robotics and AI | www.frontiersin.org 12 September 2019 | Volume 6 | Article 82

Skarbez et al. Immersive Analytics: Theory and Research Agenda

FIGURE 9 | (A) Conventional archaeological fieldwork, with an archaeologist physically present at the excavation site. (B) Conventional fieldwork, with the addition of
3D scanning (LIDAR, photogrammetry, etc.) to document the ongoing excavation. (C) An archaeologist viewing an immersive virtual reconstruction of the excavation
at a particular stage, made possible via the 3D scanning illustrated in (B). Note that the system is presenting additional information about a particular artifact, possibly
including high-resolution photographs, 3D scans, field notes, corroborating evidence from the Internet, etc. (D) Immersive archaeology applied to an educational
setting.

contain data that is directly captured from 3D scanning, but mass market. There are some differences among these devices,
also artist’s renditions of what the environment may have looked but in general, their immersion profiles are the very similar:
like, or objects that are based on real scans, but from a different Head-worn, with stereo imagery provided via one display per eye,
physical location or time. All of these data sources have their and two hand-held controllers/wands, all of which have 6-DOF
utility in reconstructions of historical sites, but it will be necessary tracking. This platform has notable strengths (stereoscopy, full-
to readily distinguish one from another. field of regard, 6-DOF tracked viewpoint and hand positions) as
well as weaknesses (limited display resolution, limited capacity
for symbolic input, limited capacity for local collaboration). This
5. CONCLUSION will likely be the immersive platform of choice for the near future,
as most users do not have the space or cost budget for installation-
In this article, we have explored and defined the research area scale immersive systems such as the CAVE (Cruz-Neira et al.,
posed by “immersive analytics.” We defined immersive analytics 1993) or CAVE2 (Febretti et al., 2013). Research regarding the
as the science of analytical reasoning facilitated by immersive immersive characteristics of such head-worn systems is likely to
human-computer interfaces. A critical result of our exploration pay immediate dividends.
is a further elucidation of these terms in order that we may more The opportunities and challenges outlined in this paper will
meaningfully discuss what does (and what does not) fall into the keep researchers and practitioners busy for years to come. As new
realm of immersive analytics. We presented and discussed several display and interaction modalities evolve, some questions will
research thrusts that are implied by this definition, and posed change or perhaps become more interesting. Still, we believe that
a set of specific research questions for the immersive analytics the essential human-computer interaction issues we enumerate
community. We also discussed three applications (from the 1990s here for immersive analytics (the interactive partnership between
to ongoing research) that we believe fall under the heading of human and computer intelligence, the value of immersive
immersive analytics research, illustrating the breadth of the area. technology for new insights and productivity, designing the
In doing so, we hope to provide both clarity and inspiration to immersive analytic user experience, augmenting cognition to
the immersive analytics research community. combat bias) will be the basis for research that could profoundly
While we have briefly discussed a number of immersive impact the way we as humans reason about complex questions.
system characteristics, including the list presented in section 2,
some immersive systems are more promising and relevant than
others. Today, we are seeing a rapid increase in quality and AUTHOR CONTRIBUTIONS
availability, with a corresponding decrease in cost, for six-degree-
of-freedom (6-DOF) head-worn displays. Devices including the RS, NP, JO, CN, and DB contributed to the conception and
Oculus Rift, Rift S, and Quest, as well as the HTC Vive and Vive discussion of this research project and participated in the
Pro, are making highly immersive technology available to the discussions that ultimately led to the creation of this manuscript.

Frontiers in Robotics and AI | www.frontiersin.org 13 September 2019 | Volume 6 | Article 82

Skarbez et al. Immersive Analytics: Theory and Research Agenda

NP and DB performed or supervised the original research FUNDING

regarding IRVEs. CN and DB performed or supervised the
original research regarding Immersive Space to Think. RS and JO This work on Immersive Space to Think was partially funded by
developed the research plan regarding Immersive Archaeology. the Data and Decisions destination area at Virginia Tech.

REFERENCES Cordeil, M., Bach, B., Li, Y., Wilson, E., and Dwyer, T. (2017a). “Design
space for spatio-data coordination: tangible interaction devices for immersive
Acevedo, D., Vote, E., Laidlaw, D. H., and Joukowsky, M. S. (2001). information visualisation,” in 2017 IEEE Pacific Visualization Symposium
“Archaeological data visualization in VR:analysis of lamp finds at the (PacificVis) (Seoul), 46–50.
Great Temple of Petra, a case study,” in Proceedings of the Conference on Cordeil, M., Dwyer, T., Klein, K., Laha, B., Marriott, K., and Thomas, B. H.
Visualization’01 (San Diego, CA: IEEE Computer Society), 493–496. (2017b). Immersive collaborative analysis of network connectivity: CAVE-
Andrews, C., Endert, A., and North, C. (2010). Space to think: large high-resolution style or head-mounted display? IEEE Trans. Vis. Comput. Graph. 23, 441–450.
displays for sensemaking,” in CHI ’10, Proceedings of the SIGCHI Conference on doi: 10.1109/TVCG.2016.2599107
Human Factors in Computing Systems (New York, NY: ACM), 55–64. Counts, S., De Choudhury, M., Diesner, J., Gilbert, E., Gonzalez, M., Keegan, B.,
Andrews, C., Endert, A., Yost, B., and North, C. (2011). Information visualization et al. (2014). “Computational social science: Cscw in the social media era,”
on large, high-resolution displays: issues, challenges, and opportunities. Inf. Vis. in CSCW Companion ’14 Proceedings of the Companion Publication of the
10, 341–355. doi: 10.1177/1473871611415997 17th ACM Conference on Computer Supported Cooperative Work & Social
Andrews, C., and North, C. (2012). “Analyst’s workspace: an embodied Computing (New York, NY: ACM), 105–108.
sensemaking environment for large, high-resolution displays,” in 2012 IEEE Crouser, R. J., and Chang, R. (2012). An affordance-based framework for human
Conference on Visual Analytics Science and Technology (VAST) (Seattle, WA), computation and human-computer collaboration. IEEE Trans. Vis. Comput.
123–131. Graph. 18, 2859–2868. doi: 10.1109/TVCG.2012.195
Andrews, C., and North, C. (2013). The impact of physical navigation on spatial Cruz-Neira, C., Sandin, D. J., and DeFanti, T. A. (1993). “Surround-screen
organization for sensemaking. IEEE Trans. Vis. Comput. Graph. 19, 2207–2216. projection-based virtual reality: The design and implementation of the
doi: 10.1109/TVCG.2013.205 CAVE,” in SIGGRAPH ’93, Proceedings of the 20th annual conference
Bacim, F., Ragan, E., Scerbo, S., Polys, N. F., Setareh, M., and Jones, B. D. (2013). on Computer graphics and interactive techniques (New York, NY: ACM),
“The effects of display fidelity, visual complexity, and task scope on spatial 135–142.
understanding of 3d graphs,” in GI ’13, Proceedings of Graphics Interface 2013 Denard, H. (2012). “A new introduction to the london charter,” in Paradata
(Toronto, ON: Canadian Information Processing Society), 25–3. and Transparency in Virtual Heritage, Digital Research in the Arts and
Ball, R., North, C., and Bowman, D. A. (2007). “Move to improve: promoting Humanities Series, eds A. Bentkowska-Kafel, D. Baker, and H. Denard
physical navigation to increase user performance with large displays,” in CHI (Farnham: Ashgate Publishing), 57–71.
’07, Proceedings of the SIGCHI Conference on Human Factors in Computing Duca, K. A., Shapiro, M., Delgado-Eckert, E., Hadinoto, V., Jarrah,
Systems (New York, NY: ACM), 191–200. A. S., Laubenbacher, R., et al. (2007). A virtual look at Epstein–
Behr, J., and Reiners, D. (2008). “Class notes: don’t be a WIMP,” in SIGGRAPH ’08, Barr virus infection: biological interpretations. PLoS Pathog. 3:e137.
ACM SIGGRAPH 2008 Classes (New York, NY: ACM). doi: 10.1371/journal.ppat.0030137
Benko, H., Ishak, E. W., and Feiner, S. (2004). “Collaborative mixed reality Endert, A., Bradel, L., and North, C. (2013). Beyond control panels: Direct
visualization of an archaeological excavation,” in Proceedings of the 3rd manipulation for visual analytics. IEEE Comput. Graph. Appl. 33, 6–13.
IEEE/ACM international Symposium on Mixed and Augmented Reality doi: 10.1109/MCG.2013.53
(Arlington, VA: IEEE Computer Society), 132–140. Endert, A., Fiaux, P., and North, C. (2012a). Semantic interaction for
Bertini, E., and Lalanne, D. (2009). “Surveying the complementary role of sensemaking: Inferring analytical reasoning for model steering. IEEE
automatic data analysis and visualization in knowledge discovery,” in Trans. Vis. Comput. Graph. 18, 2879–2888. doi: 10.1109/TVCG.
Proceedings of the ACM SIGKDD Workshop on Visual Analytics and Knowledge 2012.260
Discovery: Integrating Automated Analysis with Interactive Exploration (New Endert, A., Fiaux, P., and North, C. (2012b). “Semantic interaction for visual text
York, NY: ACM), 12–20. analytics,” in CHI ’12, Proceedings of the SIGCHI Conference on Human Factors
Bowman, D. A., Chen, J., Wingrave, C. A., Lucas, J. F., Ray, A., Polys, N. F., et al. in Computing Systems (New York, NY: ACM), 473–482.
(2006). New directions in 3d user interfaces. Int. J. Virt. Real. 5, 3–14. Available Endert, A., Hossain, M. S., Ramakrishnan, N., North, C., Fiaux, P., and Andrews,
online at: http://www.ijvr.org/web/search/singleArticle/228 C. (2014). The human is the loop: new directions for visual analytics. J. Intell.
Bowman, D. A., Hodges, L. F., Allison, D., and Wineman, J. (1999). The Inform. Syst. 43, 411–435. doi: 10.1007/s10844-014-0304-9
educational value of an information-rich virtual environment. Pres. Teleop. Evans, J. S., and Stanovich, K. E. (2013). Dual-process theories of higher
Virt. Environ. 8, 317–331. doi: 10.1162/105474699566251 cognition: advancing the debate. Perspect. Psychol. Sci. 8, 223–241.
Bowman, D. A., and McMahan, R. P. (2007). Virtual reality: how much immersion doi: 10.1177/1745691612460685
is enough? Computer 40, 36–43. doi: 10.1109/MC.2007.257 Evans, J. S. B. (1989). Bias in Human Reasoning: Causes and consequences.
Bowman, D. A., North, C., Chen, J., Polys, N. F., Pyla, P. S., and Yilmaz, U. (2003). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
“Information-rich virtual environments: theory, tools, and research agenda,” in Febretti, A., Nishimoto, A., Thigpen, T., Talandis, J., Long, L., Pirtle, J. D., et al.
VRST ’03, Proceedings of the ACM Symposium on Virtual Reality Software and (2013). “Cave2: a hybrid reality environment for immersive simulation and
Technology (New York, NY: ACM), 81–90. information analysis,” in SPIE Proceedings Vol. 8649: The Engineering Reality
Bradel, L., North, C., House, L., and Leman, S. (2014). “Multi-model semantic of Virtual Reality, eds M. Dolinsky, I. E. McDowall (Bellingham, WA: SPIE)
interaction for text analytics,” in 2014 IEEE Conference on Visual Analytics Forte, M., Dell’Unto, N., Issavi, J., Onsurez, L., and Lercari, N. (2012).
Science and Technology (VAST) (Paris), 163–172. 3D archaeology at Çatalhöyük. Int. J. Herit. Digit. Era 1, 351–378.
Chandler, T., Cordeil, M., Czauderna, T., Dwyer, T., Glowacki, J., Goncu, C., doi: 10.1260/2047-4970.1.3.351
et al. (2015). “Immersive analytics,” in 2015 Big Data Visual Analytics (BDVA), Forte, M., and Kurillo, G. (2010). “Cyberarchaeology: experimenting with
(Hobart, TAS: IEEE), 1–8. teleimmersive archaeology,” in 2010 16th International Conference on Virtual
Chen, J., Pyla, P. S., and Bowman, D. A. (2004). “Testbed evaluation of navigation Systems and Multimedia (Seoul), 155–162.
and text display techniques in an information-rich virtual environment,” in Groom, V., Bailenson, J. N., and Nass, C. (2009). The influence of racial
IEEE Virtual Reality 2004 (Chicago, IL), 181–289. embodiment on racial bias in immersive virtual environments. Soc. Influence.
Cook, K. A., and Thomas, J. J. (Eds.). (2005). Illuminating the Path: The 4, 231–248. doi: 10.1080/15534510802643750
Research and Development Agenda for Visual Analytics. Los Alamitos, CA: IEEE Gruchalla, K. (2004). “Immersive well-path editing: investigating the added value
Computer Society. of immersion,” in Proceedings-IEEE Virtual Reality 2004 (Chicago, IL), 157–164.

Frontiers in Robotics and AI | www.frontiersin.org 14 September 2019 | Volume 6 | Article 82

Skarbez et al. Immersive Analytics: Theory and Research Agenda

Henry, J. A., and Polys, N. F. (2010). The effects of immersion and navigation on Polys, N. F. (2006). Display Techniques in Information-Rich Virtual Environments
the acquisition of spatial knowledge of abstract data networks. Proc. Comput. (Ph.D. thesis). Blacksburg, VA: Virginia Tech.
Sci. 1, 1737–1746.doi: 10.1016/j.procs.2010.04.195 Polys, N. F. and Bowman, D. A. (2004). Design and display of enhancing
Heuer R. J. Jr. (1999). “Analysis of competing hypotheses,” in Psychology of information in desktop information-rich virtual environments:
Intelligence Analysis, ed H. Appelbaum (Washington, DC: Center for the challenges and techniques. Virt. Real. 8, 41–54. doi: 10.1007/s10055-004-
Study of Intelligence), 95–110. Available onine at: https://www.cia.gov/ 0134-0
library/center-for-the-study-of-intelligence/csi-publications/books-and- Polys, N. F., Bowman, D. A., and North, C. (2011). The role of depth and gestalt
monographs/psychology-of-intelligence-analysis/art11.html cues in information-rich virtual environments. Int. J. Hum Comput. Stud. 69,
Hossain, M. S., Akbar, M., and Polys, N. F. (2012). Narratives in the network: 30–51. doi: 10.1016/j.ijhcs.2010.05.007
interactive methods for mining cell signaling networks. J. Comput. Biol. 19, Polys, N. F., Bowman, D. A., North, C., Laubenbacher, R., and Duca, K. (2004).
1043–1059. doi: 10.1089/cmb.2011.0244 “Pathsim visualizer: an information-rich virtual environment framework for
Iyer, J., Polys, N. F., and Arsenault, L. (2017). “Text density and display bandwidth: systems biology,” in PWeb3D ’04, Proceedings of the Ninth International
evaluating scalability by model and experiment,” in Web3D ’17, Proceedings Conference on 3D Web Technology (New York, NY: ACM), 7–14.
of the 22nd International Conference on 3D Web Technology (New York, NY: Polys, N. F., Kim, S., and Bowman, D. A. (2007). Effects of information layout,
ACM). screen size, and field of view on user performance in information-rich
Keim, D. A., Mansmann, F., Schneidewind, J., Thomas, J., and Ziegler, H. virtual environments. Comput. Anim. Virt. Worlds 18, 19–38. doi: 10.1002/
(2008). “Visual analytics: scope and challenges,” in Visual Data Mining: Theory, cav.159
Techniques and Tools for Visual Analytics, eds S. J. Simoff, M. H. Böhlen, and A. Polys, N. F., Knapp, B., Bock, M., Lidwin, C., Webster, D., Waggoner, N.,
Mazeika (Berlin; Heidelberg: Springer), 76–90. et al. (2015). “Fusality: an open framework for cross-platform mirror world
Kirsh, D. (1995). The intelligent use of space. Artif. Intell. 73, 31–68. installations,” in Proceedings of the 20th International Conference on 3D Web
doi: 10.1016/0004-3702(94)00017-U Technology (New York, NY: ACM), 171–179.
Kurose, J., and Marzullo, K. (2016). The Federal Big Data Research and Radics, P. J., Polys, N. F., Neuman, S. P., and Lund, W. H. (2015). “OSNAP!
Development Strategic Plan. Technical Report, The Networking and Introducing the open semantic network analysis platform,” in Visualization
Information Technology Research and Development Program. and Data Analysis 2015 (Las Vegas, NV: International Society for Optics and
Kwon, O. H., Muelder, C., Lee, K., and Ma, K. L. (2016). A study of layout, Photonics).
rendering, and interaction methods for immersive graph visualization. IEEE Ragan, E. D., Bowman, D. A., and Huber, K. J. (2012). Supporting cognitive
Trans. Vis. Comput. Graph. 22, 1802–1815. doi: 10.1109/TVCG.2016.2520921 processing with spatial information presentations in virtual environments. Virt.
Laha, B., and Bowman, D. A. (2012). “Identifying the benefits of immersion Real. 16, 301–314. doi: 10.1007/s10055-012-0211-8
in virtual reality for volume data visualization,” in Immersive Visualization Robertson, G., Czerwinski, M., Larson, K., Robbins, D. C., Thiel, D., and van
Revisited Workshop of the IEEE VR conference (Orange, CA), 1–2. Dantzich, M. (1998). “Data mountain: using spatial memory for document
Laha, B., Bowman, D. A., and Socha, J. J. (2014). Effects of vr system fidelity on management,” in UIST ’98, Proceedings of the 11th Annual ACM Symposium
analyzing isosurface visualization of volume datasets. IEEE Trans. Vis. Comput. on User Interface Software and Technology (New York, NY: ACM), 153–162.
Graph. 20, 513–522. doi: 10.1109/TVCG.2014.20 Shapiro, M., Duca, K. A., Lee, K., Delgado-Eckert, E., Hawkins, J., Jarrah, A. S., et al.
Laha, B., Sensharma, K., Schiffbauer, J. D., and Bowman, D. A. (2012). Effects of (2008). A virtual look at Epstein–Barr virus infection: Simulation mechanism.
immersion on visual analysis of volume data. IEEE Trans. Vis. Comput. Graph. J. Theor. Biol. 252, 633–648. doi: 10.1016/j.jtbi.2008.01.032
18, 597–606. doi: 10.1109/TVCG.2012.42 Shipman, III, F. M., Marshall, C. C., and Moran, T. P. (1995). “Finding and using
Lazer, D., Pentland, A. S., Adamic, L., Aral, S., Barabasi, A. L., et al. (2009). Life implicit structure in human-organized spatial layouts of information,” in CHI
in the network: the coming age of computational social science. Science 323, ’95, Proceedings of the SIGCHI Conference on Human Factors in Computing
721–723. doi: 10.1126/science.1167742 Systems (New York, NY: ACM), 346–353.
Leman, S. C., House, L., Maiti, D., Endert, A., and North, C. (2013). Skarbez, R., Brooks, F. P. Jr., and Whitton, M. C. (2017). A survey of presence and
Visual to parametric interaction (v2pi). PLoS ONE 8:e50474. related concepts. ACM Comput. Surv. 50:96. doi: 10.1145/3134301
doi: 10.1371/journal.pone.0050474 Slater, M. (1999). Measuring presence: a response to the witmer and
Mann, J., Polys, N., Diana, R., Ananth, M., Herald, B., and Platel, S. (2017). singer presence questionnaire. Pres. Teleop. Virt. Environ. 8, 560–565.
“Virginia Tech’s study hall: a virtual method of loci mnemotechnic study using doi: 10.1162/105474699566477
a neurologically-based, mechanism-driven, approach to immersive learning Sowndararajan, A., Wang, R., and Bowman, D. A. (2008). “Quantifying the benefits
research,” in 2017 IEEE Virtual Reality (VR) (Los Angeles, CA), 383–384. of immersion for procedural training,” in Proceedings of the 2008 Workshop on
Milk, C. (2015). “How virtual reality can create the ultimate empathy machine,” Immersive Projection Technologies/Emerging Display Technologiges (New York,
in TED2015 (Vancouver, BC). Available online at: https://en.unesco.org/ NY: ACM).
interculturaldialogue/resources/276 (accessed May 17, 2018). Tversky, A., and Kahneman, D. (1974). Judgment under uncertainty: heuristics and
Ni, T., Bowman, D. A., and Chen, J. (2006). “Increased display size and resolution biases. Science 185, 1124–1131. doi: 10.1126/science.185.4157.1124
improve task performance in information-rich virtual environments,” in GI ’06, Van Wijk, J. J. (2005). “The value of visualization,” in IEEE Visualization, 2005
Proceedings of Graphics Interface 2006 (Toronto, ON: Canadian Information (Minneapolis, MN: IEEE), 79–86.
Processing Society),139–146. Zhang, J. and Norman, D. A. (1994). Representations in distributed cognitive tasks.
Petrovic, V., Gidding, A., Wypych, T., Kuester, F., DeFanti, T., and Levy, Cognit. Sci. 18, 87–122. doi: 10.1207/s15516709cog1801_3
T. (2011). Dealing with archaeology’s data avalanche. Computer44, 56–60. Zielasko, D., Weyers, B., Bellgardt, M., Pick, S., Meibner, A., Vierjahn, T., et al.
doi: 10.1109/MC.2011.161 (2017). “Remain seated: towards fully-immersive desktop VR,” in 2017 IEEE
Pirolli, P., and Card, S. (2005). “The sensemaking process and leverage points for 3rd Workshop on Everyday Virtual Reality (WEVR) (Los Angeles, CA: IEEE),
analyst technology as identified through cognitive task analysis,” in Proceedings 1–6.
of International Conference on Intelligence Analysis (Atlanta, GA), 2–4.
Polys, N., Hotter, J., Lanier, M., Purcell, L., Wolf, J., Hession, W. C., et al. Conflict of Interest Statement: The authors declare that the research was
(2017). “Finding frogs: Using game-based learning to increase environmental conducted in the absence of any commercial or financial relationships that could
awareness,” in Web3D ’17, Proceedings of the 22Nd International Conference on be construed as a potential conflict of interest.
3D Web Technology (New York, NY: ACM).
Polys, N., Mohammed, A., Iyer, J., Radics, P., Abidi, F., Arsenault, L., et al. (2016). Copyright © 2019 Skarbez, Polys, Ogle, North and Bowman. This is an open-access
“Immersive analytics: crossing the gulfs with high-performance visualization,” article distributed under the terms of the Creative Commons Attribution License (CC
in 2016 Workshop on Immersive Analytics (IA) (Greenville, SC: IEEE), 13–18. BY). The use, distribution or reproduction in other forums is permitted, provided
Polys, N., Newcomb, C., Schenk, T., Skuzinski, T., and Dunay, D. (2018). “The the original author(s) and the copyright owner(s) are credited and that the original
value of 3D models and immersive technology in planning urban density,” in publication in this journal is cited, in accordance with accepted academic practice.
Proceedings of the 23rd International ACM Conference on 3D Web Technology No use, distribution or reproduction is permitted which does not comply with these
(New York, NY: ACM). terms.

Frontiers in Robotics and AI | www.frontiersin.org 15 September 2019 | Volume 6 | Article 82