2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)

Mixed Methods Designs for User Studies in Cross Reality

Judith Friedl-Knirsch* Christoph Anthes†
University of Applied Sciences Upper Austria University of Applied Sciences Upper Austria

A BSTRACT We then analyse how the individual methodologies have been ap-
plied in selected published user studies in cross reality and discuss
Mixed methods methodology is a research approach that combines
their benefits and drawbacks. Finally, we summarise deliberations
qualitative and quantitative data collection and analysis. This al-
that should be taken into account when applying mixed methods
lows to synthesise both types of data for answering the complex
methodology in user studies.
research questions posed in cross reality research. However, all
research methods have benefits and drawbacks as well as method- 2 BACKGROUND OF R ESEARCH M ETHODOLOGIES
ological backgrounds. Therefore, we first discuss the underlying
foundations of qualitative, quantitative and mixed methods method- Both qualitative and quantitative methods used for user evaluation
ology, before looking into how these approaches are applied in were adapted from methods used in social science [4]. However,
cross reality research. Therefore, we selected exemplary published qualitative and quantitative methodology are based on different
work for each of the methodologies and discuss the research de- epistemological assumptions, i.e. different philosophical theories
sign, as well as limitations and benefits of the used methodologies. of how knowledge is generated as opposed to opinions.
After this discussion of real cross reality research designs, we sum- Quantitative methodology results from positivism or postposi-
marise further considerations for the practical application of mixed tivism [4, 3]. This methodological approach is based on the as-
methods research designs. sumption that there is an objective truth. To get to this truth, quanti-
tative methodology seeks to avoid all subjectivity in the knowledge
Index Terms: Mixed methods, user study methodology, quantita- seeking process. It follows a deductive approach where concrete,
tive methods, qualitative methods. testable hypotheses are formed before the data is collected. The
data is then used to specifically test these a priori hypotheses. In
1 I NTRODUCTION human-computer interaction the most common approach to do so
is through controlled experiments. This allows researchers to min-
Cross Reality (CR) deals with complex research questions, as it imise confounding variables to infer a causal relationship between
combines different stages of Milgram’s Reality-Virtuality Contin- dependant and independent variables in a statistical data analysis.
uum (RVC) [17]. This adds an additional layer of complexity as Qualitative methodology, on the other hand, is based on inter-
different virtual contexts need to be considered within one proto- pretivist or constructivist views [4, 3]. The base assumption here
type. This means, that CR prototypes are not only technologically is, that there is no objective truth. Instead there are multiple per-
more challenging than individual Augmented Reality (AR) or Vir- spectives and research that is conducted by and with humans can
tual Reality (VR) prototypes, but also often require different inter- not be objective. In contrary to positivist approaches, subjectivity
action and navigation metaphors from the individual domains. In is seen as a tool to interpret data in its context. Qualitative Re-
collaborative scenarios, moving along the RVC also has major im- search is based on an inductive theory building approach, meaning
plications on the communication between participants. The addi- that based on a research question data is collected and based on this
tional layer of the RVC also means that it is harder to isolate effects data, theories are formed. In human-computer interaction, qualita-
towards a clear causal relationship. tive methodology is often used in user studies where the participants
There are multiple different approaches and open research areas perform a specific task or explore a tool. The data collection meth-
with technology driven research, evaluation of perceptive concepts ods that are then used are often a combination of observation and
and application based research all being equally valid and relevant, interviews.
To evaluate these complex research questions, it is useful to col- The mixed methods methodology is based on pragmatism or
lect evidence from different perspectives and multiple theoretical realism [2]. Pragmatism is however the dominant approach that
approaches. In this and related communities, there have been dis- is based on the belief of using the methodology that works best,
cussions around how user studies are and should be conducted for i.e. that fits the research question. Thus, mixed methods approach
several years [15, 14, 8, 7, 16, 10, 25]. While some emphasise the draws on both qualitative and quantitative data collection and anal-
lack of qualitative approaches [14, 10], others focus on improving ysis method to gain knowledge. This allows for a combination of
the methodology and reporting of quantitative studies to avoid a the inductive, qualitative theory building approach and the deduc-
replication crisis [25]. We believe that there is benefit in going be- tive theory-testing approach of the quantitative methodology.
yond these two separate methodologies. Mixed methods could be There are different research designs for mixed methods research.
a useful approach to formalise this research process, as it allows to Qualitative and quantitative data can either be collected within the
synthesise qualitative and quantitative data to gain comprehensive same study or by conducting a quantitative and a qualitative study
and valid insights. [6]. This depends on the research questions and the remaining study
To demonstrate the relevance of mixed methods research, we first parameters. It is also necessary to consider the different require-
discuss the methodological background of the individual method- ments of qualitative and quantitative methodologies. When con-
ologies to outline the underlying assumptions of these approaches. ducting two separate studies it can be classified into an explorative
and an explanatory approach, according to the sequence of the stud-
* e-mail: [email protected] ies. For the explorative approach the qualitative study is conducted
† e-mail: [email protected] first to form theories and the quantitative study is then used to test
these theories to enable generalisation. In the explanatory approach,
the quantitative study is conducted first to test whether an a priori
hypothesis is true. The qualitative study is then added to explain
the background of why this theory is true or is not true.

2771-1110/24/$31.00 ©2024 IEEE 161

DOI 10.1109/ISMAR-Adjunct64951.2024.00042
3 A PPLICATIONS IN CR R ESEARCH a custom questionnaire or a standardised one. The second study
After discussing the background of the research methodologies, we in this area by Xia et al. [24] focuses on evaluating a system for
now look at how they have been applied in user studies in CR. collaborative CR. The system was developed for PC and HMD and
Therefore, we discuss published work that includes user studies in the evaluation collects data on the user experience of this system
CR and describe how the study methodologies have been applied. in three different contexts. For the data collection, two standard-
We also discuss limitations and advantages of the methodologies, ised questionnaires are used that also provide a benchmark dataset.
based on how they are applied. The selection of publications is not While the study uses a quantitative within subjects design, there is
a complete representation of user studies in cross-reality, but rather no comparative data analysis. Instead, the scores for the question-
provides examples for the methodologies and their application in naires are calculated and then compared to the benchmark dataset.
real CR research. Overall, quantitative studies are useful for testing preexisting hy-
potheses. They do, however, not provide information on the back-
3.1 Quantitative ground of these results. Additionally, design decisions in the im-
plementation and task design have a great influence on the results.
Pazhayedath et al. [18] conducted two within-subjects user studies Ideally, quantitative studies allow researchers to draw generalisable
on pinpointing techniques for CR collaboration. These techniques conclusions. Therefore, it is necessary to apply appropriate sam-
are used for communication between one user immersed in VR and pling strategies and to achieve sufficient statistical power by using
one non-immersed user who wants to draw the attention of their a large enough sample.
collaborator towards a specific object in the virtual environment.
In the first study, three different techniques, one of which posing
3.2 Qualitative
as a baseline condition, are compared using an objective measure-
ments of completion time and subjective measures of standardised In the context of CR, qualitative methodology is often used to
and custom questionnaires. While these questionnaires are referred gather feedback and insights from experts.
to in the publication as “qualitative” measures, they are in fact sub- Hubenschmid et al. [13] developed a study analysis tool for
jective quantitative measures as the questionnaires use Likert scales mixed reality user studies that is in itself a CR application. It al-
instead of open-ended questions. The results are then analysed us- lows users to analyse and watch the study data both in situ and
ing null-hypothesis significance testing. In the second study, the ex situ. Therefore, users can switch between a desktop view for
two main conditions were further developed to enable bi-directional both modes and an immersive view for the in situ analysis. Since
communication between the participants. This was again evaluated this tool specifically focuses on researchers in the field and not in-
through standardised and custom questionnaires and some state- tended for use in the general population, the user study is only use-
ments by participants are mentioned, without indication whether ful when conducted with users from this domain. The qualitative
they were collected during or after the study. These study designs expert study conducted by Hubenschmid et al. [13] allows for a
are typical for quantitative studies where hypotheses are tested for holistic perspective of these domain experts and systematic feed-
significance by manipulating an independent variable, in this case back integration. In the study, participants were first required to
pinpointing techniques and measuring the effect of this change on complete specific analysis tasks based on a real study analysis pro-
the dependent variables of completion time and questionnaire re- cess in a guided phase. This phase familiarised the participants
sponses. The results here can tell us whether a one condition out- with the analysis tool and its features. This was later followed by a
performed the other condition, but not why this is the case. Ad- free exploration phase of a different dataset. For the data analysis
ditionally, design decisions of how exactly the pinpointing tech- Hubenschmid et al. [13] employed a thematic analysis approach
niques were implemented influence the results and therefore pose a [5]. This is an inductive process where the qualitative artifacts such
limitation to the study. Since every study has limitations, the key as transcripts and videos are coded to summarise their content. The
point is to not overestimate the results of a single quantitative study. codes are iteratively refined clustered to find emerging themes in
Instead, multiple studies using different scenarios might be neces- the data. In this case, the coding was performed by one researcher
sary to understand how pinpointing techniques for CR collaboration and the codes were checked an validated by two further researchers.
should be applied. This qualitative analysis delivers deep insights into what experts
While these two studies also include subjective questionnaire need from a data analysis tool for mixed reality user studies. It al-
data, Schwajda et al.[20] focus solely on objective data when re- lows to draw conclusions on which aspects of the concept worked
porting on three quantitative studies on transforming graph data and which didn’t as well as directions for future research and im-
from 2D to 3D in AR. They use measurements of time and error plementation. While experts are able to imagine how to use this
to investigate different interaction techniques, data groupings and tool for their own data, the results are not generalisable to all data
visual links in the transformation of a node-link graph from a large and every use case. For qualitative expert studies, it is especially
2D screen to a 3D representation in AR. For more reliable mean relevant to provide details on the background of the experts to ac-
based statistical data analysis, each participant completed four tri- curately represent who was included in the study. In this case, for
als per condition. The first of these trials was a demo trial that example, it could be a limitation, that the experts were all computer
was not included in the data analysis. Therefore, for example, in scientists working in similar fields at the same university. There-
the first study with three conditions and 19 participants a total of fore, their approaches to studies and data analysis might be at least
171 trials were collected for the data analysis. In contrast, in the partially similar. However, the reporting of allows users to accu-
first study of Pazhayedath et al. [18] with eight participants and rately estimate the influence of this factor.
three conditions only 24 data points were analysed. Therefore, the The qualitative expert evaluation was also used for another tool
statistical power of the data analysis is too small for generalisable targeted at researchers. However, instead of the data analysis phase,
conclusions. But not only the amount of participants but also the the VRception toolkit focuses on rapid prototyping of CR systems
sampling strategy impacts the generalisability of results, as the tar- [11]. The idea here is to simulate all stages of the RVC [17] in
get population needs to be accurately represented virtual reality. For the expert study, the researchers used their pro-
Two further studies on CR collaboration use a quantitative within totyping toolkit VRception to create prototypes of published work
subjects approach. One investigates the level of involvement of a and present them to the original authors of the study. The authors
non-HMD user on their user experience when in visiting a virtual were presented with their own and other recreations of published
museum together with an HMD user [23]. User experience is mea- prototypes and were introduced to the functionality of VRception,
sured using a questionnaire. It is however, not specified if this was before being interviewed. Then, an open coding approach was per-

162
formed the codes were further clustered into categories using an position in the virtual environment, their stage on the RVC and in-
affinity diagram [12]. While it is not specifically called a thematic teractions with the prototype were logged. Additionally, multiple
coding process, it certainly reads similar to the process applied by standardised and custom questionnaires were applied. For the qual-
Hubenschmid et al. [13]. This emphasises that there is currently no itative data collection, the sessions and interviews were recorded
unified way of reporting qualitative coding. After reporting on the and a handwritten log was kept to note observations that were later
expert study, Gruenefeld et al. [11] also report a remote interview discussed in the interviews. In the data analysis, the qualitative data
study, where four pairs of two collaboratively created rapid cross- was analysed first to identify commonalities and differences in the
reality prototypes using the VRception toolkit. While this study in- collaboration, mainly through affinity diagrams. Then the quanti-
cludes some quantitative metrics, such as task completion time and tative data was analysed in the process of deriving the analytical
responses to standardised questionnaires, this data was only used in lenses for the collaboration. While the result is a synthesis of qual-
a descriptive manner with the focus of the study lying on the qual- itative and quantitative data, the analysis process deviates from tra-
itative interview data, using the same analysis process as for the ditional quantitative data analysis, as the hypotheses were formed
expert evaluation. after the data collection.
The approach of a presenting a prototype to domain experts, fol- Seraji et al. [21] used a between-subjects mixed methods design
lowed by a semi-structured interview was also used in different do- to investigate user behaviour patterns in the context of CR immer-
mains such as surgery planning. Aigner et al. [1] presented four car- sive analytics, comparing visual data analysis on a desktop to their
diologists with a cross-reality prototype that allowed them to look CR system HybridAxes. In terms of qualitative data, they collected
at the volumetric representation of a heart and its inner structures. notes during the experiment, recorded the sessions and captured the
The use case for this scenario is surgery planning. After a guided screen of the devices. On the quantitative side, each action and the
tour of the prototype and a free exploration phase of the model, the status of the system, as well as head hand and gaze movement were
semi-structured interview was conducted. The analysis also used logged. In the analysis procedure an open coding approach was ap-
an open coding approach, but described in less detail as the focus plied to the qualitative data, which was analysed first. Like in the
of the publication lies on the introduction of the application. The study by Schröder et al. [19], the quantitative data was then used
qualitative expert study is here used as a way to get early feedback afterwards to quantitatively characterise the usage patterns from the
and understand the usefulness of such a prototype for surgery plan- qualitative findings.
ning. In such early pilot studies, a small scale qualitative study is A less common approach to mixed methods research is provided
often a good fit for understanding the users’ way of working and by Wang et al. [22]. In this study, the qualitative elicitation study
the suitability of the proposed tool. method is applied to gather their preferred input methods for tran-
In summary, qualitative methodologies are useful to gain an ex- sitions of 3D objects in CR. The data is collected by asking par-
tensive understanding of what experts need and how they use a tool, ticipants about their preferred interaction methods for 20 different
as well as to get early feedback. Since qualitative results are inher- scenarios. While the collected data is qualitative, the data analy-
ently subjective, it is even more important to report the results in sis is focused on quantifying participant agreement on the different
a way that conveys their relevance in the context of use. It is also input methods. Therefore, this provides a quantitative perspective
indispensable to clearly describe the analysis process to enhance on qualitative data. Nevertheless, this focus on the quantification
the transparency of how the results were formed. A common pro- also leads to a lack in qualitative explanation of the participant’s
cedure in the context of CR is to provide a demonstration and/or reasoning for their proposed input methods.
free exploration of a prototype to the domain experts, followed by Overall, mixed methods enables the combination of different
a semi-structured interview. The first part of this procedure allows methodological viewpoints to derive valid and comprehensive con-
researchers to present their ideas to the experts and demonstrate the clusions for complex research questions. This combination of dif-
possibilities of a tool. Data is mainly collected through interview ferent methodological perspectives, however, also comes at a price.
data and observation, and the analysis is typically performed using While cross validation is a good way of forming reliable results, it
inductive open coding approaches. is not always possible to find quantitative evidence for qualitative
reasoning, especially in studies that collect quantitative and quali-
3.3 Mixed Methods tative data at the same time. By mixing qualitative and quantitative
methods in an individual research design, it is difficult to follow
Friedl-Knirsch et al. [9] use a mixed methods approach to analyse all the procedures and requirements for each methodological ap-
the influence of the different types of AR devices in a collaborative proach. For example, when trying to cross validate findings from
data analysis process. Each of the three users in a group there- qualitative data analysis with quantitative data [21, 19], it is not
fore uses a different AR device as well as a laptop during the task. possible to form hypotheses ahead of the data collection, since they
They pose three research questions which are answered by collect- are inductively generated during the qualitative data analysis.
ing quantitative and qualitative data for each of them to cross val-
idate and support the findings from different methodological per-
spectives. For the quantitative aspects, speech times, interaction 4 C ONSIDERATIONS WHEN USING M IXED M ETHODS R E -
SEARCH
log data, and several standardised and custom questionnaires are
used. For the qualitative aspects, observation and interview data In this section, we discuss considerations for using mixed methods
are collected. A within-subjects design was applied where each of study designs in CR.
the participants worked with each of the device types. In the end,
a semi-structured group interview was conducted. A limitation of Data synthesis is key One key aspect of the mixed methods ap-
this study was the selection of measures to characterise the collabo- proach is to synthesise the data in the findings. This means
ration, which relied mainly on verbal communication on the quan- that collecting some qualitative and some quantitative data
titative side. While collaboration was also part of the observation in a study is not per se a mixed methods approach. Instead
and the group interview, there are also more quantitative measures both data collections should serve a coherent goal. This can
to characterise collaboration. for example be cross validating findings with quantitative and
Also in a collaborative scenario, but with a focus on analysing qualitative data [19, 21]. This entails collecting data on the
collaboration where users could transition between different stages same phenomenon from both perspectives. Another approach
on the RVC, Schröder et al. conducted a study using a spatial op- would be to look at different aspects of the same research
timisation task. For the quantitative part of the study, the users’ question using different methods [9].

163
Different methodological requirements Each methodological the background of qualitative, quantitative and mixed methods re-
approach has different requirements and assumptions for a search, we analyse selected examples from published user studies
user study. Especially in an approach that collects both types in cross reality and discuss how the methodological approaches are
of data in one user study, these requirements can be difficult applied. Furthermore, we elaborate on the benefits and drawbacks
to combine. This is for example the case regarding the of the methodologies and discuss considerations for the practical
inherent difference of the inductive theory building approach application of mixed methods methodology.
of qualitative methodology and the deductive theory testing
approach of quantitative methodology means that. There ACKNOWLEDGMENTS
is a risk to violate either the requirement of quantitative
research to form the hypotheses before data collection, or the This work is part of X-PRO. The project X-PRO is financed by
assumption of theories being built inductively from the data. research subsidies granted by the government of Upper Austria.

There is no perfect research design Research designs need to fit R EFERENCES

their purpose. Depending on the domain, the available re-
sources, the researchers involved and other environmental fac- [1] D. Aigner, N. Wang, D. Kielmayer, J. Steiner, J. Hochpöchler,
tors, a different research design may be the most suitable. C. Heinzl, D. Roth, F. Maurer, and C. Anthes. Cardiac Visualisa-
While the study design needs to fit the research question, the tion Along the RV-Continuum - A High-Fidelity Pilot Study. In 2023
IEEE International Symposium on Mixed and Augmented Reality Ad-
research question itself depends on the framing and interpre-
junct (ISMAR-Adjunct), pp. 675–680, Oct. 2023. ISSN: 2771-1110.
tation of the researchers. In the process of designing a user
doi: 10.1109/ISMAR-Adjunct60411.2023.00145 3
study, it is also necessary to make decisions on what and how [2] F. Alele and B. Malau-Aduli. An Introduction to Research Methods for
to measure. Each decision has consequences with their ben- Undergraduate Health Profession Students. James Cook University,
efits and drawbacks. This means that every user study has Feb. 2023. 1
limitations. [3] F. Alele and B. Malau-Aduli. Philosophical Underpinnings to Mixed
Methods Research. In An Introduction to Research Methods for Un-
Clear presentation is key With mixing different methodologies dergraduate Health Profession Students. James Cook University, Feb.
there is increased potential for confusion. Combining qual- 2023. 1
itative and quantitative methodologies and approaches, adds [4] A. Blandford, D. Furniss, and S. Makri. Qualitative HCI research:
complexity to the publication as there are even more possibili- Going behind the scenes. Morgan & Claypool Publishers, 2016. 1
ties for research designs, than with purely qualitative or quan- [5] V. Braun and V. Clarke. Using thematic analysis
titative research. For readers to correctly interpret the study in psychology. Qualitative Research in Psychology,
and its results, it is therefore indispensable to aim for per- 3(2):77–101, Jan. 2006. Publisher: Routledge eprint:
fectly clear presentation in the publication. Readers must be https://www.tandfonline.com/doi/pdf/10.1191/1478088706qp063oa.
aware which part is quantitative and which is qualitative. This doi: 10.1191/1478088706qp063oa 2
includes using the correct wording, explicitly stating the re- [6] J. W. Creswell. Mixed Methods Procedures. In Research Design:
search question and its operationalisation and actively point- Qualitative, Quantitative and Mixed Methods Approaches. Sage, 3rd
ing towards the limitations of the study. edition ed., 2009. 1
[7] J. Friedl, J. D. Mayer, and C. Anthes. Germane Elements for the
Purely quantitative or qualitative research is still valid While Evaluation of Transitional Interfaces. In Workshop ”Transitional In-
combining qualitative and quantitative methodologies to form terfaces in Mixed and Cross-Reality: A new frontier?” (TI 2021) co-
a broad basis for the results of the study might seem like a located with the 15th ACM International Conference on Interactive
Surfaces and Spaces (ACM ISS 2021), 2021. 1
solution for almost any situation (and a way to comply with
[8] J. Friedl-Knirsch, F. Pointecker, S. Pfistermüller, C. Stach, C. Anthes,
all reviewer requirements) it is not suitable for all research
and D. Roth. A Systematic Literature Review of User Evaluation in
questions. For example, when the only goal of the research is Immersive Analytics. Computer Graphics Forum, 43(3), 2024. 1
to draw generalisable conclusions, a quantitative approach is [9] J. Friedl-Knirsch, C. Stach, F. Pointecker, C. Anthes, and D. Roth. A
the most suitable. Moreover, studies should not be rejected Study on Collaborative Visual Data Analysis in Augmented Reality
for not including both types of data, even if both data types with Asymmetric Display Types. IEEE Transactions on Visualization
would be applicable. Mixed methods approaches can also and Computer Graphics, 30(5):2633–2643, May 2024. Conference
be split across multiple studies and publications in order to Name: IEEE Transactions on Visualization and Computer Graphics.
provide methodologically clear and precise quantitative or doi: 10.1109/TVCG.2024.3372103 3
qualitative studies. [10] B. Fröhler, C. Anthes, F. Pointecker, J. Friedl, D. Schwajda,
A. Riegler, S. Tripathi, C. Holzmann, M. Brunner, H. Jodlbauer,
Required resources Combining both quanitative and qualitative H. Jetter, and C. Heinzl. A Survey on Cross-Virtuality Analytics.
methodologies drastically increases the ressources necessary Computer Graphics Forum, 41(1):465–494, Feb. 2022. doi: 10.1111/
for a user study. Qualitative data analysis is a time consum- cgf.14447 1
ing endeavour, with large amounts of unstructured qualitative [11] U. Gruenefeld, J. Auda, F. Mathis, S. Schneegass, M. Khamis, J. Gu-
data that need to be revisited. Additionally, it requires more genheimer, and S. Mayer. VRception: Rapid Prototyping of Cross-
than one researcher to at least check and validate the results Reality Systems in Virtual Reality. In CHI Conference on Human Fac-
in the data as subjectivity is inherent for the method. Addi- tors in Computing Systems, pp. 1–15. ACM, New Orleans LA USA,
tionally, researchers need to be experienced in both types of Apr. 2022. doi: 10.1145/3491102.3501821 2, 3
data collection and analysis. However, there is also the issue [12] G. Harboe and E. M. Huang. Real-World Affinity Diagramming Prac-
of enrolling participants for the study. Complex study designs tices: Bridging the Paper-Digital Gap. In Proceedings of the 33rd An-
nual ACM Conference on Human Factors in Computing Systems, CHI
often require more time per session making it harder to find
’15, pp. 95–104. Association for Computing Machinery, New York,
enough participants for the study.
NY, USA, Apr. 2015. doi: 10.1145/2702123.2702561 3
[13] S. Hubenschmid, J. Wieland, D. I. Fink, A. Batch, J. Zagermann,
5 C ONCLUSION
N. Elmqvist, and H. Reiterer. ReLive: Bridging In-Situ and Ex-
In this position paper, we emphasise the relevance of mixed meth- Situ Visual Analytics for Analyzing Mixed Reality User Studies. In
ods study designs for cross reality research. After discussing CHI Conference on Human Factors in Computing Systems, pp. 1–

164
 20. ACM, New Orleans LA USA, Apr. 2022. doi: 10.1145/3491102.
3517550 2, 3
[14] T. Isenberg, P. Isenberg, Jian Chen, M. Sedlmair, and T. Moller. A
Systematic Review on the Practice of Evaluating Visualization. IEEE
Transactions on Visualization and Computer Graphics, 19(12):2818–
2827, Dec. 2013. doi: 10.1109/TVCG.2013.126 1
[15] H. Lam, E. Bertini, P. Isenberg, C. Plaisant, and S. Carpendale. Em-
pirical Studies in Information Visualization: Seven Scenarios. IEEE
Transactions on Visualization and Computer Graphics, 18(9):1520–
1536, Sept. 2012. doi: 10.1109/TVCG.2011.279 1
[16] L. Merino, M. Schwarzl, M. Kraus, M. Sedlmair, D. Schmalstieg,
and D. Weiskopf. Evaluating Mixed and Augmented Reality: A
Systematic Literature Review (2009-2019). In 2020 IEEE Interna-
tional Symposium on Mixed and Augmented Reality (ISMAR), pp.
438–451. IEEE, Porto de Galinhas, Brazil, Nov. 2020. doi: 10.1109/
ISMAR50242.2020.00069 1
[17] P. Milgram, H. Takemura, A. Utsumi, and F. Kishino. Augmented
reality: a class of displays on the reality-virtuality continuum. pp.
282–292. Boston, MA, Dec. 1995. doi: 10.1117/12.197321 1, 2
[18] P. Pazhayedath, P. Belchior, R. Prates, F. Silveira, D. S. Lopes,
R. Cools, A. Esteves, and A. L. Simeone. Exploring Bi-Directional
Pinpointing Techniques for Cross-Reality Collaboration. In 2021
IEEE Conference on Virtual Reality and 3D User Interfaces Ab-
stracts and Workshops (VRW), pp. 264–270, Mar. 2021. doi: 10.1109/
VRW52623.2021.00055 2
[19] J.-H. Schröder, D. Schacht, N. Peper, A. M. Hamurculu, and H.-C.
Jetter. Collaborating Across Realities: Analytical Lenses for Under-
standing Dyadic Collaboration in Transitional Interfaces. In Proceed-
ings of the 2023 CHI Conference on Human Factors in Computing
Systems, pp. 1–16. ACM, Hamburg Germany, Apr. 2023. doi: 10.
1145/3544548.3580879 3
[20] D. Schwajda, J. Friedl, F. Pointecker, H.-C. Jetter, and C. Anthes.
Transforming graph data visualisations from 2D displays into aug-
mented reality 3D space: A quantitative study. Frontiers in Virtual
Reality, 4, Mar. 2023. Publisher: Frontiers. doi: 10.3389/frvir.2023.
1155628 2
[21] M. R. Seraji, P. Piray, V. Zahednejad, and W. Stuerzlinger. Analyz-
ing User Behaviour Patterns in a Cross-Virtuality Immersive Analytics
System. IEEE Transactions on Visualization and Computer Graphics,
30(5):2613–2623, May 2024. Conference Name: IEEE Transactions
on Visualization and Computer Graphics. doi: 10.1109/TVCG.2024.
3372129 3
[22] N. Wang, D. Zielasko, and F. Maurer. User Preferences for Interactive
3D Object Transitions in Cross Reality - An Elicitation Study. In
Proceedings of the 2024 International Conference on Advanced Visual
Interfaces, pp. 1–9. ACM, Arenzano, Genoa Italy, June 2024. doi: 10
.1145/3656650.3656698 3
[23] X. Xia, Y. Li, and H.-N. Liang. CovisitVM: Cross-Reality Virtual
Museum Visiting. In 2024 IEEE Conference on Virtual Reality and
3D User Interfaces Abstracts and Workshops (VRW), pp. 1074–1075,
Mar. 2024. doi: 10.1109/VRW62533.2024.00333 2
[24] X. Xia, J. Liang, R. Zhao, Z. Zhao, M. Wu, Y. Li, and H.-N. Liang.
Cross-Reality Interaction and Collaboration in Museums, Education,
and Rehabilitation. In 2023 IEEE International Symposium on Mixed
and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 815–820, Oct.
2023. ISSN: 2771-1110. doi: 10.1109/ISMAR-Adjunct60411.2023.
00180 2
[25] D. Zielasko and T. Weissker. Stay Vigilant: The Threat of a Replica-
tion Crisis in VR Locomotion Research. In Proceedings of the 29th
ACM Symposium on Virtual Reality Software and Technology, VRST
’23, pp. 1–10. Association for Computing Machinery, New York, NY,
USA, Oct. 2023. doi: 10.1145/3611659.3615697 1

165