See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/342964910

Improving UI Test Automation using Robotic Process Automation

Conference Paper · January 2020

DOI: 10.5220/0009911202600267

CITATIONS READS

9 830

3 authors, including:

Marina Cernat Alin Stefanescu

University of Bucharest University of Bucharest
2 PUBLICATIONS 17 CITATIONS 64 PUBLICATIONS 746 CITATIONS

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Modelling and Analysis of Cyber-Physical Systems View project

ATLAS: Innovation hub for advanced cyber-security technology View project

All content following this page was uploaded by Alin Stefanescu on 20 July 2020.

The user has requested enhancement of the downloaded file.

 Improving UI Test Automation using Robotic Process Automation

Marina Cernat1, Adelina-Nicoleta Staicu2 and Alin Stefanescu1 a

1Department of Computer Science and Research Institute of the University of Bucharest, University of Bucharest, Romania
2Cegeka, Romania

[email protected], [email protected], [email protected]

Keywords: Robotic Process Automation, RPA, UI Testing, Test Automation.

Abstract: Robotic Process Automation (RPA) is now one of the fastest growing segments in enterprise software. This
technology uses so called “software robots” that can mimic humans interacting with various applications at
the UI level. Thus, RPA achieves automation of various UI scenarios, without writing dedicated software to
implement them. In this position paper, we open a discussion on the opportunities and challenges of using
RPA to improve the test automation.

1 INTRODUCTION IT spending dedicated to QA and testing (Sogeti,

2019). However, less than 20% of the testing
Robotic Process Automation (RPA) (van der Aalst, processes are automated, so there is a lot of potential
2018; Syed, 2020) is one of the strongest contenders to improve the state-of-the-practice. In fact, the need
in the enterprise software market, being also the for test automation is reflected in its healthy growth –
fastest growing segment in 2019 (Gartner, 2019). The the automation testing market is estimated to double
technology found a sweet spot in enterprise its size from USD 12.6 billion in 2019 to USD 28.8
automation by emulating repetitive tasks done by a billion in 2024 (Markets and Markets, 2019). Testing
human operating a computer. The first-class citizen in activities are very diverse: from unit testing, to
this context is a so-called “software robot” that integration testing, to UI testing. While unit testing is
interacts with the system at the UI level, e.g., clicking almost entirely automatic, UI testing still involves a
or filling web forms with data copied from various lot of manual testing, with testers clicking through
data sources that are accessible through the front-end. interfaces following certain scenarios for acceptance
Being in principle agnostic to the technology behind testing or using their intuition in exploratory testing.
the UI, RPA can be easily deployed and integrated in In general, automating UI testing is regarded as
the workflows of a company including legacy challenging (Aho, 2018) due to fragile scripts when
applications that do not offer proper APIs. In fact, one the interface is changing, high maintenance
of the promises of RPA is that it is cheaper and (Alegroth, 2016), difficulty in defining the test
quicker to implement compared to writing a dedicated oracles (Memon, 2013), estimating the costs of
software. More precisely, the most suitable tasks for automation (Dobslaw, 2019), or combinatorial
RPA are those that are repetitive, but not frequent and explosion of the states to be explored (Nguyen, 2014;
structured enough to economically justify full Vos, 2015). Moreover, other complications appear
enterprise automation. On the other end of the when the UI testing needs to deal with scenarios
spectrum are those tasks that require a lot of creative involving several applications or even remote ones.
human input, are ad-hoc, or too infrequent to be worth In this situation, one does not have the same control
the investment in automation (van der Aalst, 2018). of the GUI elements as when only one web
In fact, due to its nature, UI testing is an area that application is under test (via e.g., Selenium). RPA
would be amenable to RPA automation, and this is the may provide a solution to some of these problems
very idea that we explore in this paper. since it works very well at the UI level, in a
Software testing is a very important stage in heterogeneous environment, is in many cases
software development, with a proportion of 25% of scriptless (thus easier to learn), more stable, and

a https://orcid.org/0000-0002-8418-2643
profits from recent advanced features such as • UiPath Robot: The workflows created in UiPath
computer vision. Industry already started to Studio are executed by a UiPath robot. There are
understand its potential benefits to UI test automation two main types of robots: attended and
(Bhukan, 2017; Murphy, 2019), even though RPA unattended, the difference being that the former
cannot be applied to UI testing out-of-the-box requires human inputs at some point during the
(Ariola, 2019). We hope that we spark the interest of execution.
the academic community to investigate and combine • UiPath Orchestrator: A component that manages
state-of-the-practice in RPA with state-of-art in the UiPath robots across various platforms.
testing to improve test automation (Arcuri, 2018). Due to requests and needs from the users, RPA has
The structure of the paper is the following: After become, in the past years, a conglomerate of different
a short introduction in RPA, we describe two usage technologies that are combined in order to help and
scenarios examples. Then, we discuss opportunities ease the development of an RPA project. Advanced
and challenges as well as related work. Finally, we plugins, including machine learning, cognitive
conclude with future work and an appendix. automation and computer vision, can now be used by
the robots. Also, third party integration with popular
cognitive services from Microsoft, Google, IBM are
2 A SHORT INTRO TO RPA readily available.
Another challenge and opportunity for RPA is the
Robotic Process Automation is the technology that automation in virtual or remote environments a.k.a.
enables the automation of a process by imitating and VDIs (virtual desktop interfaces). There is a clear
integrating the actions of a person who interacts with growth in VDI usage among enterprise customers,
digital systems when executing a process. Note that proving a need of quick and stable UI automations for
the RPA does not replace existing systems. Instead, it technologies such as Citrix, VMware and Windows
works with the current enterprise landscape and Remote Desktop. UiPath was among the first ones to
executes well-established actions. RPA interacts with implement it using latest breakthroughs in computer
the system in the same way a human would, but faster, vision research. Thus, it solved for the robot the
at a lower cost, and with less errors. This technology difficult problem to have obtain the underlying
offers the possibility to integrate modules developed properties of UI elements (buttons, text fields, etc.).
in programming languages such as VB.NET, C#, In VDI, this ability is damaged because of the lack of
Python, or Java. a traditional interface since the robot only obtains an
The RPA tool market is highly dynamic with image of a remote desktop. Mixing computer vision,
several companies competing in a fast-expanding machine learning, OCR, text fuzzy matching and a
environment (Gartner, 2019). The biggest players are multi-anchoring system, robots now automatically
UiPath, Automation Anywhere, and Blue Prism. In recognize on-screen elements, not relying on IDs,
the rest of the paper, we will refer to the UiPath hidden properties or metadata. Also, this solution for
platform, because it is currently the leader in the field VDIs can be extended also to work for similar
and, also, the second author of the paper has 2+ years situation of scanned PDFs, Microsoft Silverlight,
of industrial experience using its technology. images etc.
The main components of UiPath platform are: Yet another area of great progress of RPA recently
• UiPath Studio: An IDE that relies on Microsoft is that of processing and understanding unstructured
Workflow Designer and Microsoft Visual Studio information, especially text, through data mining,
natural language processing, and machine learning.
tools and uses the Visio-style process views to
graphically model workflows as sequences, These RPA advancements could be very helpful
when solving some of the problems of UI testing.
flowcharts, or state machines. It gives the user an
ergonomic experience. Built on .Net framework,
UiPath Studio allows working with all types of
variables, arguments, reading data from several 3 A SIMPLE EXAMPLE OF
formats (Excel, PDF, common databases, Word, TESTING USING RPA
desktop or web-based applications), writing data,
creating reports, handling keyboard strokes and To illustrate better how RPA could be used for test
mouse clicks, as well as Optical Character automation, we will provide a small example.
Recognition (OCR). Assume that we want to test two calculator apps, one
installed locally (e.g., the Windows calculator app),
but also one online (e.g., the calculator that appears 4 AN END-TO-END EXAMPLE
on top of the Google page when you search for
“calculator”). A scenario that a human would follow
OF TESTING USING RPA
to test them could be: take two numbers, for instance
1 and 2, add them on the calculator (desktop and To understand how RPA may improve UI testing, we
online) and verify if the output of the calculator is exemplify a more complex testing scenario that uses
indeed the expected value of 3. several applications and technologies.
To automate this in RPA, we quickly build a robot Assume we want to test various functionalities of
that can test in parallel both calculators using test data a back-office banking application. An unattended
extracted from an Excel spreadsheet. The spreadsheet robot could be implemented to do the following tasks.
contains test data with operands and an operator (1+2, First of all, we want to test if the application is
5+7, 3*8) and the expected computation result for working on several web browsers, so the robot should
them (3, 12, 24, respectively). The robot can successfully login regardless of the used browser. The
implement two test cases, one for each app, in the robot keeps an oracle application testing suite logging
usual way of doing testing, by reading the data, if the test passed or not in an Excel format. If the login
executing the application under test and comparing is successful, the test has passed, else the test failed.
the output with the expected values. If the values are After testing the login functionality, the robot tests
equal, the test has passed, otherwise it failed. The the ability of internal money transfers in the
graphical model of the robot is provided in Fig. 1 in application. So, it transfers an amount of money to
the Appendix, where we have the sequence with the another account, then it checks if the money arrived.
steps. When it runs, the robot will do the following If the transfer is successful, the test passed, and the
result is saved. After doing the payment, the bank
• Step 1 (test setup): It reads the values from Excel clerk, and now also the robot, should be able to see
(see Fig. 2 in the Appendix) and keeps them into and save the payment confirmation in PDF format.
an internal data table variable. So, the robot also tests this functionality, opening the
• Step 2: It opens the desktop and online calculator menu where it finds the payment confirmations and
in the same time, but in different threads. saves the most recent file. In order to check if the PDF
• Step 3 (test execution): For each row of the data file contains the fields that represent the amount of
table (which contained the test data, 1, +, and 2), money sent and the account to which the money was
the robot performs the encountered operation by transferred, the robot opens the PDF file and checks
clicking the buttons of the calculator, then it takes the corresponding fields. If they exist and are correct,
the obtained result and writes them back in the the test passed.
Excel spreadsheet (to be used for debugging, if the After this check, we move forward and test an
tests fail). See Fig. 3 in the Appendix. application that loads these PDF files in a database.
• Step 4: It closes both calculators. First, the robot checks the database connection.
• Step 5 (test assert): It compares the column of the Then, the robot checks if the files are correctly loaded
actual results and the column of the expected in the database. For all the files previously processed,
results from the spreadsheet and put into the the robot interrogates the database and checks if the
“Status Desktop”, respectively “Status Online” actual result corresponds to the expected result.
columns, the test results for each test being Last but not least, the robot will access for some
“Passed”, if the results were equal, and “Failed” final verifications a sensitive, more restricted area of
otherwise. See Fig. 4 in the Appendix. the bank system remotely through a VDI. In this case,
the robot has access only to screenshots of the remote
Thus, the robot is graphically modelled and will applications, without access to the logical elements of
automatically click through the interfaces of a the UI. However, also in this case the robot can use
desktop app, but also of a web app in the browser, the latest computer vision RPA add-ons (see previous
implementing a data-driven test suite and writing the section) and smoothly solve the given tasks.
results back into a file for the test reports. Note that We notice that writing test scripts to implement all
everything is done at the UI level and the robot these test cases is not at all easy, whereas using the
seamlessly switches between applications (Excel, advanced capabilities of RPA, the task becomes
desktop app, web browser). This would not be feasible.
possible using a classical UI web testing framework
such as Selenium, which would only be able to deal
with the web app in the browser, but not with the
desktop app.
5 OPPORTUNITIES AND used also in the context of RPA-based testing. So, we
should investigate how model-based testing, search-
CHALLENGES OF USING RPA based testing, automatic test generation, keyword-
FOR TESTING driven and data-driven testing, fuzz-testing,
exploratory testing, to name a few, could be
As already hinted until now, we consider that there embedded into RPA test robots. Also, the challenges
are many opportunities of using RPA to improve UI of UI testing (Aho, 2018) should be revisited,
test automation. investigating which of them could be solved by RPA-
First of all, RPA is much more accessible than test based UI testing. Last but not least, in order to be
scripting or programming. Most RPA tool providers, attractive to industry, cost models that include RPA
including UiPath, offer some products that do not licenses should be devised such that the final testing
require technical knowledge (see the lightweight toolchain and process will provide a good return-on-
UiPath StudioX solution aimed at non-technical investment (Dobslaw, 2019).
users). Therefore, businesspersons, field experts or
manual testers without technical knowledge can
automate some of their UI testing tasks. Thus, using 6 RELATED WORK
RPA, we can imagine that acceptance tests can be
automated easier and at a lower cost even by the end
We could not find any reference in the academic
users (for example, the doctors from a hospital) with
literature reading the idea of using RPA technologies
some training before, because they are the experts in
to automate UI testing. This is understandable since
the respective field and this is the best way to check
the RPA is a very recent technology that only recently
if the tested application or usage scenario is correct.
reached maturity (van der Aalst, 2018) and whose
Then, a robot can access through the UI any tools
challenges are started to be discussed (Syed, 2020).
available irrespective if APIs are provided or not.
There are only a handful of research papers that
Thus, if certain tools, including testing tools, are used
may be relevant to our discussion involving RPA for
inside an organization, the robot can use them to
quality assurance. (Lübke, 2016) discussed the
achieve its goal. An RPA robot could implement test
possibility of using BPMN for test case design, which
scenarios that are more flexible, maintainable, stable,
is similar to the graphical modeling of RPA robots.
easier to integrate (ERP legacy systems, CRM,
(Moffitt, 2019) proposed an approach of using RPA for
calendar, email, PDF, VDIs), and also accurate. This
auditing, which resembles in some way a testing
was proved by the fast adoption of RPA in enterprise
process. (Beschastnikh, 2017) envisaged a framework
automation, but we expect that this will still hold in
that deploys “bots” to do code analysis during the
the UI test automation domain (in fact, that could
software development process, but they do not work
constitute a good topic for an empirical software
at UI level and do not use RPA. (Enoiu, 2019)
engineering research project).
investigated the general concept of test agents that
Last but not least, using RPA for testing could
could distributively and cooperatively run a test plan.
capitalize on the high growth and advanced
It is an interesting idea to adapt for the context of RPA
developments of the RPA tool providers. They are
test robots. (White, 2019) studied the problem of UI
now in a race to include as many sophisticated
testing for VDIs and implemented a solution to
features as possible, and UI testing could benefit from
extract UI test relevant information from images. This
them, since the UI testing tool providers are not so
could complement the current RPA plugins that do
fast in implementing the latest technologies,
the same task.
especially in the field of AI. So, the robots will
We analysed the industrial UI test tools. We tried
become smarter in several dimensions and this could
to be as comprehensive as possible in our
clearly benefit a UI testing process deploying them.
comparative analysis and we downloaded (whenever
However, with great opportunities come also
possible) the tools and experimented with them in
challenges. First of all, the RPA as-is must be adapted
order to understand their main features, strengths and
to the UI testing requirements. This means that one
weaknesses. We do not have space in this short paper
should define a suitable test infrastructure, RPA tools
to discuss the findings, but we only mention those that
must be better integrated with existing test tools for
offer interesting or advanced features that could be
test management, reporting, test execution, and other
used in combination with an RPA-based test
test automation tools such that the strengths of all are
framework: Selenium (see also (Besant
combined. Then, one should see what can be further
Technologies, 2019)), Applitools, Robotframework,
automated such that state-of-the-art in testing can be
Power Automate, Eggplant, Eyeautomate, Squish,
Tricentis, Mabl, test.ai, apptest.ai, Functionize, REFERENCES
testim.io, as well as academic tools such as GUITAR
(Nguyen, 2014) and TESTAR (Vos, 2015). van der Aalst, W., Bichler, M., A. Heinzl, A., 2018. Robotic
Note that some of the RPA tools evolved from UI Process Automation. Business & Information Syst. Eng.
testing tools, e.g., Automation Anywhere (which in 60 (4), pp. 269-272, Springer.
the meanwhile is not active anymore in the testing Aho, P., Vos, T., 2018. Challenges in Automated Testing
market), while UI testing companies, e.g., Tricentis Through Graphical User Interface. In Proc. of ICST
(Murphy, 2019), Leapwork, want to enter the RPA Workshops 2018, pp. 118-121, IEEE.
market. On the other hand, some RPA companies Alegroth, E., Feldt, R., Kolstrom, P., 2016. Maintenance of
automated test suites in industry: An empirical study on
such as UiPath are starting to provide RPA-based visual GUI testing. Information and Software
testing solutions1. Technology, 73, pp. 66–80, Elsevier.
Arcuri, A., 2018. An experience report on applying
software testing academic results in industry: we need
7 CONCLUSIONS usable automated test generation. Empirical Software
Engineering 23 (4), pp. 1959-1981, Springer.
Ariola, W., 2019. RPA for Software Test Automation: Not
The goal of this short paper is to promote the idea of So Simple. CIO Magazine. Online at:
using the latest RPA technologies and research in https://www.cio.com/article/3409056/rpa-for-
order to improve the state-of-the-art in UI test software-test-automation-not-so-simple.html
automation. Based on our initial investigations, we Besant Technologies, 2019. RPA vs Selenium. Industry
believe that there is a great potential in this idea both blog post, 2019. Online at:
from an academic as well as industrial point of view. https://www.besanttechnologies.com/rpa-vs-selenium
As future work, there are many aspects that we Bhukan, S., 2017. Robotic Process Automation and the
Testing Future. Industry blog post. Online at:
plan to address. First, we will implement several
https://www.testingbits.com/robotic-process-
complex test scenarios using UiPath tooling to make automation-and-the-testing-future
an inventory of strengths and weaknesses with respect Beschastnikh, I., Lungu, M., Zhuang, Y., 2017.
to testing. Then, we will try to enhance them by Accelerating Software Engineering Research Adoption
integrating state-of-the-art tools and approaches from with Analysis Bots. In Proc. of ICSE-NIER, 2017, pp.
the testing research (see also the discussion at the end 35-38, IEEE.
of Section 5), but also existing commercial UI tools Cewe, C., Koch, D., Mertens, R., 2018. Minimal Effort
(see the tool list in Section 6). Some of the topics to Requirements Engineering for Robotic Process
focus on are: the use of AI to obtain a “smarter” test Automation with Test Driven Development and Screen
Recording. In Proc. of BPM’18 Workshops, LNBIP vol.
robot; the generation of test robots using process
308, pp. 642-648, Springer.
discovery and understanding (Gao, 2019) (see also Dobslaw F., et. al, 2019. Estimating Return on Investment
UiPath Explorer component); but also the for GUI Test Automation Tools. arXiv report no.
development of methods to test the RPA CoRR abs/1907.03475, 12 pp.
implementations themselves (see also (Cewe, 2018)). Enoiu E., Frasheri, M., 2019. Test Agents: The Next
Last but not least, we will perform all the above in the Generation of Test Cases. In Proc. of NEXTA’19, ICST
context of a collaboration with UiPath, which showed Workshops 2019, pp. 305-308, IEEE.
interest to provide feedback, access to tooling and Gao, J., van Zelst, S., Lu, X., van der Aalst, W, 2019.
industrial use cases to validate the resulting ideas and Automated Robotic Process Automation: A Self-
Learning Approach. In Proc. of OTM Conferences
prototypes.
2019, LNCS 11877, pp. 95-112, Springer.
Gartner, 2019. Magic Quadrant for Robotic Process
Automation Software. Market research report, no.
ACKNOWLEDGEMENTS G00379618.
Lübke D., van Lessen, T., 2016. Modeling test cases in
BPMN for behavior-driven development. IEEE
This work was supported by a grant of Romanian
Software 33 (5), pp. 15-21, IEEE.
Ministry of Research and Innovation CCCDI- Moffitt, K., Rozario, A., Vasarhelyi M., 2019. Robotic
UEFISCDI, project no. 17PCCDI/2018. We also Process Automation for Auditing. J. of Emerging
thank to Ingo Philipp, Vice President at UiPath, for Technologies in Accounting 15 (1), pp. 1-10.
inspiring discussions on the topic of the paper. Markets and Markets, 2019. Automation Testing Market by
Component, Endpoint Interface, Organization Size,

1 https://www.uipath.com/product/test-suite
 Vertical, and Region - Global Forecast to 2024. Market Syed R. et al., 2020. Robotic Process Automation:
research report no. TC 6163. Contemporary themes and challenges. Computers in
Memon, A., Banerjee, I., Nagarajan, A., 2013. What Test Industry 115.
Oracle Should I Use for Effective GUI Testing? In Vos, T., Kruse, P., Condori-Fernández, N., Bauersfeld, S.,
Proc. of ASE 2003, pp. 164-173, IEEE. Wegener, J., 2015. TESTAR: Tool support for test
Murphy, T., 2019. Test automation + RPA,” Tricentis automation at the user interface level. Int. Journal of
industry webinar Online at: Information System Modeling and Design 6 (3), pp. 46-
https://www.tricentis.com/resources/test-automation-rpa 83, IGI Global.
Nguyen, B., Robbins, B., Banerjee, I., Memon, A., 2014. White, T., Fraser, G., Brown, G., 2019. Improving random
GUITAR: an innovative tool for automated testing of GUI testing with image-based widget detection. In
GUI-driven software. Automated Softw Engineering 21 Proc. of ISSTA 2019, pp. 307-317, ACM.
(1), pp. 65-105, Springer.
Sogeti, 2019. World Quality Report 2019-2020, 11th
edition. Market research report.

APPENDIX

Figure 1: Implementation of a test robot in UiPath for a test suite with two test cases: the one on the left to verify the desktop
calculator and the one on the right to verify an online calculator.
Figure 2: Excel file containing the test data: The first input data in the first 3 columns and output data in the 4th column. The
rest of the columns are used to record the results of the tests.

Figure 3: The RPA sequence for automating the calculation 1+2 in the desktop calculator (the screenshots show the clicks
performed by the robot).
 Figure 4: The implementation of the comparison between the expected value (oracle) and the calculated value.

View publication stats