Scribd
Upload
78 views

Lesson 9 - Image & Text Based Automation

This document discusses various techniques for automating virtualized applications including image recognition, text recognition, keyboard automation, and best practices. It provides examples of automating the Expenseit application using only image, text, and keyboard inputs. Resources for learning more about automating data entry, retrieving data, and using OCR in Citrix environments are also included.

Uploaded by

PVParameswaran
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
78 views

Lesson 9 - Image & Text Based Automation

This document discusses various techniques for automating virtualized applications including image recognition, text recognition, keyboard automation, and best practices. It provides examples of automating the Expenseit application using only image, text, and keyboard inputs. Resources for learning more about automating data entry, retrieving data, and using OCR in Citrix environments are also included.

Uploaded by

PVParameswaran
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 18

Image & Text Automation

Image & Text Automation - Agenda

How to automate virtualized


applications
Automation Techniques Best Practices
Image Q&A
Text Test
Keyboard automation
Online Resources
Insert data
Retrieve data
How to Automate Virtualized Apps
Things you should know

 UI Elements are not available in Citrix.

 Everything appears like an image.

 App must always be visible on screen.

 Only working with the main window/element.


Image Automation
Input Output
Activities Activities

 Click Image used to identify Not Applicable


elements based on its image

 Find Image used to verify the


active window
Insert data in Citrix
 Click relative to image or text For instance in this application we indicate with
yellow the text and with red the area which we
want to be clicked.

 Keyboard hotkeys, tab, enter

 Paste text via shared clipboard


Retrieve data
 Copy selected text from text areas, tables and so on.

 Click relative to a static text and then get the


data in clipboard
Retrieve data
 Scrape - relative to image or text

Example: Scrape relative to “Cash In”.


Text Automation
Input Output
Activities Activities

● Click Text used to identify ● Get Text used to identify


elements based on its image elements based on its image

● Find Image used to verify the


active window
OCR Engines

Filter Better on
More Preffered Can invert Set expected
allowed Microsoft
languages area size colors text format
characters Fonts

Google
Tesseract
can add small ✓ ✓ ✓

Microsoft
Modi
default large ✓
Keyboard Automation
Input Output
Activities Activities
 Send Hotkey  Select Text

 Type - send keyboard commands Type: "[k(end)d(shift)d(home)u(shift)u(home)]"

 Copy Selected Text


Review

Input Output

Image Yes No

Text Yes Yes

Keyboard Yes Yes


Identify elements
Keyboard automation

Sometimes could be very difficult to do a good text/image automation so you


can use the keyboard.

 Attach to the window.

 Use “Send Hotkeys” activity.


Identify Elements
Text - identified by specific text/word regions

1) OCR - 98% accurate

▪ Google OCR Engine


▪ Microsoft OCR Engine

2) Native - 100% accurate


Best practices
 Be careful when you try to click on a text to be unique. For instance if you
want to click on a “Cancel” button and the text “cancel” is somewhere else
then you better use click image and indicate that button.

 The text/image always should be visible. In Citrix automation if the user is


not able to see an image or a text neither the robot.

 Feel free to use .NET methods like “Split()” , “Substring()”, “ StartsWith() ”


and so on when you scrape a text.
Exercise 1: add new expense
Download Expenseit application and add a new expense in the application
using only image/text and hotkeys.

1. Download the application from here:


https://drive.google.com/file/d/0BxVAKI3l9Pc5d0JqY2E2RGYtZDA/vie
w?usp=sharing

2. Add expenses rows just by using text or image techniques


Online Resources
Data entry automation in Citrix
http://www.uipath.com/tutorials/citrix-automation-data-entry

Retrieve data using clipboard


http://www.uipath.com/tutorials/citrix-automation-keyboard-hotkeys

Retrieve data using OCR


http://www.uipath.com/tutorials/citrix-automation-ocr-data-extraction

Automation using text scraping


http://www.uipath.com/tutorials/sap-gui-automation-data-entry-scraping

You might also like