SikuliX Documentation

Release 1.1.0-Beta1
Raimund Hocke aka RaiMan
March 19, 2014
Contents
i
ii
SikuliX Documentation, Release 1.1.0-Beta1
Getting Help
Looking for specic information?
Try the Table of Contents
Look through the genindex
Use the search
See other peoples questions or ask a question yourself.
If you think youve found bugs, search or report bugs in our bug tracker.
This document is being maintained by Raimund Hocke aka RaiMan.
If you have any questions or ideas about this document, you are welcome to contact him directly via this link and the
email behind.
For questions regarding the functions and features of Sikuli itself please use the Sikuli Questions and Answers Board.
For hints and links of how to get more information and help, please see the sidebar.
Documentation for previous versions
The documentation for the versions up to SikuliX-1.0rc3 is still available here.
Contents 1
SikuliX Documentation, Release 1.1.0-Beta1
2 Contents
CHAPTER 1
How to use this document
Sikuli at the top supports Jython (Python for the Java platform currently at language level 2.5 and later 2.7) as the rst
scripting language. If you are new to programming, you can still enjoy using Sikuli to automate simple repetitive tasks
without learning Python using the Sikuli IDE. A good start might be to have a look at the tutorials.
If you plan to write more powerful and complex scripts, which might even be structured in classes and modules, you
have to dive into the Python Language.
The features in Sikuli nally are implemented using Java. So you might as well use Sikuli on this Java level API in
your Java projects or other Java aware environments. Though this documentation is targeted at the scripting people it
contains basic information about the Java level API as well at places, where there are major differences between the
two API levels.
The preface of each chapter in this documentaton briey describes a class or a group of methods regarding its basic
features. It provides general usage information and hints that apply to all methods in that chapter. We recommend to
read carefully before using the related features.
If you are totally new with Sikuli, it might be a good idea to just read through this documentation sequentially. An
alternative way might be to jump to the chapters that you are interested in by scanning the table of contents.
A way in the middle would be reading the core classes: Region, then Match, and nally Screen.
After that, you can go to any places of interest using the table of contents or use the genindex to browse all
classes, methods and functions in alphabetical order.
3
SikuliX Documentation, Release 1.1.0-Beta1
4 Chapter 1. How to use this document
CHAPTER 2
Getting Started
2.1 Tutorials
2.1.1 Tutorials
Hello World (Mac)
Let us begin with a customary Hello World example!
You will learn how to capture a screenshot of a GUI element and write a Sikuli Script to do two things:
1. Click on that element
2. Type a string in that element
The goal of the Hello World script is to automatically type Hello World into the spotlight search box, like this:
Now, open the Sikuli IDE. We begin by taking the screenshot of our target, the Spotlight symbol in the upper-right
corner, so that we can tell Sikuli Script what to look for and click on.
To simulate a mouse click on the Spotlight symbol, we are going to use the click function. To tell Sikuli how the
target looks like, we need to capture the targets image on the screen.
Sikuli IDE provides two methods to capture screen images. The rst method is to click on the camera button in the
toolbar. This will bring you to the screen capturing mode.
The second method is to press a hot-key (Command + Shift + 2). Often the target whose image you wish to capture
may be covered by the Sikuli IDEs window. You can minimize the IDEs window and use this hot-key to switch to
the capturing mode.
In the screen capturing mode, the screen will look darker and freeze momentarily. The entire desktop becomes like a
canvas where you can draw a rectangle around the target you want to capture an image of. In this case, the target is
the spotlight symbol. The cross of red dotted lines shows the center of the rectangle you just drew.
5
SikuliX Documentation, Release 1.1.0-Beta1
After you have drawn (or selected) a rectangle, the image within the rectangle will be captured and inserted into the
script editor at the current cursor position.
Now, you can write the click function using this image as an argument to tell Sikuli to click on spotlight symbol.
As a convenience, Sikuli IDE provides a Command List on the left panel. It shows a list of the most often used
functions. Camera icons in the functions indicate these functions expect a captured image as an argument.
Locate the click() function in the list and click on it. If Auto Capture is on (default), you will be directed to the screen
capturing mode in which you can capture an image of an interface target to be inserted into the click() function as an
argument.
The next step is to tell Sikuli to enter the string Hello World into spotlights search box, which can be done with a
simple type function.
6 Chapter 2. Getting Started
SikuliX Documentation, Release 1.1.0-Beta1
This function will type the string given in the argument into whichever input control that has the focus. After clicking
on the spotlight symbol, we can expect the spotlight search box will be the input that has the focus.
Congratulations! You have just completed your rst Sikuli Script. Press the run button to see this script in action!
Hello World (Windows)
Let us begin with a customary Hello World example!
You will learn how to capture a screenshot of a GUI element and write a Sikuli Script to do two things:
1. Click on that element
2. Type a string in that element
The goal of the Hello World script is to automatically type Hello World into the Start menu search box, like this:
Now, open the Sikuli IDE. We begin by taking the screenshot of our target, the Start menu symbol that is usually
located in the lower-left corner of the desktop. Using this screenshot, we can tell Sikuli script what to click on.
To simulate a mouse click on the Start symbol, we are going to use the click function. To tell Sikuli how the Start
symbol look like, we need to capture its image on teh screen.
Sikuli IDE provides two methods to capture screen images. The rst method is to click on the camera button in the
toolbar. This will bring you to the screen capturing mode.
The second method is to press a hot-key (Ctrl + Shift + 2). Often the target whose image you wish to capture may
be covered by the Sikuli IDEs window. You can minimize the IDEs window and use this hot-key to switch to the
capturing mode.
2.1. Tutorials 7
SikuliX Documentation, Release 1.1.0-Beta1
In the screen capturing mode, the screen will look darker and freeze momentarily. The entire desktop becomes like a
canvas where you can draw a rectangle around the target you want to capture an image of. In this case, the target is
the Start symbol. The cross of red dotted lines shows the center of the rectangle you just drew.
After you have drawn (or selected) a rectangle, the image within the rectangle will be captured and inserted into the
script editor at the current cursor position.
Now, you can write the click function using this image as an argument to tell Sikuli to click on start symbol.
As a convenience, Sikuli IDE provides a Command List on the left panel. It shows a list of the most often used
functions. Camera icons in the functions indicate these functions expect a captured image as an argument.
8 Chapter 2. Getting Started
SikuliX Documentation, Release 1.1.0-Beta1
Locate the click() function in the list and click on it. If Auto Capture is on (default), you will be directed to the screen
capturing mode in which you can capture an image of an interface target to be inserted into the click() function as an
argument.
The next step is to tell Sikuli to enter the string Hello World into search box, which can be done with a simple type
function.
This function will type the string given in the argument into whichever input control that has the focus. After clicking
on the Start symbol, we can expect the search box will be the input that has the focus.
Congratulations! You have just completed your rst Sikuli Script. Press the run button to see this script in action!
Goodbye Trash (Mac)
In this tutorial, we will write a Sikuli Script to automate the operation of clearing the content of the trash bin. On Mac
OS X, this operation takes three steps:
1. Click on the Trash icon in the dock
2. Click on the Empty button in the container window
3. Click on the Empty Trash button to conrm
To automate this operation, rst we need to capture the images of the GUI elements that need to be clicked.
2.1. Tutorials 9
SikuliX Documentation, Release 1.1.0-Beta1
These captured images will be inserted into the Script editor.
Then, we can write a sequence of three click() statements to click on the three elements that need to be clicked in
order.
10 Chapter 2. Getting Started
SikuliX Documentation, Release 1.1.0-Beta1
Notice how closely these three lines of code are mapped to the text description of the three-step operation earlier. We
simply replace the description of each GUI element (e.g., Trash bin) by its image directly. How intuitive it is!
Before running the script, make sure the Trash icon is visible on the screen, otherwise Sikuli Script can not nd it.
Uncheck All Checkboxes
In this tutorial, we will demonstrate how to use a for loop to interact with multiple instances of a GUI component.
Suppose we want to uncheck all the check boxes in a window, such as the Sharing preferences window shown below:
Unfortunately, there is no uncheck all function available. The solution? Write a Sikuli Script to look for ALL the
checked items and uncheck them automatically. The function needed for this operation is findAll().
First, lets capture the screenshot image of a checked item.
2.1. Tutorials 11
SikuliX Documentation, Release 1.1.0-Beta1
Then, we can insert the image into the findAll() function.
findAll() searches the entire screen for all the matching visual patterns and returns a list of locations of those
similar patterns. This capability allows us to obtain all the checked items are on the screen. Then, we can simply write
a for loop in standard Python syntax and call click() on each element in the list.
When this script is executed, Sikuli will nd all the items that are currently checked and click on each item one by one
in the loop.
Check the Right Checkbox
This tutorial demonstrates how to check a particular checkbox when there are multiple checkboxes in a window.
Consider the following window, which is the window for setting the preferences for Dock on Mac, and we want to
check the checkbox indicated by the circle that reads Minimize Windows into Application Icons.
12 Chapter 2. Getting Started
SikuliX Documentation, Release 1.1.0-Beta1
Simply looking for the image of the checkbox like below will not work.
Sikuli does not know which checkbox we are referring to; it simply clicks on the rst one it nds. Below is what we
will do instead.
First we want to obtain the region covering the content area of the Dock pane. One way to accomplish this is to use a
spatial operator to obtain a region below the title bar of the Dock pane. The gure below illustrates this strategy.
2.1. Tutorials 13
SikuliX Documentation, Release 1.1.0-Beta1
The Sikuli script to do this is:
It nds the title bar and then extend the matched region 300 pixels below, using the spatial operator below. The
resulting region is assigned to the variable r, which is the orange rectangle in the gure above.
Next, we can search within the content region r for the label text of the checkbox we want to check and click on it.
If we do click(t), Sikuli will click on the center of the label. However, what we want is to click on the right of the
label where the check box is.
Sikuli IDE provides a convenient interface for specifying where to click relative to the center of a pattern. This is
known as the target offset. The interface is shown below.
14 Chapter 2. Getting Started
SikuliX Documentation, Release 1.1.0-Beta1
In this interface, we can click on the checkbox to indicate the desired location of the click point. In this example,
the offset is then determined to be -137 in x, which means 137 pixels to the left of the center of the text label. After
selecting the offset, the thumbnail in the script editor will be updated with a small red cross to indicate the new click
point.
Then, the call click(t) will do the right thing, clicking on the checkbox instead of the center of the text label.
Working with Sliders
In this tutorial, we will learn how to use dragDrop() and spatial operators by writing a number of scripts to
manipulate sliders.
Suppose we wish to lower the speaking rate of the Text to Speech function. We want to drag the slider to the slow side
in the Speech preferences window shown below.
2.1. Tutorials 15
SikuliX Documentation, Release 1.1.0-Beta1
The function that can perform the dragging is dragDrop(). This function takes two images as arguments. The
rst image describes the source GUI object to drag and the second image describes the appearance of the destination
location where the GUI object should be dragged to and dropped.
Let us capture the source and destination images respectively.
What happen if there are more than two sliders. How can we make sure the right slider is dragged? The above example
works because the particular window only has one slider. How can we deal with cases when there are several instances
of similar looking GUI components? Let us consider the Sound preferences window shown below.
Suppose we wish to lower the Alert volume. To make sure Sikuli Script drags the right slider, we need a way to tell
Sikuli Script to look for the one that is to the right of the Alert volume label, not the one next to the Output volume
label. Sikuli Script provides a set of spatial operators to do exactly this. Here we will apply the right() operator as
follows.
This statement tells Sikuli Script to rst nd the Alert volume label and then nd the slider thumb only within the
region strictly to the right of the result of the rst nd. The slider thumb found is then stored in the variable t. Now
that we have identied the desired slider thumb, we can call dragDrop() to drag it to the left by giving the image
of the Alter volume as the target.
In the above example, we use the image of the Alert volume label to implicitly guide the direction of dragging to the
left. It is also possible to use relative coordinates to explicitly drag to the left, as shown below.
Here, the (x,y) coordinates of the slider thumb are accessible as the attributes of t. We can thus calculate the position
200 pixels to the left of t and ask Sikuli Script to drag the thumb to that position.
16 Chapter 2. Getting Started
SikuliX Documentation, Release 1.1.0-Beta1
Desktop Surveillance
Sikuli Script can be used to perform desktop surveillance. In this tutorial, we will go through a number of exercises
to create scripts that can monitor the screen and notify us when certain interesting visual event happens.
Facebook App
The rst exercise is to create a Facebook app to periodically check our Facebook homepage visually and see if a
particular friend has recently updated the status message. One easy way to detect this event is to look for the friends
face image on our Facebook homepage. If found, the friend must have posted a new status message. If not found, we
should check back again in a few moments.
Lets implement this operation using a while: loop. First we need to capture a screenshot of the friends face image.
To check if the friends face can be seen on the screen, we use exists(), which returns True when the face image
is found. We set the looping condition to Not Found so that the while loop will terminate only when the face image is
found. We add a sleep(n) statement in the body of the loop to introduce a 5 second interval between attempts to
look for the face image on the screen.
Alternatively, Sikuli provides a convenient wait() function that periodically checks the screen to wait for a given
image pattern to appear. Using wait(), the above code can be rewritten as:
The constant FOREVER means we want Sikuli to wait indenitely. If we do not want to wait forever, we can replace
FOREVER with a number to indicate the number of seconds to wait until Sikuli should giveup.
After the while loop exits or the wait function returns, we can call popup() to display a notication message.
This will display a popup message that looks like below:
Now we can run this Sikuli Facebook App, sit back and relax, and get notied when our friend updates his message.
Skype App
In the previous exercise, we wrote a script to detect an images appearance. In this exercise, we will do the opposite
detecting the disappearance of a visual pattern.
2.1. Tutorials 17
SikuliX Documentation, Release 1.1.0-Beta1
Skype is a great tool that allows us to stay in close contact with our friends even if they are in remote parts of the
world. However, there might be some unfortunate circumstances we may want to avoid being seen online by a certain
individual. Perhaps the individual talks too much. Perhaps we owe the individual some money. It would be nice to
know when the individual is ofine so that it is safe to get online. While Skype does provide a feature to notify us
when an individual is online, there is no notication feature when the opposite happens.
An automatic logoff notier would be desirable to deal with this situation. Let us build this tool using Sikuli Script.
Notice that if an individual is no longer online, the combined visual pattern of the green status icon and the individuals
screen name will disappear. Thus, we can take a screenshot that includes both the green icon and the screen name as
follows.
Then, we can write a Sikuli Script to watch for the disappearance of the screenshot image we just captured.
This script is very similar to the one in the previous exercise. The only difference is the removal of the NOT operator
from the condition statement for the while loop, since we are trying to do the opposite.
Another way to wait for the disappearance of an image is to use the waitVanish() function. The above script can
be rewritten as follows:
Bus Arrival Notier
The third exercise is to build a bus arrival notication tool. For many bus riders, online GPS-based tracking services
are very useful. Instead of patiently standing outside at a bust stop, braving the freezing wind in the winter or scorching
sun in the summer, riders can sit comfortably inside in front of their computers, checking emails, updating Facebook
status, or watching YouTube? videos, or what have you. They only need to look at the map every few moments to
check the location of the bus symbol on the map. Only when the bus is close enough do they have to nally get out
and walk to the bus top.
Since we care about whether the bus is getting close to the stop, we only need to look at the neighborhood around the
stop. Thus, we can resize the browser to show just that portion of the map, while leaving a lot of screen space to do
something else, in this case, reading CNN news.
18 Chapter 2. Getting Started
SikuliX Documentation, Release 1.1.0-Beta1
Let us write a Sikuli Script to do the bus tracking for us. It is possible to dene a region and ask Sikuli Script to focus
only on that region to search for a particular visual pattern. This way, Sikuli dont have to waste precious computing
cycles scanning the whole screen. To do so, let us click on the select a region button in the toolbar as indicated
below.
The entire screen will freeze and turn darker, similar to what happen in the screen capture mode. Simply draw a
rectangle to cover the entire neighborhood map. The region covered by this rectangle is visually represented as a
thumbnail image of the entire desktop where the region is shaded in red.
Next, we capture the screenshot image of the bus symbol so that we can tell Sikuli Script to watch for its appearance.
Now. we can write the following script to check the map and watch for the appearance of a bus symbol:
Interpreting the meaning of the while loop is straightforwardwhile it is not the case that a bus symbol can be found
inside the region, sleep for 5 seconds. In other words, the while loop will exit only when the bus symbol is found
inside the region. Then, the popup statement will be executed to notify us the bus has arrived.
2.1. Tutorials 19
SikuliX Documentation, Release 1.1.0-Beta1
Again, the same effect can be achived using wait(). The script can be rewritten as:
Resizing a Window
This tutorial demonstrates how to use Sikuli script to resize a window by dragging its bottom-right corner. In order to
do so, you have to evaluate the current position of this corner on the screen, move the mouse pointer to the applicable
click point and then perform one or more drag actions.
This example denes a function resizeApp to enlarge a window on a Mac, and then call the function to enlarge the
size of Safari by 50x50 pixels. There are several approaches to do this task. The simplest one is directly looking for
the corner and drag it to enlarge the window.
In addition to this simplest approach, we also want to show you more possible ways to do the same thing. The
following example demonstrates how to use the spatial operators to extend or restrict your searching regions.
The approach to nd this corner is to rst identify the most characteristic corner of the window (mostly the one with
the window buttons) and then try to nd the other relevant corners, to be sure you get the click point you need. The
strategy is illustrated below. We want to nd the top-right corner, then go to the right to nd the top-right corner, then
go below to nd the bottom-right corner.
The script that implements this plan can be written as below:
The workow and the mouse-move to the click point can be compressed to a one-liner (no check on the title in this
case).
20 Chapter 2. Getting Started
SikuliX Documentation, Release 1.1.0-Beta1
General comments:
whenever possible the nd operations are restricted to a region, that should contain the relevant match, to
minimize the risk, that something else is found. On top this speeds it up.
this example could easily be turned into a helper function, where all images and variables can be given as
parameters.
the click point was evaluated in the IDE using the preview window, but then put in the code for exibility and
readability.
the low level mouse functions are used, so you would be more exible with continuous movements.
the development approach was, to rst get everything running to evaluate the relevant corner. In this phase, the
comments having a # debug were all uncommented to have some feedback. Especially setShowActions() and
exit() are very helpful.
Slide: Practical Sikuli: using screenshots for GUI automation and testing.
2.2 FAQ
2.2.1 Can I do X or Y or Z in Sikuli?
If you are wondering if Sikuli can do X or Y, these two rules apply:
If you can do X with Java, you can also do it in Sikuli by simply adding the respective Java resources to the
classpath (the standard Java classes are already there). For example, you can create a GUI with Java Swing, so
you can do it in the same way in Sikuli.
If you can do X with Python, you probably can do it in Sikuli as well. This actually depends on what Python
modules you use. Sikuli is with the contained Jython on language level 2.5 and language level 2.7 will be useable
soon. So everything available in the respective Python base package is available in Sikuli too. If modules are
written in pure Python, you can use them in Sikuli as well. A typical example are the Excel access modules xlrd
and xlwt. If they are written in C or depend on C-based modules, unfortunately, you cant (e.g. the support for
Win32API calls).
Can I write a loop in Sikuli?
Yes. Sikuli uses Jython (Python). You can use all constructs that are available in standard Python. See Jythons While
loop and For loop.
Can I create a GUI in Sikuli?
Yes, you can create GUIs with Java Swing or any other Java/Jython GUI toolkits. See Jythons Swing examples for
examples.
Can I connect to MySQL/MS SQL/PostgreSQL or any database systems in Sikuli?
You can use JDBC or zxJDBC.
Can I read/write les in Sikuli?
Yes. See Jythons File I/O.
2.2. FAQ 21
SikuliX Documentation, Release 1.1.0-Beta1
2.2.2 How to run Sikuli from Command Line
SikuliX IDE or Script can be used on command line to run a Sikuli script or open it in the IDE.
The usage on each platform:
Windows
PATH-TO-SIKULIX/runIDE.cmd [options] or PATH-TO-SIKULIX/runScript.cmd [options]
Mac OS X
PATH-TO-SIKULIX/runIDE [options] or PATH-TO-SIKULIX/runScript [options]
Linux
PATH-TO-SIKULIX/runIDE [options] or PATH-TO-SIKULIX/runScript [options]
runIDE(.cmd) without any options simply starts SikuliX IDE.
PATH-TO-SIKULIX is the folder containing the Sikuli stuff after having run setup.
Command Line Options
SikuliX IDE or Script
-- <arguments>
the space delimeted and optionally quoted arguments are passed to Jythons sys.argv and hence are available to
your script
-h,-help
print the help message showing the available options
-r,-run <sikuli-folder/file>
run .sikuli or .skl le
-c,-console
all output goes to stdout
-i,-interactive
open an interactive Jython session that is prepared for the usage of the Sikuli features
2.2.3 How to use Sikuli Script in your JAVA programs
The core of Sikuli Script is written in Java, which means you can use Sikuli Script as a standard JAVA library in your
program. This document lets you know how to do that.
After having setup Sikuli on your system, as recommended on the download page, you have to do the following:
1. Include sikuli-script.jar in the CLASSPATH of your Java project.
We use Eclipse as an example. After adding sikuli-script.jar or preferably sikuli-java.jar as a library reference into
your project, the project hierarchy should look like this.
22 Chapter 2. Getting Started
SikuliX Documentation, Release 1.1.0-Beta1
2. Import the Sikuli classes you need
You can simply import org.sikuli.script.* or import the classes you need. In most cases, you would need at least
Region or Screen.
3. Write code!
Here is a hello world example on Mac. The program clicks on the spotlight icon on the screen, waits until spotlights
input window appears, activates it by clicking and then writes hello world into the eld and hits ENTER.
import org.sikuli.script.
*
;
public class TestSikuli {
public static void main(String[] args) {
Screen s = new Screen();
try{
s.click("imgs/spotlight.png");
s.wait("imgs/spotlight-input.png");
s.click();
s.write("hello world#ENTER.");
}
catch(FindFailed e){
e.printStackTrace();
}
}
}
See also
Be aware, that some method signatures in the Java API differ from the scripting level.
Javadoc of SikuliX.
Everything about Sikuli Script.
2.2. FAQ 23
SikuliX Documentation, Release 1.1.0-Beta1
2.2.4 How to use Sikuli together with other IDEs
New in version X1.0-rc2.
currently available for Netbeans and Eclipse.
In this section we want to talk about requirements and todos, when you want to develop Sikuli scripts in other IDEs
than the one that comes with the distribution. You should read further, if you want to develop at least a part of your
package using Python language. If you only want to develop in Java using the Java level API of Sikuli, then you
should look here: How to use Sikuli Script in your JAVA programs.
Very detailed background information and helpful examples can be found in the Jython e-book Chapter 11: Using
Jython in a IDE. So we will concentrate on the key points here.
The following approaches are evaluated and tested on Mac OS X 10.6 and Windows 7 32Bit. Linux users have to nd
the appropriate setups themselves - but they should be similar or even identical.
These and more might be your reasons:
you want to stick with your favorite IDE
you want more features, than Sikuli-IDE currently has
you want to develop packages/applications combining Sikuli script and Java or even other languages
you want to develop a Sikuli Extension
you need a debugger for your script/package/application
you want to integrate with a versioning system
you want to do professional unittesting
General Requirements and Comments
Install Sikuli
Windows: install Sikuli X using the installer (the system %PATH% will be set as needed) There is a new
environment variable %SIKULI_HOME% that is expected to contain the directory, where Sikuli X is
installed. You have to set it, if you have Sikuli X in a different place.
Be aware: using the zipped version, you have to take care for %PATH%and %SIKULI_HOME%yourself.
Mac: have Sikuli-IDE.app in its standard place /Applications. There is generally no need to put it elsewhere.
Linux: you generally have to take care for the environment yourself.
Meanings of shortcuts used:
path-to/sikuli-script.jar if we use this, it should be replaced by the absolute path to the le sikuli-script.jar,
where you installed Sikuli.
Images and importing other Sikuli scripts
It is recommended to carefully read through Importing other Sikuli Scripts (reuse code and images), since you need to
have a concept how to capture, how to name and where to store your images. As a rst start, just decide to use Sikuli
IDE in parallel to capture and store your images and use the import feature or the image path to make them available
in your script.
24 Chapter 2. Getting Started
SikuliX Documentation, Release 1.1.0-Beta1
Netbeans
This is based on Netbeans 6.9. We suppose you have it running in a setup that allows to at least develop Java
applications.
Since this is not a tutorial how to use NetBeans in general, you should be familiar with the basic concepts.
Install Python plugin
Since the top level language used by Sikuli script is Python, you need the Python plugin that comes bundled with
Jython 2.5.1.
In the Python plugin NetBeans is not strict in naming: though they mainly talk about Python, Jython is meant as well.
The Python interpreter to use is selected on the project level.
Go to menu Tools -> Plugins -> Available Plugins
In the list select everything that is in the category Python and install. If suggested by the install process always restart
Netbeans.
Congure for using Sikuli script features at runtime
The plugin itself has nothing to congure, that is of value for the Sikuli usage. Everything is done on the project level,
though some basic preferences act like plugin globals - we just have to know and accept this ;-)
These are the setup steps:
Start a new project: File -> New Project -> Categories: Python -> Python Project -> click Next
In the second step Name and Location (last line), select as Python Platform: Jython 2.5.1.
you might want to click Make Default, if you have real Python also available (Remember: Sikuli scripts cannot
be run with real Python!)
To integrate Sikuli: click the button Manage (only needed with the rst project - it is remembered)
on the tab Python Path we need an additional entry, that points to path-to/sikuli-script.jar/Lib.
Windows: click the button Add and click through to path-to/sikuli-script.jar. Before clicking Open, edit
the lename so that it shows sikuli-script.jar\Lib.
Mac: since the Netbeans le dialog does not allow to dive into a bundle like Sikuli-IDE.app, we have to
edit the preference le directly. Since it is the same with the Java path, we have documented it seperatly
below Mac/Netbeans: select library path.
on the tab Java Path we need an additional entry, that points to path-to/sikuli-script.jar. If you have other Java
libraries, that you need in your project, add them here too.
Windows: click the button Add and click through to path-to/sikuli-script.jar. Clicking Open.
Mac: same again, see below Mac/Netbeans: Select library path.
Now you are able to run your rst script. Remember, that in every script including the main script, that you are editing
now, as the rst line you need from sikuli.Sikuli import *, to have access to the Sikuli features at runtime.
Everytime later on you might come back to the projects preferences with File -> Project Properties (your-projects-
name) . You will nd the above information in the categorie Python.
Prepare to use Code Completion for the Sikuli methods
The NetBeans editor in the Python plugin editor is not able to recognize Python classes and methods, as long as the
respective source code is embedded inside a jar-le as it is the fact with Sikuli. If you want Code Completion to work,
you have to extract the folder Lib/sikuli from sikuli-script.jar, place it somewhere and add the reference to this folder
to the Python Path in the projects preferences (see above: Congure Python Path). To extract the folder Lib/sikuli
2.2. FAQ 25
SikuliX Documentation, Release 1.1.0-Beta1
from sikuli-script.jar you might use the jar utility, that comes with the Java JDK or any other utility, that allows to
unjar a jar-le.
This is a Mac example how to use the jar uility, supposing it can be found on the system path:
in a Terminal window go to an appropriate folder, that should contain Lib/sikuli afterwards
run: jar -xf /Applications/Sikuli-IDE.app/Contents/Resources/Java/sikuli-script.jar Lib/sikuli
You might decide, to use a folder, that is already on the Python path. One folder that is designated by Jython to contain
complementary sources, that needs to be imported, is the folder Lib/site-packages in the Jython installation directory.
If you copy the folder sikuli here, you do not need an additional Python path entry. Mac/NetBeans: Select library
path
To perpare the preference le modication we rst add the path to Sikuli-IDE.app on both tabs: click the button Add
and click through to /Applications/Sikuli-IDE.app. Depending on your selected language, the folder Applications is
named like you see it in the Finder. Save the preferences by clicking close and/or ok.
The preference le is located at /Users/your-name/.netbeans/6.9/build.properties. Since it is hidden, you might have
to use the Finder option Go to Folder (Shift-Command-G), to open the folder /Users/your-name/.netbeans/.
Open the le build.properties in your favorite editor and search for the text /Applications/Sikuli-IDE.app. It should be
at the end of 2 lines beginning with pythonplatform..
On the line beginning pythonplatform. ... .javalib= extend /Applications/Sikuli-IDE.app so that it reads:
/Applications/Sikuli-IDE.app/Contents/Resources/Java/sikuli-script.jar.
On the line beginning pythonplatform. ... .pythonlib= extend /Applications/Sikuli-IDE.app so that it reads:
/Applications/Sikuli-IDE.app/Contents/Resources/Java/sikuli-script.jar/Lib.
Save the le and restart Netbeans. It is a good idea to check, that the correct entries are found on the 2 tabs now.
Eclipse
This is based on Eclipse Helios 3.6. We suppose you have it running in a setup that allows to at least develop Java
applications.
Since this is not a tutorial how to use Eclipse in general, you should be familiar with the basic concepts.
Install Python plugin (PyDev) and Jython
Since the top level language used by Sikuli script is Python, you need the Python plugin PyDev. Different from
NetBeans, though PyDev is prepared to use Jython as interpreter, you have to install Jython on your own seperately.
So rst install Jython, by downloading the version you want to use from the Jython download page. Sikuli currently
is based on Jython 2.5.1, but you may choose Jython 2.5.2 as well. Install it according to Jythons installation HowTo.
After installation, make a test from a commandline by typing jython.bat (Windows) or jython (Mac/Linux) to open an
interactive Jython session.
You might type the following (<enter> means pressing the enter/return key)
import os <enter>
import sys <enter>
for e in sys.path: print e <enter> <enter>
type exit() <enter> (to leave interactive Jython)
This shows the current Jython conguration of the Python path. Remember the place where Jython is installed.
Now we install the PyDev plugin from inside Eclipse: Menu Help -> Eclipse Marketplace. Either search it or nd
it on the Popular tabs lower part. Simply click the Install button and do what you are asked to do ;-)
26 Chapter 2. Getting Started
SikuliX Documentation, Release 1.1.0-Beta1
The next step is to tell Eclipse PyDev, where it can nd the Jython interpreter. Navigate to the Eclipse Preferences
pane and open the category PyDev and inside the subcategory Interpreter-Jython. First try to Auto Cong by clicking
the appropriate button. If this does not work, click the button New, name the entry and navigate to the folder, where
Jython was installed and select jython.jar. After clicking ok, a window Selection needed might come up: click Select
all and ok to nalize this step.
Other options with PyDev are available, but not relevant for our actual matter (see documentation).
Congure for using Sikuli script features at runtime
Again we have a difference to Netbeans: The PyDev plugin does not allow library references to folders inside of
jar-les in the respective conguration dialog (it does not insert them to the Python path), though Jython itself accepts
them, when specied on the Python path. So if you want to run and debug your script in Eclipse, you have to extract
the folder Lib/sikuli from sikuli-script.jar (nd one HowTo here).
As with Netbeans, the library conguration is done in the project itself. So we open a new project:
Menu File -> New -> Project -> PyDev -> (open sublist) -> PyDev Project -> click Next button.
On the conguration pane name your project, select Jython as Project type, grammar version 2.5 (higher Python
language versions are not supported by Jython 2.5.x) and click the Finish button. Your project is created. Add at least
one you-name-it.py le to the source folder and put from sikuli.Sikuli import * as the rst line.
In the last step, we tell PyDev, where to nd the Sikuli libraries.
Goto Menu Project -> Properties -> select category PyDev - PYTHONPATH and go to the tab External Libraries.
We need a reference to path-to/sikuli-script.jar and another one to the extracted folder Lib containing the folder sikuli.
reference to path-to/sikuli-script.jar
Windows: click button Add zip/jar/egg and select sikuli-script.jar from the Sikuli installation.
Mac: As with NetBeans, the le dialog does not allow to step inside Sikuli-IDE.app. So we use the trick,
to dene a String Substitution Variable: on the respective tab click Add variable, name it e.g. sikuli-script
and enter as value: /Applications/Sikuli-IDE.app/Contents/Resources/Java/sikuli-script.jar. Go back to the
tab External Libraries and click Add based on variable. In the entry eld enter: ${sikuli-script} and
click OK.
reference to the extracted folder Lib containing the folder sikuli
click Add source folder and select the folder Lib in the place you had it extracted.
this is not needed, if you have moved the extracted folder sikuli to a folder, that is already on the Python
path (e.g. jython-intallation/Lib/site-packages).
Now you are able to run your rst script. Remember, that in every script including the main script, that you are editing
now, as the rst line you need from sikuli.Sikuli import *, to have access to the Sikuli features at runtime.
Everytime later on you might come back to the projects preferences with Project -> Properties.
Code Completion works from the start without any further conguration and even steps into the Java classes where
appropriate.
Read more FAQs on Launchpad
2.2. FAQ 27
SikuliX Documentation, Release 1.1.0-Beta1
28 Chapter 2. Getting Started
CHAPTER 3
Complete Guide
3.1 Everything about Sikuli Script
3.1.1 Global Functions and Features
Importing other Sikuli Scripts (reuse code and images)
When getting more experienced with scripting or when you are used to structure your solutions into a modular system,
you might want to have access to the related features of the programming environment - in this case the Python/Jython
features of module support - for your scripts too.
This is possible with Sikuli X:
import other .sikuli in a way that is fully compatible with Python import
import a python module structure including underlying Java classes from a jar-le, that is dynamically loaded
using the function load(jar-file)
automatically access images contained in the imported .sikuli (no need to use setBundlePath())
Note: Currently a .skl cannot be imported. As a circumvention it is up to you to unzip the .skl on the y (e.g. with
gzip on the command line) to a place of your choice as .sikuli (e.g. temp directory) and import it from there.
The prerequisites:
the directories/folders containing your .sikulis you want to import have to be in sys.path (see below: Usage)
New in version X1.0-rc3.
Sikuli automatically nds other Sikuli scripts in the same directory, when they are imported
your imported script must contain (recommendation: as rst line) the following statement: from sikuli
import
*
(this is necessary for the Python environment to know the Sikuli classes, methods, functions and
global names)
Usage:
Add the path to the Sikuli module into sys.path (since X-1.0rc3: If the modules to import are in the same
directory as the main script, skip this step.)
Import your .sikuli using just its name. For example, to import a_module.sikuli, just write import a_module.
the example contains a recommendation to avoid double entries:
29
SikuliX Documentation, Release 1.1.0-Beta1
# an example - choose your own naming
# on Windows
myScriptPath = "c:\\someDirectory\\myLibrary"
# on Mac/Linux
myScriptPath = "/someDirectory/myLibrary"
# all systems
if not myScriptPath in sys.path: sys.path.append(myScriptPath)
# supposing there is a myLib.sikuli
import myLib
# supposing myLib.sikuli contains a function "def myFunction():"
myLib.myFunction() # makes the call
Note on contained images: Together with the import, Sikuli internally uses the new SIKULI_IMAGE_PATH to make
sure that images contained in imported .sikulis are found automatically.
Some comments for readers not familiar with Python import
An import is only processed once (the rst time it is found in the program ow). So be aware:
If your imported script contains code outside of any function denitions ( def() ), this code is only
processed once at the rst time, when the import is evaluated
Since the IDE does not reload the modules while running a script each time, if you are changing imported
scripts while they are in use, you have to use the Jythons reload() function:
# instead of: import module
import module
reload(module)
# instead of: from module import
*
import module
reload(module)
from module import
*
Python has a so called namespace concept: names (variables, functions, classes) are only known in its names-
pace your main script has its own namespace
Each imported script has its own namespace. So names contained in an imported script have to be qualied
with the module name (e.g. myLib.myFunction() )
You may use from myLib import
*
, which integrates all names from myLib into your current
namespace. So you can use myFunction() directly. When you decide to use this version, be sure
you have a naming convention that prevents naming conicts.
Another example: Importing from the same directory
This approach allows to develop a modularized script app that is contained in one directory. This directory can be
moved around with no changes and even distributed as a zipped le:
# works on all platforms
import os
# get the directory containing your running .sikuli
myPath = os.path.dirname(getBundlePath())
if not myPath in sys.path: sys.path.append(myPath)
# now you can import every .sikuli in the same directory
import myLib
30 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
New in version X1.0-rc3.
Since scripts in same directory are found automatically:
# nothing else needed
# now you can import every .sikuli in the same directory
import myLib
Loading a jar-le containing Java/Python modules
New in version X1.0-rc2.
load(jar-le)
Loads a jar-le and puts the absolute path to it into sys.path, so the Java or Python code in that jar-le can be
imported afterwards.
Parameters jar-le either a filename.jar without any path or the absolute path to
filename.jar
Returns True if the le was found, otherwise False
Note: if no path is specied, Sikuli rst looks into the bundle (the Sikuli folder of the running script) and then
into the extensions folder. (more information: Sikuli Extensions)
Controlling Sikuli Scripts and their Behavior
setShowActions(False | True)
If set to True, when a script is run, Sikuli shows a visual effect (a blinking double lined red circle) on the spot
where the action will take place before executing actions (e.g. click(), dragDrop(), type(), etc) for
about 2 seconds in the standard (see Settings.SlowMotionDelay ). The default setting is False.
exit([value ])
Stops the script gracefully at this point. The value is returned to the calling environment.
class Settings
New in version X1.0-rc2.
Settings.ActionLogs
Settings.InfoLogs
Settings.DebugLogs
Either option might be switched on (True) or off (False), to show or hide the respective message type in the IDE
console or on command line ([log], [info], [debug]).
Settings.MinSimilarity
The default minimum similiarty of nd operations. While using a Region.find() operation, if only an
image le is provided, Sikuli searches the region using a default minimum similarity of 0.7.
Settings.MoveMouseDelay
Control the time taken for mouse movement to a target location by setting this value to a decimal value (default
0.5). The unit is seconds. Setting it to 0 will switch off any animation (the mouse will jump to the target
location).
As a standard behavior the time to move the mouse pointer from the current location to the target location given
by mouse actions is 0.5 seconds. During this time, the mouse pointer is moved continuosly with decreasing
speed to the target point. An additional benet of this behavior is, that it gives the active application some time
to react on the previous mouse action, since the e.g. click is simulated at the end of the mouse movement:
mmd = Settings.MoveMouseDelay # save default/actual value
click(image1) # implicitly wait 0.5 seconds before click
Settings.MoveMouseDelay = 3
3.1. Everything about Sikuli Script 31
SikuliX Documentation, Release 1.1.0-Beta1
click(image2) # give app 3 seconds time before clicking again
Settings.MoveMouseDelay = mmd # reset to original value
Settings.DelayAfterDrag
Settings.DelayBeforeDrop
DelayAfterDrag species the waiting time after mouse down at the source location as a decimal value (seconds).
DelayBeforeDrop species the waiting time before mouse up at the target location as a decimal value (seconds).
Usage: When using Region.dragDrop() you may have situations, where the operation is not processed
as expected. This may be due to the fact, that the Sikuli actions are too fast for the target application to re-
act properly. With these settings the waiting time after the mouse down at the source location and before
the mouse up at the target location of a dragDrop operation are controlled. The standard settings are 0.3
seconds for each value. The time that is taken, to move the mouse from source to target is controlled by
Settings.MoveMouseDelay:
# you may wish to save the actual settings before
Settings.DelayAfterDrag = 1
Settings.DelayBeforeDrop = 1
Settings.MoveMouseDelay = 3
dragDrop(source_image, target_image)
# time for complete dragDrop: about 5 seconds + search times
Settings.SlowMotionDelay
Control the duration of the visual effect (seconds).
Settings.WaitScanRate
Settings.ObserveScanRate
Specify the number of times actual search operations are performed per second while waiting for a pattern to
appear or vanish.
As a standard behavior Sikuli internally processes about 3 search operations per second, when processing a
Region.wait(), Region.exists(), Region.waitVanish(), Region.observe()). In cases
where this leads to an excessive usage of system ressources or if you intentionally want to look for the visual
object not so often, you may set the respective values to what you need. Since the value is used as a rate per
second, specifying values between 1 and near zero, leads to scans every x seconds (e.g. specifying 0.5 will lead
to scans every 2 seconds):
def myHandler(e):
print "it happened"
# you may wish to save the actual settings before
Settings.ObserveScanRate = 0.2
onAppear(some_image, myHandler)
observe(FOREVER, background = True)
# the observer will look every 5 seconds
# since your script does not wait here, you
# might want to stop the observing later on ;-)
New in version X1.0-rc2.
Settings.ObserveMinChangedPixels
The minimum size in pixels of a change to trigger a change event when using Region.onChange() without
specifying this value. The default value is 50.
32 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
Controlling Applications and their Windows
Here we talk about opening or closing other applications, switching to them (bring their windows to front) or accessing
an applications windows.
The three global functions openApp(), switchApp() and closeApp() introduced in Sikuli 0.9 and 0.10 are
still valid in the moment, but they should be considered as deprecated. They are being replaced by a new App class
introduced in Sikuli X. This class makes it possible to treat a specic application as an object with attributes and
methods. We recommend to switch to the class App and its features, the next time you work with one of your existing
scripts and in all cases, when developing new scripts.
General hint for Windows users on backslashes \ and double apostrophes
In a Sikuli script in normal strings enclosed in (double apostrophes), these special characters \ and have to be
escaped using a backslash, when you have them inside the string. So for one backslash you need \\ and for one you
need \. In a string enclosed in (single apostrophes), a has to be \ and a is taken as such.
To avoid any problems, it is recommended to use the raw string rsome text with \\ and " ..., since
there is no need for escaping. This is especially useful, when you have to specify Windows paths or want to setup
command lines for use with App.open(), openApp(), os.popen or Jythons Subprocess module.
a ctive command line example:
cmd = rc:\Program Files\myapp.exe -x "c:\Some Place\some.txt" >..\log.txt
openApp(cmd)
This is a comparism of old (xxxApp) and new (App.xxx) functions:
Open an application: openApp() > App.open()
Switch to an application or application window: switchApp() > App.focus()
Close an application: closeApp() > App.close()
openApp(application)
Open the specied application.
Parameters application a string containing the name of an application (case-insensitive), that can
be found in the path used by the system to locate applications. Or it can be the full path to an
application.
Note for Windows: (since X-1.0rc3) The string may contain commandline parameters for the
specied program or batch le after the name or full path.
This function opens the specied application and brings its windows to the front. This is equivalent to
App.open(). Depending on the system and/or the application, this function may switch to an already opened
application or may open a new instance of the application.
Examples:
# Windows: opens command prompt (found through PATH)
openApp("cmd.exe")
#Windows (since X-1.0rc3): with parameters (no sense, only to show ;-)
openApp(rcmd.exe /c start c:\Program Files\myapp.bat)
# Windows: opens Firefox (full path specified)
openApp("c:\\Program Files\\Mozilla Firefox\\firefox.exe")
# Mac: opens Safari
openApp("Safari")
3.1. Everything about Sikuli Script 33
SikuliX Documentation, Release 1.1.0-Beta1
switchApp(application)
Switch to the specied application.
Parameters application the name of an application (case-insensitive) or (part of) a window title
(Windows/Linux).
This function switches the focus to the specied application and brings its windows to the front. This function
is equivalent to App.focus().
On Windows/Linux, the window is the one identied by the application string. This string is used to search the
title text of all the opened windows for any part of the title matching the string. Thus, this string needs not be an
applications name. For example, it can be a lename of an opened document that is displayed in the title bar. It
is useful for choosing a particular window out of the many windows with different titles.
On Mac, the application string is used to identify the application. If the application has multiple windows
opened, all these windows will be brought to the front. The relatively ordering among these windows remain
the same.
Example:
# Windows: switches to an existing command prompt or starts a new one
switchApp("cmd.exe")
# Windows: opens a new browser window
switchApp("c:\\Program Files\\Mozilla Firefox\\firefox.exe")
# Windows: switches to the frontmost opened browser window (or does nothing
# if no browser window is currently opened)
switchApp("mozilla firefox")
# Mac: switches to Safari or starts it
switchApp("Safari")
closeApp(application)
Close the specied application.
Parameters application the name of an application (case-insensitive) or (part of) a window title
(Windows/Linux)
This function closes the application indicated by the string application (Mac) or the windows whose titles
contain the string application (Windows/Linux). this function is equivalent to App.close(). On Win-
dows/Linux, the application itself may be closed if the main window is closed or if all the windows of the
application are closed.
Example:
# Windows: closes an existing command prompt
closeApp("cmd.exe")
# Windows: does nothing, since text can not be found in the window title
closeApp("c:\\Program Files\\Mozilla Firefox\\firefox.exe")
# Windows: stops firefox including all its windows
closeApp("mozilla firefox")
# Mac: closes Safari including all its windows
closeApp("Safari")
run(command)
Run command in the command line
34 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
Parameters command a command that can be run from the command line.
This function executes the command and the script waits for its completion.
The Application Class
class App
Sikuli-X introduces the new class called App to provide a more convenient and exible way to control the application
and its windows.
go directly to the methods
Using class methods or instance methods
Generally you have the choice between using the class methods (e.g.
App.open("application-identifier")) or rst create an App instance and use the instance meth-
ods afterwards (e.g. myApp = App("application-identifier") and then later on myApp.open()). In
the current state of the feature developement of the class App, there is no recomendation for a preferred usage. The
only real difference is, that you might save some ressources, when using the instance approach, since using the class
methods produces more intermediate objects. How to create an App instance
The basic choice is to just say someApp = App("some-app-identifier") and you have your app instance,
that you can later on use together with its methods, without having to specify the string again.
Additionally App.open("some-app-identifier") and App.focus("some-app-identifier") re-
turn an app instance, that you might save in a variable to use it later on in your script.
Differences between Windows/Linux and Mac
Windows/Linux: Sikulis strategy on these systems in the moment is to rely on implicit or explicit path specications to
nd an application, that has to be started. Running applications can either be identied using their PID (process ID)
or by using the window titles. So using a path specication will only switch to an open application, if the application
internally handles the more than one instance situation.
You usually will use App.open("c:\\Program Files\\Mozilla Firefox\\Firefox.exe") to start
Firefox. This might open an additional window. And you can use App.focus("Firefox") to switch to the
frontmost Firefox window (which has no effect if no window is found). To clarify your situation you may use the
new window() method, which allows to look for existing windows. The second possible approach is to store the App
instance, that is returned by App.open(), in a variable and use it later on with the instance methods (see examples
below).
If you specify the exact window title of an open window, you will get exactly this one. But if you specify some
text, that is found in more than one open window title, you will get all these windows in return. So this is good e.g.
with Firefox, where every window title contains Mozilla Firefox, but it might be inconvenient when looking for
Untitled which may be in use by different apps for new documents. So if you want exactly one specic window, you
either need to know the exact window title or at least some part of the title text, that makes this window unique in the
current context (e.g. save a document with a specic name, before accessing its window).
On Mac OS X, on the system level the information is available, which windows belong to which applications. Sikuli
uses this information. So by default using e.g. App.focus("Safari") starts Safari if not open already and
switches to the application Safari if it is open, without doing anything with its windows (the z-order is not touched).
Additionally, you can get all windows of an application, without knowing its titles.
Note on Windows: when specifying a path in a string, you have to use \ (double backslash) for each (backslash) e.g.
myPath = "c:\\Program Files\\Sikuli-IDE\\Lib\\" )
class App
3.1. Everything about Sikuli Script 35
SikuliX Documentation, Release 1.1.0-Beta1
classmethod open(application)
Usage: App.open(application)
Open the specied application.
Parameters application The name of an application (case-insensitive), that can be found in
the path used by the systemto locate applications, or the full path to an application (Windows:
use double backslash \ in the path string to represent a backslash)
Note for Windows: (since X-1.0rc3) The string may contain commandline parameters for
the specied program or batch le after the name or full path (see: openApp())
Returns an App object, that can be used with the instance methods.
This method is functionally equivalent to openApp(). It opens the specied application and brings
its window the front. Whether this operation switches to an already opened application or opens a new
instance of the application depends on the system and application.
open()
Usage: someApp.open() where App instance someApp was created before.
Open this application.
classmethod focus(application)
Usage: App.focus(application)
Switch the focus to an application.
Parameters application The name of an application (case-insensitive) or (part of) a window
title (Windows/Linux).
Returns an App object, that can be used with the instance methods.
focus()
Usage: someApp.focus() where App instance someApp was created before.
Switch the focus to this application.
classmethod close(application)
Usage: App.close(application)
Close the specied application.
Parameters application The name of an application (case-insensitive) or (part of) a window
title (Windows/Linux).
This method is functionally equivalent to closeApp(). It closes the given application or the matching
windows (Windows/Linux). It does nothing if no opened window (Windows/Linux) or running application
(Mac) can be found. On Windows/Linux, whether the application itself is closed depends on weather all
open windows are closed or a main window of the application is closed, that in turn closes all other opened
windows.
close()
Usage: someApp.close() where App instance someApp was created before.
Close this application.
classmethod focusedWindow()
Usage: App.focusedWindow()
Identify the currently focused or the frontmost window and switch to it. Sikuli does not tell you, to which
application this window belongs.
Returns a Region object representing the window or None if there is no such window.
36 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
On Mac, when starting a script, Sikuli hides its window and starts processing the script. In this moment,
no window has focus. Thus, it is necessary to rst click somewhere or use App.focus() to focus on a
window. In this case, this method may return None.
On Windows, this method always returns a region. When there is no window opened on the desktop, the
region may refer to a special window such as the task bar or an icon in the system tray.
Example:
# highlight the currently fontmost window for 2 seconds
App.focusedWindow().highlight(2)
# save the windows region before
firstWindow = App.focusedWindow()
firstWindow.highlight(2)
window([n])
Usage 1: App(application).window([n]) an App instance is created on the y.
Usage 2: someApp.window([n]) where App instance someApp was created before.
Get the region corresponding to the n-th window of this application (Mac) or a series of windows with the
matching title (Windows/Linux).
Parameters n 0 or a positive integer number. If ommitted, 0 is taken as default.
Returns the region on the screen occupied by the window, if such window exists and None if
otherwise.
Below is an example that tries to open a Firefox browser window and switches to the address eld (Win-
dows):
# using an existing window if possible
myApp = App("Firefox")
if not myApp.window(): # no window(0) - Firefox not open
App.open("c:\\Program Files\\Mozilla Firefox\\Firefox.exe")
wait(2)
myApp.focus()
wait(1)
type("l", KEY_CTRL) # switch to address field
Afterwards, it focuses on the Firefox application, uses the window() method to obtain the region of the
frontmost window, applies some operations within the region, and nally closes the window:
# using a new window
firefox = App.open("c:\\Program Files\\Mozilla Firefox\\Firefox.exe");
wait(2)
firefox.focus()
wait(1)
# now your just opened new window should be the frontmost
with firefox.window(): # see the general notes below
# some actions inside the window(0)s region
click("somebutton.png")
firefox.close() # close the window - stop the process
Below is another example that highlights all the windows of an application by looping through them (Mac):
# not more than 100 windows should be open ;-)
myApp = App("Safari")
for n in range(100):
w = myApp.window(n)
3.1. Everything about Sikuli Script 37
SikuliX Documentation, Release 1.1.0-Beta1
if not w: break # no more windows
w.highlight(2) # window highlighted for 2 second
General notes:
Be aware, that especially the window handling feature is experimental and under further development.
Especially on Windows be aware, that there might be many matching windows and windows, that
might not be visible at all. Currently the window() function has no feature to identify a special
window besides returning the region. So you might need some additional checks to be sure you are
acting on the right window.
Windows/Linux: The close() function currently kills the application, without closing its windows
before. This is an abnormal termination and might be recognized by your application at the next start
(e.g. Firefox usually tries to reload the pages).
Even if the windows are hidden/minimized, their region that they have in the visible state is returned.
Currently there is no Sikuli feature, to decide wether the given window(n) is visible or not or if it
is currently the frontmost window. The only guarentee: window()/window(0) is the topmost
window of an application (Mac) or a series of matching windows (Windows/Linux).
Currently there are no methods available to act on such a window (resize, bring to front, get the
window title, ...).
Some tips:
Check the position of a windows returned region: some apps hide there windows by giving them
outside coordinates (e.g. negative)
Check the size of a windows returned region: normally your app windows will occupy major parts of
the screen, so a windows returned region of e.g. 150x30 might be some invisible stuff or an overlay
on the real app window (e.g. the search in history input eld on the Safari Top-Sites page, which is
reported as windows(0))
If you have more than one application window, try to position them at different coordinates, so you
can decide which one you act on in the moment.
It is sometimes possible to use the OCR text extraction feature Region.text() to obtain the win-
dow title.
Interacting with the User
New in version X1.0-rc3.
popup(text[, title ])
Display a dialog box with an OK button and text as the message. The script then waits for the user to click the
OK button.
Parameters
text text to be displayed as message
title optional title for the messagebox
Example:
popup("Hello World!\nHave fun with Sikuli!")
A dialog box that looks like below will popup (Note: n can break a line).
38 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
input([text ][, default ])
Display a dialog box with an input eld, a Cancel button, and an OK button. The optional text can be displayed
as a caption. The script then waits for the user to click either the Cancel or the OK button.
Parameters
text optional text to be displayed as message
default optional preset text for the input eld (new in later then X-1.0rc3)
Returns
the text, contained in the input eld, when the user clicked OK
None, if the user pressed the Cancel button
Example:
name = input("Please enter your name to log in:")
name = input("Please enter your name to log in:", "anonymous") # a preset input text
A dialog box that looks like above will appear to allow the user to interactively enter some text. This text is then
assigned to the variable name, which can be used in other parts of the script, such as paste(name) to paste
the text to a login box.
When using the parameter default, the text input eld will be pre-populated with the given text, so the user
might just click OK/Cancel or edit the content of the input eld.
3.1. Everything about Sikuli Script 39
SikuliX Documentation, Release 1.1.0-Beta1
Listening to Global Hotkeys
Sikuli can listen to global hotkeys that you register with Env.addHotkey and call the corresponding handler (sikuli
functions) when the user presses the hotkeys.
New in version X1.0-rc3.
Env.addHotkey(key, modiers, handler)
Register the specied key + modiers as a global hotkey. When the hotkey is pressed, the specied function
handler will be called.
Parameters
key a character or a constant value dened in Key.
modiers Key modiers, which can be one or multiple constants dened in
KeyModifier.
Returns True if success.
New in version X1.0-rc3.
Env.removeHotkey(key, modiers)
Unregister the registered global hotkey key + modiers.
Parameters
key a character or a constant value dened in Key.
modiers Key modiers, which can be one or multiple constants dened in
KeyModifier.
Returns True if success.
General Settings and Access to Environment Information
Sikuli Level
Sikuli internally uses the class Settings to store globally used settings. Publicly available attributes may be accessed
by using Settings.[name-of-an-attribute] to get its value and Settings.attribute = value to
set it. It is highly recommended to only modify attributes, that are described in this document or when you really
know, what you are doing.
Actually all attributes of some value for scripting are described in the topic Controlling Sikuli Scripts and their Behav-
ior.
Jython/Python Level
You may use all settings, that are dened in standard Python/Jython and that are available in your system environment.
The modules sys and time are already imported, so you can use their methods without the need for an import statement.
sys.path may be one of the most valuable settings, since it is used by Python/Jython to locate modules, that are
referenced using import module. It is a list of paths, that is e.g. maintained by Sikuli to implement Importing
other Sikuli Scripts as a standard compliant feature.
If you want to use sys.path, it is recommended to do it as shown in the following example, to avoid appending the
same entry again:
myPath = "some-absolute-path"
if not myPath in sys.path:
sys.path.append(myPath)
40 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
Java Level
Java maintains a global storage for settings (key/value pairs), that can be accessed by the program/script. Sikuli uses it
too for some of its settings. Normally it is not necessary to access these settings at the Java level from a Sikuli script,
since Sikuli provides getter and setter methods for accessing values, that make sense for scripting. One example is the
list of paths, that Sikuli maintains to specify additional places to search for images (please refer to Importing other
Sikuli Scripts for more information).
If needed, you may access the java settings storage as shown in the following example:
import java
# get a value
val = java.lang.System.getProperty("key-of-property")
# set a propertys value
java.lang.System.getProperty("key-of-property", value)
Image Search Path
Sikuli maintains a list of locations to search for images when they are not found in the current .sikuli folder (a.k.a.
bundle path). This list is maintained internally but can be inspected and/or modied using the following functions:
getImagePath()
Get a list of paths where Sikuli will search for images.
# getImagePath() returns a Java array of unicode strings
imgPath = list(getImagePath()) # makes it a Python list
# to loop through
for p in imgPath:
print p
addImagePath(a-new-path)
Add a new path to the list of image search paths
removeImagePath(a-path-already-in-the-list)
Remove a path from the list of image search paths
Note: paths must be specied using the correct path separators (slash on Mac and Unix and double blackslashes on
Windows).
This list is automatically extended by Sikuli with script folders, that are imported (see: Importing other Sikuli Scripts),
so their contained images can be accessed. If you want to be sure of the results of your manipulations, you can use
getImagePath and check the content of the returned list. When searching images, the paths are scanned in the
order of the list. The rst image le with a matching image name is used.
Note: Behind the scenes this list is maintained in the java property store with the key SIKULI_IMAGE_PATH. This
can be preset when starting the JVM using the environment variable SIKULI_IMAGE_PATH and can be accessed at
runtime using the approach as mentioned under Accessing Settings - Java level. Be aware, that this is one string, where
the different entries are separated with a colon ( : ).
The default bundle path can also be accessed and modied by the two functions below:
setBundlePath(path-to-a-folder)
Set the path for searching images in all Sikuli Script methods. Sikuli IDE sets this automatically to the path of
the folder where it saves the script (.sikuli). Therefore, you should use this function only if you really know what
you are doing. Using it generally means that you would like to take care of your captured images by yourself.
Additionally images are searched for in the SIKULI_IMAGE_PATH, that is a global list of other places to look
for images. It is implicitly extended by script folders, that are imported (see: Reuse of Code and Images).
3.1. Everything about Sikuli Script 41
SikuliX Documentation, Release 1.1.0-Beta1
getBundlePath()
Get a string containing a fully qualied path to a folder containing your images used for nding patterns. Note:
Sikuli IDE sets this automatically to the path of the folder where it saves the script (.sikuli). You may use this
function if, for example, to package your private les together with the script or to access the picture les in
the .sikuli bundles for other purposes. Sikuli only gives you to access to the path name, so you may need other
python modules for I/O or other purposes.
Other places, where Sikuli looks for images, might be in the SIKULI_IMAGE_PATH.
Other Environment Information
Env.getOS()
Env.getOSVersion()
Get the type ( getOS() ) and version ( getOSVersion() ) of the operating system your script is running
on.
An example using these methods on a Mac is shown below:
# on a Mac
myOS = Env.getOS()
myVer = Env.getOSVersion()
if myOS == OS.MAC:
print "Mac " + myVer # e.g., Mac 10.6.3
else:
print "Sorry, not a Mac"
myOS = str(Env.getOS())
if myOS == "MAC" or myOS.startswith("M"):
print "Mac " + myVer # e.g., Mac 10.6.3
else:
print "Sorry, not a Mac"
New in version X1.0-rc2.
Env.getSikuliVersion()
Get the version of Sikuli.
Returns a string containing the version text of the IDE window title without Sikuli
An example for Sikuli X-1.0rc2:
if not Env.getSikuliVersion() == "X-1.0rc2":
print "This script needs Sikuli X-1.0rc2"
exit(1)
Env.getClipboard()
Get the content of the clipboard if it is text, otherwise an empty string.
Note: Be careful, when using Env.getClipboard() together with paste(), since paste internally uses
the clipboard to transfer text to other applications, the clipboard will contain what you just pasted. Therefore, if
you need the content of the clipboard, you should call Env.getClipboard() before using paste().
Tip: When the clipboard content was copied from a web page that mixes images and text, you should be aware,
that there may be whitespace characters around and inside your text, that you did not expect. In this case, you
can use Env.getClipboard().strip() to get rid of surrounding white spaces.
New in version X1.0-rc2.
Env.isLockOn(key-constant)
Get the current status ( on / off ) off the respective key. Only one key can be specied.
42 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
Parameters key-constant one of the key constants Key.CAPS_LOCK, Key.NUM_LOCK,
Key.SCROLL_LOCK
Returns True if the specied key is on, False otherwise
Further information about key constants can be found in Class Key.
New in version X1.0-rc2.
Env.getMouseLocation()
Get the current location of the mouse cursor.
Returns a Location object of the position of the mouse cursor on the screen.
Advanced Settings for Speed and Robustness
New in version X1.0-rc3.
Vision.setParameter(param, value)
Set the parameter param of the vision algorithm to value.
Parameters
param a string that indicates the parameter to set.
value a oat value.
Vision.getParameter(param)
Get the parameter param of the vision algorithm.
Parameters param a string that indicates the parameter to get.
Returns the oat value of the specied parameter.
The available parameters for tuning the vision algorithm of Sikuli is listed as follows. New in version X1.0-rc3.
MinTargetSize
MinTargetSize is the minimum image size to which Sikuli can resize.
Sikuli resizes the screen images to a smaller scale for faster matching. This scaling process speeds up the matching
process, but also increases the possibility of false matching. The default value of MinTargetSize in X-1.0rc3 is
12, which makes the matching algorithm be balanced between speed and robustness. If you feel that Sikuli is running
too slow, try a smaller value than 12. On the other hand, if you see Sikuli returns a match that is not what you expect,
i.e. a false match, try to increase MinTargetSize to make Sikuli be more robust to small details.
You can tune this parameter using the following Jython code.
3.1.2 Region
class Region
Region is a rectangular area on a screen, which is dened by
1. its upper left corner (x, y) as a distance relative to the upper left corner of the screen (0, 0) and
2. its dimension (w, h) as its width and height.
3.1. Everything about Sikuli Script 43
SikuliX Documentation, Release 1.1.0-Beta1
x, y, w, h are integer numbers counting a distance in pixels.
A region knows nothing about its visual content (windows, pictures, graphics, text, ...). It only knows the position on
the screen and its dimension.
New regions can be created, based on an existing region: you can extend a region in all directions or get the adjacent
rectangle up to the bounds of the screen horizontally or vertically.
The visual content of a region is evaluated by using methods like Region.find(), which looks for a given rect-
angular pixel pattern (dened by a Pattern object) within the region. The matching content in the region has a
similarity between 0 (not found) and 1 (found and it is per pixel exactly matches to the pattern). The nd can be
advised, to search with a minimum similarity, so that some minor variations in shape and color can be ignored. If
nothing else is specied, Sikuli searches with a minimum similarity of 0.7, which does what is expected in general
cases. (This default value can be changed in Settings.MinSimilarity.)
Find operations return a Match object, which has all the attributes and methods of a region and can be used in exactly
the same way (e.g. to nd or click another target within it). A Match has the dimension of the pattern used for
searching and also knows the position where it was found and its similarity score. A region preserves the best match
of the last successful nd operation and all matches of the last successful Region.findAll() (accessible with
Region.getLastMatches(). You can wait for patterns to show up using Region.wait(), to vanish using
Region.waitVanish() or just check whether a pattern exists without the need to handle exceptions.
Sikuli supports visual event driven programming. You can tell a region to observe that something appears, vanishes or
changes. It is possible to wait for the completion of an observation or let it run in the background while your following
script continues running. When one of the visual events happens, a handler in your script is called. Each region has
one observer and each observer can handle multiple visual events. Its your responsibility to stop an observation.
Creating a Region, Setting and Getting Attributes
In this chapter, you can nd information on how to create a new region object. The attributes of a region object can be
accessed directly or via method calls, e.g. to change their position and/or dimension. Here you will nd the HowTos.
class Region
Region(x, y, w, h)
Region(region)
Region(Rectangle)
Create a region object
Parameters
x x position of a rectangle.
y y position of a rectangle.
w height of a rectangle.
h width of a rectangle.
region an existing region object.
rectangle an existing object of Java class Rectangle
Returns a new region object.
In addition to creating a region by using the tool provided by the IDE, a region can be created by specifying
a rectangle. This is how the visual representation in the IDE of such a region is internally set up in the
script. A region can also be created by users in run-time using Region.selectRegion().
44 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
You can create a region based on another region. This just duplicates the region (new object). This
can be useful, if you need the same region with different attributes, such as another observation loop or
use Region.setThrowException() to control whether throwing an exception when nding fails.
Another way to create a region is to specify a rectangle object or to extend an existing region.
Note: The position and dimension attributes are named x, y (values representing the top left corner) and
w, h (width and height). You might use these or the get/set methods.
topLeft = Location(reg.x, reg.y) # equivalent to
topLeft = reg.getTopLeft()
theWidth = reg.w # equivalent to
theWidth = reg.getW()
reg.h = theWidth # equivalent to
reg.setH(theWidth)
Note: Additionally you might use selectRegion() to interactively create a new region.
setX(number)
setY(number)
setW(number)
setH(number)
Set the respective attribute of the region to the new value. This effectively moves the region around and/or
changes its dimension.
Parameters number the new value
New in version X1.0-rc2.
moveTo(location)
Set the position of this region regarding its top left corner to the given location (the x and y values are
modied).
Parameters location location object becomes the new top left corner
Returns the modied region object
reg.moveTo(anotherLocation) # equivalent to
reg.setX(anotherLocation.x); reg.setY(anotherLocation.y)
setROI(x, y, w, h)
setROI(rectangle)
setRect(x, y, w, h)
setRect(rectangle)
Both methods are doing exactly the same: setting position and dimension to new values. The motivation
for two names is to make scripts more readable: setROI() is intended to restrict the search to a smaller
area to speed up processing searches (region of interest), whereas setRect() should be used to redene
a region (which could be enlarging it).
Parameters
x, y, w, h (all) the attributes of a rectangle
rectangle a rectangle object
Returns None
New in version X1.0-rc2.
morphTo(region)
Set the position and dimension of this region to the corresponding values of the region given as parameter.
3.1. Everything about Sikuli Script 45
SikuliX Documentation, Release 1.1.0-Beta1
(see: setRect())
Parameters region a region object
Returns the modied region object
reg.morphTo(anotherRegion) # equivalent to
r = anotherRegion; reg.setX(r.x); reg.setY(r.y); reg.setW(r.w); reg.setH(r.h)
getX()
getY()
getW()
getH()
Get the respective attribute of the region.
Returns integer value
getCenter()
Get the center of the region.
Returns an object of Location
New in version X1.0-rc2.
getTopLeft()
getTopRight()
getBottomLeft()
getBottomRight()
Get the location of the regions respective corner
Returns Location object
getScreen()
Returns the screen object that contains this region.
Returns a new Screen object
This method only makes sense in Multi Monitor Environments, since it always returns the default screen
in a single monitor environment.
getLastMatch()
getLastMatches()
Returns the best match as a Match object or one or more match objects as an Iterator
object respectively
All successful nd operations (explicit like find() or implicit like click()), store the best match in the
lastMatch attribute of the region that was searched. findAll() stores all found matches into lastMatches
atribute of the region that was searched as an iterator.
To access these attributes use region.getLastMatch() or region.getLastMatches() re-
spectively.
How to use the iterator object returned by getLastMatches() is documented here.
setAutoWaitTimeout(seconds)
Set the maximum waiting time for all subsequent nd operations.
Parameters seconds a number, which can have a fraction. The internal granularity is millisec-
onds.
This method enables all nd operations to wait for the given pattern to appear until the specied amount
of time has elapsed. The default is 3.0 seconds. This method is intended for users to override this default
46 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
setting. As such it lets Region.find() work like Region.wait(), without being able to set an
individual timeout value for a specic nd operation.
getAutoWaitTimeout()
Get the current value of the maximum waiting time for nd operations.
Returns timeout in seconds
Extending a Region
These methods (exception inside()) return a new region object, that is based on the specied region (sometimes
called spatial operators). The range parameter, if given as positive integer number, restricts the dimension of the new
region (width and/or height respectively) to that value. If range is not specied, the newregion extends to the respective
boundary of the screen the given region belongs to. An exception is nearby(), which uses 50 as its default range.
Note: In all cases the current region remains unchanged.
Note: In all cases the new region does not extend beyond any boundary of the screen that contains the given region.
class Region
offset(location)
Returns a new Region object, whose upper left corner is relocated adding the locations x and y value to the
respective values of the given region. Width and height are the same. So this clones a region at a different
place.
Parameters location a Location object
Returns a new Region object
new_reg = reg.offset(Location(xoff, yoff)) # same as
new_reg = Region(reg.x + xoff, reg.y + yoff, reg.w, reg.h)
inside()
Returns the same object. Retained for upward compatibility.
Returns the Region object
This method can be used to make scripts more readable. region.inside().find() is totally equiv-
alent to region.find().
3.1. Everything about Sikuli Script 47
SikuliX Documentation, Release 1.1.0-Beta1
nearby([range ])
Returns a new Region that includes the nearby neighbourhood of the the current region. The new region
is dened by extending the current regions dimensions in all directions by range number of pixels. The
center of the new region remains the same.
Parameters range a positive integer indicating the number of pixels (default = 50).
Returns a Region object
above([range ])
Returns a new Region that is dened above the current regions top border with a height of range number
of pixels. So it does not include the current region. If range is omitted, it reaches to the top of the screen.
The new region has the same width and x-position as the current region.
Parameters range a positive integer dening the new height
Returns a Region object
below([range ])
Returns a new Region that is dened below the current regions bottom border with a height of range
number of pixels. So it does not include the current region. If range is omitted, it reaches to the bottom of
the screen. The new region has the same width and x-position as the current region.
Parameters range a positive integer dening the new height
Returns a Region object
left([range ])
Returns a new Region that is dened left of the current regions left border with a width of range number
of pixels. So it does not include the current region. If range is omitted, it reaches to the left border of the
screen. The new region has the same height and y-position as the current region.
Parameters range a positive integer dening the new width
Returns a Region object
48 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
right([range ])
Returns a new Region that is dened right of the current regions right border with a width of range
number of pixels. So it does not include the current region. If range is omitted, it reaches to the right
border of the screen. The new region has the same height and y-position as the current region.
Parameters range a positive integer dening the new width
Returns a Region object
Finding inside a Region and Waiting for a Visual Event
Besides acting on visual objects, nding them is one of the core functions of Sikuli.
PS: means, that either a Pattern or a string (path to an image le or just plain text) can be used as parameter. A
nd operation is successful, if the given image is found with the given minimum similarity or the given text is found
exactly. Similarity is a value between 0 and 1 to specify how likely the given image looks like the target. By default,
the similarity is 0.7 if an image rather than a pattern object with a specic similarity is given to Region.find().
Normally all these region methods are used as reg.find(PS), where reg is a region object. If written as find(PS)
it acts on the default screen, which is an implicit region in this case (see: SCREEN as Default Region). But in
most cases it is a good idea to use region.find() to restrict the search to a smaller region in order to speed up
processing.
If a nd operation is successful, the returned match is additionally stored internally with the region that was
used for the search. So instead of using a variable to store the match ( m = reg.find() ), you can use
reg.getLastMatch() to access it afterwards. Unsuccessful nd operations will leave these values unchanged.
By default, if the visual object (image or text) cannot be found, Sikuli will stop the script by raising an Exception
FindFailed. This follows the standards of the Python language, so that you could handle such exceptions using try:
... except: ....
If you are not used to programming using the Python language or because of other reasons, you might just want to
bypass the exception handling, which means just ignoring it (None is returned in that case). Or you might interactively
react on a FindFailed situation (e.g. optionally repeat the nd). Read more about concepts and options at: Exception
FindFailed.
If you have multiple monitors, please read Multi Monitor Environments.
Note on IDE: Capturing is a tool in the IDE, to quickly set up images to search for. These images are named
automatically by the IDE and stored together with the script, at the time it is saved (we call the location in the le
system bundle-path). Behind the curtain, the images itself are specied simply by using a string containing the le
name (path to an image le).
class Region
find(PS)
Parameters PS a Pattern object or a string (path to an image le or just plain text)
Returns a Match object that contains the best match or fails if not found
Find a particular GUI element, which is seen as the given image or just plain text. The given le name of an
image species the elements appearance. It searches within the region and returns the best match, which
shows a similarity greater than the minimum similarity given by the pattern. If no similarity was set for
the pattern by Pattern.similar() before, a default minimum similarity of 0.7 is set automatically.
If autoWaitTimeout is set to a non-zero value, nd() just acts as a wait().
Side Effect lastMatch: the best match can be accessed using Region.getLastMatch() afterwards.
findAll(PS)
3.1. Everything about Sikuli Script 49
SikuliX Documentation, Release 1.1.0-Beta1
Parameters PS a Pattern object or a string (path to an image le or just plain text)
Returns one or more Match objects as an iterator object or fails if not found
How to iterate through is documented here.
Repeatedly nd ALL instances of a pattern, until no match can be found anymore, that meets the require-
ments for a single Region.find() with the specied pattern.
By default, the returned matches are sorted by the similiarty. If you need them ordered by their positions,
say the Y coordinates, you have to use Pythons sorted function. Here is a example of sorting the matches
from top to bottom.
Side Effect lastMatches: a reference to the returned iterator object containing the found matches is stored
with the region that was searched. It can be accessed using getLastMatches() afterwards. How to iterate
through an iterator of matches is documented here.
wait([PS ][, seconds ])
Parameters
PS a Pattern object or a string (path to an image le or just plain text)
seconds a number, which can have a fraction, as maximum waiting time in seconds. The
internal granularity is milliseconds. If not specied, the auto wait timeout value set by
Region.setAutoWaitTimeout() is used. Use the constant FOREVER to wait for
an innite time.
Returns a Match object that contains the best match or fails if not found
If PS is not specied, the script just pauses for the specied amount of time. It is still possible to use
sleep(seconds) instead, but this is deprecated.
If PS is specied, it keeps searching the given pattern in the region until the image appears ( would have
been found with Region.find()) or the specied amount of time has elapsed. At least one nd opera-
tion is performed, even if 0 seconds is specied.)
Side Effect lastMatch: the best match can be accessed using Region.getLastMatch() afterwards.
Note: You may adjust the scan rate (how often a search during the wait takes place) by setting
Settings.WaitScanRate appropriately.
waitVanish(PS[, seconds ])
Wait until the give pattern PS in the region vanishes.
Parameters
PS a Pattern object or a string (path to an image le or just plain text)
seconds a number, which can have a fraction, as maximum waiting time in seconds. The
internal granularity is milliseconds. If not specied, the auto wait timeout value set by
Region.setAutoWaitTimeout() is used. Use the constant FOREVER to wait for
an innite time.
Returns True if the pattern vanishes within the specied waiting time, or False if the pattern
stays visible after the waiting time has elapsed.
This method keeps searching the given pattern in the region until the image vanishes (can not be found with
Region.find() any longer) or the specied amount of time has elapsed. At least one nd operation is
performed, even if 0 seconds is specied.
Note: You may adjust the scan rate (how often a search during the wait takes place) by setting
Settings.WaitScanRate appropriately.
50 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
exists(PS[, seconds ])
Check whether the give pattern is visible on the screen.
Parameters
PS a Pattern object or a string (path to an image le or just plain text)
seconds a number, which can have a fraction, as maximum waiting time in seconds. The
internal granularity is milliseconds. If not specied, the auto wait timeout value set by
Region.setAutoWaitTimeout() is used. Use the constant FOREVER to wait for
an innite time.
Returns a Match object that contains the best match. None is returned, if nothing is found
within the specied waiting time
Does exactly the same as Region.wait(), but no exception is raised in case of FindFailed. So it can be
used to symplify scripting in case that you only want to know wether something is there or not to decide
how to proceed in your workow. So it is typically used with an if statement. At least one nd operation
is performed, even if 0 seconds is specied. So specifying 0 seconds saves some time, in case there is no
need to wait, since its your intention to get the information not found directly.
Side Effect lastMatch: the best match can be accessed using Region.getLastMatch() afterwards.
Note: You may adjust the scan rate (how often a search during the wait takes place) by setting
Settings.WaitScanRate appropriately.
Observing Visual Events in a Region
You can tell a region to observe that something appears or vanishes, or something changes in that region. Using
the methods Region.onAppear(), Region.onVanish() and Region.onChange(), you register an event
observer that starts its observation when you call Region.observe(). Each region object can have exactly one
observation active and running. For each observation, you can register as many event observers as needed. So you can
think of it as grouping some wait() and waitVanish() together and have them processed simultanouesly, while
you are waiting for one of these events to happen.
It is possible to let the script wait for the completion of an observation or let it run in the background, while your script
is continuing. With a timing parameter you can tell Region.observe() to stop observation anyway after the given
time.
When one of the visual events happens, an event handler written by you is called. An event handler is a function
contained in your script and expects a SikuliEvent object as a parameter. During the processing in your handler,
the observation is paused until your handler has ended. Information between the main script and your handlers can be
given forward and backward using global variables.
Its your responsibility to stop an observation. This can either be done by calling Region.stopObserver() or
by starting the observation with a timing parameter.
Since you can have as many region objects as needed and each region can have one observation active and running.
So theoretically its possible to have as many visual events being observered at the same time as needed. But in reality,
the number of observation is still limited by the system resources available to Sikuli at that time.
Be aware, that every observation is a number of different nd operations that are processed repeatedly. So
to speed up processing and keep your script acting, you may want to dene a region for observation as small
as possible. You may adjust the scan rate (how often a search during the observation takes place) by setting
Settings.ObserveScanRate appropriately.
PS: means, that either a Pattern or a String (path to an image le or just plain text) can be used as parameter. handler:
as a parameter in the following methods, you have to specify the name of a function, which will be called by the
observer, in case the observed event happens. The function itself has to be dened in your script before using the
3.1. Everything about Sikuli Script 51
SikuliX Documentation, Release 1.1.0-Beta1
method that references the function. The existance of the function will be checked before starting the script. So to get
your script running, you have to have at least the following statements in your script:
def myHandler(event): # you can choose any valid function name
# event: can be any variable name, it references the SikuliEvent object
pass # add your statements here
onAppear("path-to-an-image-file", myHandler) # or any other onEvent()
observe(10) # observes for 10 seconds
Read SikuliEvent to know what is contained in a SikuliEvent object
Note on performance: Normally all the region methods are used as reg.onAppear(PS), where reg is a region
object. If written as onAppear(PS) it operates on the default screen being the implicit region in this case. Using
region.onEvent() instead will restrict the search to the regions rectangle and speed up processing, if region is
signicantly smaller than the whole screen.
Note: In case of having more than one Monitor active, read Multi Monitor Environments before.
Note on IDE: Capturing is a tool in the IDE, to quickly set up images to search for. These images are named
automatically by the IDE and stored together with the script, at the time it is saved (we call the location in the le
system bundle-path). Behind the curtain the images itself are specied by using a string containing the le name (path
to an image le).
class Region
onAppear(PS, handler)
Parameters
PS a Pattern object or a string (path to an image le or just plain text.
handler the name of a handler function in the script
With the given region you register an observer, that should wait for the pattern to be there or to appaear
and is activated with the next call of observe(). In the moment the internal nd operation on the given
pattern is successful during observation, your handler is called and the observation is paused until you
return from your handler.
onVanish(PS, handler)
Parameters
PS a Pattern object or a string (path to an image le or just plain text.
handler the name of a handler function in the script
With the given region you register an observer, that should wait for the pattern to be not there or to vanish
and is activated with the next call of observe(). In the moment the internal nd operation on the given
pattern fails during observation, your handler is called and the observation is paused until you return from
your handler.
onChange([minChangedSize ], handler)
Parameters
minChangedSize the minimum size in pixels of a change to trigger a change event. If
omitted: 50 is used (see Settings.ObserveMinChangedPixels).
handler the name of a handler function in the script
New in version X1.0-rc2: minChangedSize
52 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
With the given region you register an observer, that should wait for the visual content of the given region
to change and is activated with the next call of observe(). In the moment the visual content changes
during observation, your handler is called and the observation is paused until you return from your handler.
Here is a example that highlights all changes in an observed region.
def changed(event):
print "something changed in ", event.region
for ch in event.changes:
ch.highlight() # highlight all changes
sleep(1)
for ch in event.changes:
ch.highlight() # turn off the highlights
with selectRegion("select a region to observe") as r:
# any change in r larger than 50 pixels would trigger the changed function
onChange(50, changed)
observe(background=True)
wait(30) # another way to observe for 30 seconds
r.stopObserver()
observe([seconds ][, background = False | True ])
Begin observation within the region.
Parameters
seconds a number, which can have a fraction, as maximum observation time in seconds.
Omit it or use the constant FOREVER to tell the observation to run for an innite time (or
until stopped by a call of stopObserve()).
background a ag indicating whether observation is run in the background. when set to
True, the observation will be run in the background and processing of your script is contin-
ued immediately. Otherwise the script is paused until the completion of the observation.
For each region object, only one observation can be running at a given time.
Note: You may adjust the scan rate (how often a search during the observation takes place) by setting
Settings.ObserveScanRate appropriately.
stopObserver()
Stop observation within the region.
This must be called on a valid region object. The source region of an observed visual event is available
as one of the attributes of the event parameter that is passed to the handler function when the function is
invoked.
For example, to stop observation within a handler function, simply call
event.region.stopObserver() inside the handler function.:
def myHandler(event):
event.region.stopObserver() # stops the observation
onAppear("path-to-an-image-file", myHandler)
observe(FOREVER) # observes until stopped in handler
New in version X1.0-rc2.
class SikuliEvent
When processing an observation in a region, a handler function is called, when one of your registered events
Region.onAppear(), Region.onVanish() or Region.onChange() happen.
3.1. Everything about Sikuli Script 53
SikuliX Documentation, Release 1.1.0-Beta1
The one parameter, you have access to in your handler function is an instance of Class SikuliEvent. You have
access to the following attributes of the event, that might help to identify the cause of the event and act on the
resulting matches.
Usage: event.attribute where event is the parameter from the denition of your handler function:
def myHandler(event):
type
The type of this event can be SikuliEvent.Type.APPEAR, SikuliEvent.Type.VANISH, or
SikuliEvent.Type.CHANGE.
pattern
The source pattern that triggered this event. This is only valid in APPEAR and VANISH events.
region
The source region of this event.
match
In an APPEAR event, match saves the top Match object that appeared in the observed region.
In a VANISH event, match saves the last Match object that was in the observed region but vanished.
This attribute is not valid in a CHANGE event.
changes
This attribute is valid only in a CHANGE event.
This changes attribute is a list of Match objects that changed and their sizes are at least minChangedSize
pixels.
Acting on a Region
Besides nding visual objects on the screen, acting on these elements is one of the kernel operations of Sikuli. Mouse
actions can be simulated as well as pressing keys on a keyboard.
The place on the screen, that should be acted on (in the end just one specic pixel, the click point), can be given either
as a pattern like with the nd operations or by directly referencing a pixel location or as center of a region
object (match or screen also) or the target offset location connected with a match. Since all these choices can be
used with all action methods as needed, they are abbreviated and called like this:
PSMRL: which means, that either a Pattern object or a string (path to an image le or just plain text) or a Match
or a Region or a Location can be used as parameter, in detail:
P: pattern: a Pattern object. An implicit nd operation is processed rst. If successful, the center of the
resulting matches rectangle is the click point. If the pattern object has a target offset specied, this is used as
click point instead.
S: string: a path to an image le or just plain text. An implicit nd operation with the default minimum similarity
0.7 is processed rst. If successful, the center of the resulting match object is the click point.
M: match: a match object from a previous nd operation. If the match has a target specied it is used as the
click point, otherwise the center of the matches rectangle.
R: region: a region object whose center is used as click point.
L: location: a location object which by denition represents a point on the screen that is used as click point.
54 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
It is possible to simulate pressing the so called key modiers together with the mouse operation or when simulating
keyboard typing. The respective parameter is given by one or more predened constants. If more than one modier is
necessary, they are combined by using + or |.
Normally all these region methods are used as reg.click(PS), where reg is a region object. If written as
click(PS) the implicit nd is done on the default screen being the implicit region in this case (see: SCREEN
as Default Region). But using reg.click(PS) will restrict the search to the regions rectangle and speed up pro-
cessing, if region is signicantly smaller than the whole screen.
Generally all aspects of nd operations and of Region.find() apply.
If the nd operation was successful, the match that was acted on, can be recalled using Region.getLastMatch().
As a default, if the visual object (image or text) cannot be found, Sikuli will stop the script by raising an Exception
FindFailed (details see: not found).
Note on IDE: Capturing is a tool in the IDE, to quickly set up images to search for. These images are named
automatically by the IDE and stored together with the script, at the time it is saved (we call the location in the le
system bundle-path). Behind the curtain the images itself are specied by using a string containing the le name (path
to an image le).
Note: If you need to implement more sophisticated mouse and keyboard actions look at Low Level Mouse and Key-
board Actions.
Note: In case of having more than one Monitor active, refer to Multi Monitor Environments for more details.
Note on Mac: it might be necessary, to use switchApp() before, to prepare the application for accepting the action.
class Region
click(PSMRL[, modiers ])
Perform a mouse click on the click point using the left button.
Parameters
PSMRL a pattern, a string, a match, a region or a location that evaluates to a click point.
modiers one or more key modiers
Returns the number of performed clicks (actually 1). A 0 (integer null) means that because of
some reason, no click could be performed (in case of PS may be not Found).
Side Effect if PS was used, the match can be accessed using Region.getLastMatch() afterwards.
Example:
doubleClick(PSMRL[, modiers ])
Perform a mouse double-click on the click point using the left button.
Parameters
PSMRL a pattern, a string, a match, a region or a location that evaluates to a click point.
modiers one or more key modiers
Returns the number of performed double-clicks (actually 1). A 0 (integer null) means that
because of some reason, no click could be performed (in case of PS may be not Found).
Side Effect if PS was used, the match can be accessed using Region.getLastMatch() afterwards.
rightClick(PSMRL[, modiers ])
Perform a mouse click on the click point using the right button.
Parameters
3.1. Everything about Sikuli Script 55
SikuliX Documentation, Release 1.1.0-Beta1
PSMRL a pattern, a string, a match, a region or a location that evaluates to a click point.
modiers one or more key modiers
Returns the number of performed right cilicks (actually 1). A0 (integer null) means that because
of some reason, no click could be performed (in case of PS may be not Found).
Side Effect if PS was used, the match can be accessed using Region.getLastMatch() afterwards.
highlight([seconds ])
Highlight the region for some period of time.
Parameters seconds a decimal number taken as duration in seconds
The region is highlighted showing a red colored frame around it. If the parameter seconds is given, the
script is suspended for the specied time. If no time is given, the highlighting is started and the script
continues. When later on the same highlight call without a parameter is made, the highlighting is stopped
(behaves like a toggling switch).
Example:
m = find(some_image)
# the red frame will blink for about 7 - 8 seconds
for i in range(5):
m.highlight(1)
wait(0.5)
# a second red frame will blink as an overlay to the first one
m.highlight()
for i in range(5):
m.highlight(1)
wait(0.5)
m.highlight()
# the red frame will grow 5 times
for i in range(5):
m.highlight(1)
m = m.nearby(20)
Note: The red frame is just an overlay in front of all other screen content and stays in its place, inde-
pendently from the behavior of this other content, which means it is not connected to the content of
the dening region. But it will be adjusted automatically, if you change position and/or dimension of this
region in your script, while it is highlighted.
hover(PSMRL)
Move the mouse cursor to hover above a click point.
Parameters
PSMRL a pattern, a string, a match, a region or a location that evaluates to a click point.
modiers one or more key modiers
Returns the number 1 if the mousepointer could be moved to the click point. A 0 (integer null)
returned means that because of some reason, no move could be performed (in case of PS may
be not Found).
Side Effect if PS was used, the match can be accessed using Region.getLastMatch() afterwards.
dragDrop(PSMRL, PSMRL[, modiers ])
Perform a drag-and-drop operation from a starting click point to the target click point indicated by the two
PSMRLs respectively.
56 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
Parameters
PSMRL a pattern, a string, a match, a region or a location that evaluates to a click point.
modiers one or more key modiers
If one of the parameters is PS, the operation might fail due to not Found.
Sideeffect: when using PS, the match of the target can be accessed using Region.getLastMatch()
afterwards. If only the rst parameter is given as PS, this match is returned by
Region.getLastMatch().
When the operation does not performas expected (usually caused by timing problems due to delayed re-
actions of applications), you may adjust the internal timing parameters Settings.DelayAfterDrag
and Settings.DelayBeforeDrop eventually combined with the internal timing parameter
Settings.MoveMouseDelay.
Another solution might be, to use a combination of Region.drag() and Region.dropAt() com-
bined with your own wait() usages. If the mouse movement from source to target is the problem, you
might break up the move path into short steps using Region.mouseMove().
Note: If you need to implement more sophisticated mouse and keyboard actions look at Low Level Mouse
and Keyboard Actions.
drag(PSMRL)
Start a drag-and-drop operation by dragging at the given click point.
Parameters PSMRL a pattern, a string, a match, a region or a location that evaluates to a click
point.
Returns the number 1 if the operation could be performed. A 0 (integer null) returned means
that because of some reason, no move could be performed (in case of PS may be not Found).
The mousepointer is moved to the click point and the left mouse button is pressed and held, until another
mouse action is performed (e.g. a Region.dropAt() afterwards). This is nomally used to start a
drag-and-drop operation.
Side Effect if PS was used, the match can be accessed using Region.getLastMatch() afterwards.
dropAt(PSMRL[, delay ])
Complete a drag-and-drop operation by dropping a previously dragged item at the given target click point.
Parameters
PSMRL a pattern, a string, a match, a region or a location that evaluates to a click point.
delay time to wait after in seconds as decimal value
Returns the number 1 if the operation could be performed. A 0 (integer null) returned means
that because of some reason, no move could be performed (in case of PS may be not Found).
The mousepointer is moved to the click point. After waiting for delay seconds the left mouse button is
released. This is normally used to nalize a drag-and-drop operation. If it is necessary to visit one ore
more click points after dragging and before dropping, you can use Region.mouseMove() inbetween.
Side Effect if PS was used, the match can be accessed using Region.getLastMatch() afterwards.
type([PSMRL], text[, modiers ])
Type the text at the current focused input eld or at a click point specied by PSMRL.
Parameters
PSMRL a pattern, a string, a match, a region or a location that evaluates to a click point.
modiers one or more key modiers
3.1. Everything about Sikuli Script 57
SikuliX Documentation, Release 1.1.0-Beta1
Returns the number 1 if the operation could be performed, otherwise 0 (integer null), which
means, that because of some reason, it was not possible or the click could be performed (in
case of PS may be not Found).
This method simulates keyboard typing interpreting the characters of text based on the layout/keymap
of the standard US keyboard (QWERTY). Special keys (ENTER, TAB, BACKSPACE, ...) can be in-
corporated into text by using the constants dened in Class Key using the standard string concatenation
(+).
If PSMRL is given, a click on the clickpoint is performed before typing, to gain the focus. (Mac: it might
be necessary, to use switchApp() to give focus to a target application before, to accept typed/pasted
characters.)
If PSMRL is omitted, it performs the typing on the current focused visual component (normally an input
eld or an menu entry that can be selected by typing something).
Side Effect if PS was used, the match can be accessed using Region.getLastMatch() afterwards.
Note: If you need to type international characters or you are using layouts/keymaps other than US-
QWERTY, you should use Region.paste() instead. Since type() is rather slow because it simulates
each key press, for longer text it is preferrable to use Region.paste().
Best Practice: As a general guideline, the best choice is to use paste() for readable text and type()
for action keys like TAB, ENTER, ESC, .... Use one type() for each key or key combination and be
aware, that in some cases a short wait() after a type() might be necessary to give the target application
some time to react and be prepared for the next Sikuli action.
paste([PSMRL], text)
Paste the text at a click point.
Parameters
PSMRL a pattern, a string, a match, a region or a location that evaluates to a click point.
modiers one or more key modiers
Returns the number 1 if the operation could be performed, otherwise 0 (integer null), which
means, that because of some reason, it was not possible or the click could be performed (in
case of PS may be not Found).
Pastes text using the clipboard (OS-level shortcut (Ctrl-V or Cmd-V)). So afterwards your clipboard con-
tains text. paste() is a temporary solution for typing international characters or typing on keyboard
layouts other than US-QWERTY.
If PSMRL is given, a click on the clickpoint is performed before typing, to gain the focus. (Mac: it might
be necessary, to use switchApp() to give focus to a target application before, to accept typed/pasted
characters.)
If PSMRL is omitted, it performs the paste on the current focused component (normally an input eld).
Side Effect if PS was used, the match can be accessed using Region.getLastMatch() afterwards.
Note: Special keys (ENTER, TAB, BACKSPACE, ...) cannot be used with paste(). If needed, you
have to split your complete text into two or more paste() and use type() for typing the special keys
inbetween. Characters like \n (enter/new line) and \t (tab) should work as expected with paste(), but be
aware of timing problems, when using e.g. intervening \t to jump to the next input eld of a form.
Extracting Text from a Region
class Region
58 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
text()
Extract the text contained in the region using OCR.
Returns the text as a string. Multiple lines of text are separated by intervening n.
Note: Since this feature is still in an experimental state, be aware, that in some cases it might not work
as expected. If you face any problems look at the Questions & Answers / FAQs and the Bugs.
Low-level Mouse and Keyboard Actions
class Region
mouseDown(button)
Press the mouse button down.
Parameters button one or a combination of the button constants Button.LEFT,
Button.MIDDLE, Button.RIGHT.
Returns the number 1 if the operation is performed successfully, and zero if otherwise.
The mouse button or buttons specied by button are pressed until another mouse action is performed.
mouseUp([button])
Release the mouse button previously pressed.
Parameters button one or a combination of the button constants Button.LEFT,
Button.MIDDLE, Button.RIGHT.
Returns the number 1 if the operation is performed successfully, and zero if otherwise.
The button specied by button is released. If button is omitted, all currently pressed buttons are released.
mouseMove(PSRML)
Move the mouse pointer to a location indicated by PSRML.
Parameters PSMRL a pattern, a string, a match, a region or a location that evaluates to a click
point.
Returns the number 1 if the operation could be performed. If using PS (which invokes an
implicity nd operation), nd fails and you have switched off FindFailed exception, a 0
(integer null) is returned. Otherwise, the script is stopped with a FindFailed exception.
Sideeffects: when using PS, the match can be accessed using Region.getLastMatch() afterwards
wheel(PSRML, WHEEL_DOWN | WHEEL_UP, steps)
Move the mouse pointer to a location indicated by PSRML and turn the mouse wheel in the specied
direction by the specied number of steps.
Parameters
PSMRL a pattern, a string, a match, a region or a location that evaluates to a click point.
WHEEL_DOWN|WHEEL_UP one of the two constants denoting the wheeling direc-
tion.
steps an integer indicating the amoung of wheeling.
Sideeffects: when using PS, the match can be accessed using Region.getLastMatch() afterwards
keyDown(key | list-of-keys)
Press and hold the specied key(s) until released by a later call to Region.keyUp().
3.1. Everything about Sikuli Script 59
SikuliX Documentation, Release 1.1.0-Beta1
Parameters key|list-of-keys one or more keys (use the consts of class Key). A list of keys is
a concatenation of several key constants using +.
Returns the number 1 if the operation could be performed and 0 if otherwise.
keyUp([key | list-of-keys ])
Release given keys. If no key is given, all currently pressed keys are released.
Parameters key|list-of-keys one or more keys (use the consts of class Key). A list of keys is
a concatenation of several key constants using +.
Returns the number 1 if the operation could be performed and 0 if otherwise.
Exception FindFailed
As a default, nd operations (explicit and implicit) when not successful raise an Exception FindFailed, that
will stop the script immediately. This is of great help, when developing a script, to step by step adjust timing and
similarity. When the script runs perfectly, then an exception FindFailed signals, that something is not as it should be.
To implement some checkpoints, where you want to asure your workow, use Region.exists(), that reports a
not found situation without raising FindFailed (returns False instead).
To run all or only parts of your script without FindFailed exceptions to be raised, use
Region.setThrowException() or Region.setFindFailedResponse() to switch it on and off
as needed.
For more sophisticated concepts, you can implement your own exception handling using the standard Python construct
try: ... except: ... .
New in version X1.0-rc2.
These are the possibilities to handle not found situations:
generally abort a script, if not handled with try: ... except: ... (the default setting or
using setThrowException(True) or setFindFailedResponse(ABORT))
generally ignore all not found situations (using setThrowException(False) or
setFindFailedResponse(SKIP)),
want to be prompted in such a case (using setFindFailedResponse(PROMPT))
advise Sikuli to wait forever (be careful with that!) (using setFindFailedResponse(RETRY))
Comment on using PROMPT:
This feature is helpful in following situations:
you are developing something, that needs an application with its windows to be in place, but this workow you
want to script later. If it comes to that point, you get the prompt, arrange the app and click Retry. Your workow
should continue.
you have a workow, where the user might do some corrective actions, if you get a FindFailed
guess you nd more ;-)
In case of a FindFailed, you get the following prompt:
60 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
Clicking Retry would again try to nd the image. Skip would continue the script and Abort would end the script.
Examples: 4 solutions for a case, where you want to decide how to proceed in a workow based on the fact that a
specic image can be found. (pass is the python statement, that does nothing, but maintains indentation to form the
blocks):
# --- nice and easy
if exists("path-to-image"): # no exception, returns None when not found
pass # it is there
else:
pass # we miss it
# --- using exception handling
# every not found in the try block will switch to the except block
try:
find("path-to-image")
pass # it is there
except FindFailed:
pass # we miss it
# --- using setFindFailedResponse
setFindFailedResponse(SKIP) # no exception raised, not found returns None
if find("path-to-image"):
setFindFailedResponse(ABORT) # reset to default
pass # it is there
else:
setFindFailedResponse(ABORT) # reset to default
pass # we miss it
# --- using setThrowException
setThrowException(False) # no exception raised, not found returns None
if find("path-to-image"):
setThrowException(True) # reset to default
pass # it is there
else:
setThrowException(True) # reset to default
pass # we miss it
class Region
Reminder If used without specifying a region, the default/primary screen (default region SCREEN) is used.
3.1. Everything about Sikuli Script 61
SikuliX Documentation, Release 1.1.0-Beta1
New in version X1.0-rc2.
setFindFailedResponse(ABORT | SKIP | PROMPT | RETRY)
For the specied region set the option, how Sikuli should handle not found situations. The option stays
in effect until changed by another setFindFailedResponse().
Parameters
ABORT all subsequent nd failed operations (explicit or implicit) will raise exception
FindFailed (which is the default when a script is started).
SKIP all subsequent nd operations will not raise exception FindFailed. Instead, explicit
nd operations such as Region.find() will return None. Implicit nd operations (ac-
tion functions) such as Region.click() will do nothing and return 0.
PROMPT all subsequent nd operations will not raise exception FindFailed. Instead
you will be prompted for the way to handle the situation (see using PROMPT). Only the
current nd operation is affected by your response to the prompt.
RETRY all subsequent nd operations will not raise exception FindFailed. Instead,
Sikuli will try to nd the target until it gets visible. This is equivalent to using wait(
... , FOREVER) instead of find() or using setAutoWaitTimeout(FOREVER).
getFindFailedResponse()
Get the current setting in this region.
Returns ABORT or SKIP or PROMPT or RETRY
Usage:
val = getFindFailedResponse()
if val == ABORT:
print "not found will stop script with Exception FindFailed"
elif val == SKIP:
print "not found will be ignored"
elif val == PROMPT:
print "when not found you will be prompted"
elif val == RETRY:
print "we will always wait forever"
else:
print val, ": this is a bug :-("
Note: It is recommended to use set/getFindFailedResponse() instead of
set/getThrowException() since the latter ones might be deprecated in the future.
setThrowException(False | True)
By using this method you control, how Sikuli should handle not found situations in this region.
Parameters
True all subsequent nd operations (explicit or implicit) will raise exception FindFailed
(which is the default when a script is started) in case of not found.
False all subsequent nd operations will not raise exception FindFailed. Instead, explicit
nd operations such as Region.find() will return None. Implicit nd operations (ac-
tion functions) such as Region.click() will do nothing and return 0.
getThrowException()
Returns True or False
Get the current setting as True or False (after start of script, this is True by default) in this region.
62 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
Grouping Method Calls ( with Region: )
Deprecated for 1.1.0+ Instead of with use the new feature use(aRegionOrAScreen).
Instead of:
# reg is a region object
if not reg.exists(image1):
reg.click(image2)
reg.wait(image3, 10)
reg.doubleClick(image4)
you can group methods applied to the same region using Pythons with syntax:
# reg is a region object
with reg:
if not exists(image1):
click(image2)
wait(image3, 10)
doubleClick(image4)
All methods inside the with block (mind indentation) that have the region omitted are redirected to the region object
specied at the with statement.
IMPORTANT Usage Note
This only works without problems for region objects created on the script level using one of the constructors
Region().
New region objects created with Region methods, that return new region objects, can only be used with with, when
casting the new region to a scripting level region object like this:
# reg is a region object
regNew = reg.left() # returns a non-scripting-level region object
with Region(regNew):
if not exists(image1):
click(image2)
wait(image3, 10)
doubleClick(image4)
If you do not do that, you will get a syntax error saying, that __enter__ is not dened for this region. In all these
cases use the casting approach using castedRegion = Region(regionNotWorking) (might also be needed
with Match objects or window regions returned by the App class).
3.1.3 Screen
class Screen
Class Screen is there, to have a representation for a pysical monitor where the capturing process (grabbing a rect-
angle from a screenshot, to be used for further processing with nd operations is implemented. For Multi Monitor
Environments it contains features to map to the relevant monitor.
Since Screen extends class Region, all methods of class Region can be used with a screen object.
Of special interest might be the grouping of region method calls using with: in Multi Monitor Environments: use it
for other screens, than the default/primary screen, where you have this feature by default.
Be aware, that using the whole screen for nd operations may have an impact on performance. So if possible either
use setROI() or restrict a nd operation to a smaller region object (e.g. reg.find()) to speed up processing.
3.1. Everything about Sikuli Script 63
SikuliX Documentation, Release 1.1.0-Beta1
Screen: Setting, Getting Attributes and Information
class Screen
Screen([id ])
Create a new Screen object
Parameters id an integer number indicating which monitor in a multi-monitor environment.
Returns a new screen object.
It creates a new screen object, that represents the default/primary monitor (whose id is 0), if id is omitted.
Numbers 1 and higher represent additional monitors that are available at the time, the script is running
(read for details).
Using numbers, that do not represent an existing monitor, will stop the script with an error. So you may
either use getNumberScreens() or exception handling, to avoid this.
Note: If you want to access the default/primary monitor ( Screen(0) ) without creating a new screen object,
use the constant reference SCREEN, that is initiated when your script starts: SCREEN=Screen(0).
getNumberScreens()
Get the number of screens in a multi-monitor environment at the time the script is running
getBounds()
Get the dimensions of monitor represented by the screen object.
Returns a rectangle object
The width and height of the rectangle denote the dimensions of the monitor represented by the screen
object. These attributes are obtained from the operating system. They can not be modied using Sikuli
script.
Screen as (Default) Region
Normally all region methods are used as reg.nd(PS), where reg is a region object (or a screen or a match object). If
written as nd(PS) it acts on the default screen being the implicit region in this case (mapped to the constant reference
SCREEN). In Multi Monitor Environments this is the primary monitor (use the constant reference SCREEN, to access
it all the time), that normally is Screen(0), but might be another Screen() object depending on your platform.
So its a convenience feature, that can be seen as an implicit use of the python construct with object:.
On the other hand this may slow down processing speed, because of time consuming searches. So to speed up
processing, saying region.nd() will restrict the search to the specied rectangle. Another possibility is to say setROI()
to restrict the search for all following nd operations to a smaller region than the whole screen. This will speed up
processing, if the region is signicantly smaller than the whole screen.
Capturing
Capturing is the feature, that allows to grab a rectangle from a screenshot, to save it for later use. At each time, a
capturing is initiated, a new screenshot is taken.
There are two different versions: the rst one Screen.capture() saves the content of the selected rectangle in
a le and returns its le name, whereas the second one Screen.selectRegion() just returns the position and
dimension of the selected rectangle.
Both features are available in the IDE via the buttons in the toolbar.
64 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
class Screen
capture([region | rectangle | text ])
capture(x, y, w, h)
Parameters
region an existing region object.
rectangle an existing rectangle object (e.g., as a return value of another region method).
text text to display in the middle of the screen in the interactive mode.
x x position of the rectangle to capture
y y position of the rectangle to capture
w width of the rectangle to capture
h height of the rectangle to capture
Returns the path to the le, where the captured image was saved. In interactive mode, the user
may cancel the capturing, in which case None is returned.
Interactive Mode: The script enters the screen-capture mode like when clicking the button in the IDE,
enabling the user to capture a rectangle on the screen. If no text is given, the default Select a region on
the screen is displayed.
If any arguments other than text are specied, capture() automatically captures the given rectangle of the
screen. In any case, a new screenshot is taken, the content of the selected rectangle is saved in a temporary
le. The le name is returned and can be used later in the script as a reference to this image. It can be used
directly in cases, where a parameter PS is allowed (e.g. Region.find(), Region.click(), ...).
selectRegion([text ])
Select a region on the screen interactively
Parameters text Text to display in the middle of the screen.
Returns a new Region object or None, if the user cancels the capturing process.
text is displayed for about 2 seconds in the middle of the screen. If text is omitted, the default Select a
region on the screen is displayed.
The interactive capture mode is entered and allows the user to select a region the same way as using the
selection tool in the IDE.
Note: You should check the result, since the user may cancel the capturing.
Multi-Monitor Environments
If more than one monitor is available, Sikuli is able to manage regions and click points on these monitors.
3.1. Everything about Sikuli Script 65
SikuliX Documentation, Release 1.1.0-Beta1
The base is the coordinate system (picture above), that positions the primary monitor with its upper left corner at
(0,0) extending the x-direction to the right and the y-direction towards the lower boundary of the screen. The position
of additional monitors can be congured in the operating system to be on either side of the primary monitor, with
different positions and sizes. So monitors left of the primary will have pixels with negative x-values and monitors
above will have negative y-values (left and above both x and y are negative).
At script start, Sikuli gets the relevant information from the operating system and creates respective screen objects, that
have an ID (0 for the rst or primary monitor, 1 and higher for additional monitors with a maximum of one less than
the number of screens) and know the rectangle, they cover in the coordinate system. These informations are readonly
for a script.
These predened screen objects can be accessed with Screen(0), Screen(1), ... and are normally used to create your
own screen objects. The possibility to use the region methods on a default region mapped to the primary monitor is
implemented with the constant reference SCREEN. This concept is only available for the primary monitor.
How to get the relevant information:
getNumberScreens() returns the number of available screens.
getBounds() returns the rectangle covered by the default/primary monitor.
Screen.getBounds() returns the rectangle covered by a screen object created using Screen(id).
Be aware: Changes in your system settings are only recognized by the IDE, when it is started.
Windows: The monitor, that is the rst one based on hardware mapping (e.g. the laptop monitor), will always be
Screen(0). In the Windows settings it is possible to place the taskbar on one of the secondary monitors, which makes it
the primary monitor getting the base coordinates (0,0). The other available monitors are mapped around based on your
settings. But the Sikuli internal mapping is not changed, so the primary monitor might be any of your Screen() objects.
Sikuli takes care for that and maps SCREEN always to the primary monitor (the one with the (0,0) coordinates). So
for example you have a laptop with an external monitor, that shows the taskbar (is primary monitor):
SCREEN maps to Screen(1)
Screen(0) is your laptop monitor
66 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
Mac: The monitor, that has the System Menu Bar, is always Screen(0) and mapped to the default SCREEN.
Linux (Under construction)
With its rectangle, a screen object is always identical with the monitor it was created using Screen(id). Using
Region.setROI() to restrict the region of interest for nd operations has no effect on the base rectangle of the
screen object.
On the other hand region objects and location objects can be positioned anywhere in the coordinate system. Only when
a nd operation or a click action has to be performed, the objects rectangle or point has to be inside the rectangle of
an existing monitor (basically repersented by Screen(0), Screen(1), ...). When this condition is met, everything works
as expected and known from a single monitor system.
With nding and acting there are the following exceptions:
Point Outside: a click point is outside any monitor rectangle. The clickpoint will be mapped to the edges or
corners of the primary monitor according to the relative position:
to the edges if its x or y value is in the range of the respective edge (right, left, above, below)i
to the corners, if x and y are outside any range of any edge (left/above -> upper left corner, ...)
Region Outside: a region is completely outside any monitor
a click action is handled in thesame way as Point Outside
a nd operation will always fail
Region Partially Outside: a region is partially outside a monitor but not overlapping another monitor
a click action is handled in the same way as Point Outside
a nd operation will be carried out only in the part of region within the bounds of the monitor, excluding
the area outside the monitor.
Region Across Monitors: a region lies across multiple monitors:
a click action is handled in the same way as Point Outside
a nd operation will be restricted to the region within the bounds of the monitor that has a smaller id.
An interactive capture (the user is asked to select an image or a rectangle via Screen.capture() or
Screen.selectRegion()) will automatically be restricted to the monitor, where it was started.
A scripted capture using a rectangle or a region (i.e. Screen.capture( region | rectangle )), will be
handled accordingly:
Region Outside: no image is captured, None is returned
Region Partially Outside: the returned image will only cover the part inside the monitor
Region Across Monitors: the returned image will only cover the part inside the monitor with the smallest id.
Based on the knowledge of your monitor conguration, you can now start some further evaluations using e.g.
Region.hover() together with setShowActions(True) and highlighting using Region.highlight().
3.1.4 Location
This class is there as a convenience, to handle single points on the screen directly by its position (x, y). It is mainly
used in the actions on a region, to directly denote the click point. It contains methods, to move a point around on the
screen.
class Location
3.1. Everything about Sikuli Script 67
SikuliX Documentation, Release 1.1.0-Beta1
Location(x, y)
Parameters
x x position
y y position
Returns a new location object representing the position (x,y) on the screen
getX()
getY()
Get the x or y value of a location object
It is possible to get the values directly by location.x or location.y. It is also possible to set these
values directly by location.x = value or location.y = value.
Note: getX()and getY() currently (versions 0.10.2 and X 1.0rc2) return oat values (Java:
double), whereas location.x and location.y return integer values.
setLocation(x, y)
Set the location of this object to the specied coordinates.
offset(dx, dy)
Get a new location which is dx and dy pixels away horizontally and vertically from the current location.
above(dy)
Get a new location which is dy pixels vertically above the current location.
below(dy)
Get a new location which is dy pixels vertically below the current location.
left(dx)
Get a new location which is dx pixels horizontally to the left of the current location.
right(dx)
Get a new location which is dx pixels horizontally to the right of the current location.
3.1.5 Match
class Match
An object of class Match represents the result of a successful nd operation. It has the rectangle dimension of the
image, that was used to search. It knows the point of its upper left corner on an existing monitor, where it was found.
Since class Match extends class Region, all methods of class Region can be used with a match object.
Creating a Match, Getting Attributes
A match object is created as the result of an explicit nd operation. It can be saved in a variable for later use with
actions like click().
It has the rectangle dimension of the image, that was used to search. It knows the point of its upper left corner on an
existing monitor, where it was found. It knows the similarity it was found with and a click point to be used, if set by a
pattern.
For all other aspects, the features and attributes of class Region apply.
class Match
68 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
getScore()
Get the similarity score the image or pattern was found. The value is between 0 and 1.
getTarget()
Get the location object that will be used as the click point.
Typically, when no offset was specied by Pattern.targetOffset(), the click point is the center
of the matched region. If an offset was given, the click point is the offset relative to the center.
Iterating over Matches after ndAll()
A nd operation Region.findAll() returns an iterator object that can be used to fetch all found matches as
match objects one by one. A reference to the iterator is stored in the respective region and can be accessed using
Region.getLastMatches().
Important to know:
per denition, an iterator can be stepped through only once - it is empty afterwards
You can read more about the basics of operations with iterators from the description of Finder class. To save
contained matches for later use, you can convert them to list.
Example: using while: with default screen
Example: using with: with default screen
3.1.6 Pattern
class Pattern
A pattern is used, to associate an image le with additional attributes used in nd operations and when acting on a
match object.
Minimum Similarity:
While using a Region.find() operation, if only an image le is provided, Sikuli searches the region using a default
minimum similarity of 0.7. This default value can be changed in Settings.MinSimilarity.
Using similar() you can associate a specic similarity value, that will be used as the minimum value, when this
pattern object is searched. The IDE supports adjusting the minimum similarity of captured images using the Preview
Pane (internally in the script, the images are turned into a pattern object automatically).
Click Point:
Normally when clicking on a match, the center pixel of the associated rectangle is used. With a pattern object, you
can dene a different click point relative to the center using targetOffset().
class Pattern
Pattern(string)
Parameters string a path to an image le
Returns a new pattern object
This will initialize a new pattern object without any additional attributes. As long as no pattern methods
are used additionally, it is the same as just using the image le name itself in the nd operation.
similar(similarity)
Return a newPattern object containing the same attributes (image, click point) with the minimumsimilarity
set to the specied value.
3.1. Everything about Sikuli Script 69
SikuliX Documentation, Release 1.1.0-Beta1
Parameters similarity the minimum similarity to use in a nd operation. The value should be
between 0 and 1.
Returns a new pattern object
exact()
Return a newPattern object containing the same attributes (image, click point) with the minimumsimilarity
set to 1.0, which means exact match is required.
Returns a new pattern object
targetOffset(dx, dy)
Return a new Pattern object containing the same attributes (image, similarity), but a different denition
for the click. By default, the click point is the center of the found match. By setting the target offset, it
is possible to specify a click point other than the center. dx and dy will be used to calculate the position
relative to the center.
Parameters
dx x offset from the center
dy y offset from the center
Returns a new pattern object
getFilename()
Get the lename of the image contained in the Pattern object.
Returns a lename as a string
getTargetOffset()
Get the target offset of the Pattern object.
Returns a Location object as the target offset
3.1.7 Finder
class Finder
A Finder object implements an iterator of matches and allows to search for a visual object in an image le that you
provide (e.g. a screenshot taken and saved in a le before). After setting up the nder object and doing a nd operation,
you can iterate through the found matches if any.
Important to know:
per denition, an iterator can be stepped through only once - it is empty afterwards
it has to be destroyed using finder.destroy(), especially when used with for: or while:
when used in a with: construct, it is destroyed automatically
Compared with the region based nd operation, no exception FindFailed is raised in case nothing is found at all (use
hasNext() to check). The nder object can be compared to what you get with region.getLastMatches()
when using findAll().
Note: With this version, there is no chance, to get the number of matches in advance. If you would iterate through to
count, afterwards your nder would be empty. So in this case, you have to save your matches somehow (one possible
solution see example below).
The workow always is, that you rst do a nd operation and afterwards go through the matches found. After a
complete iteration, the nder object is empty. So you could start a new nd operation again.
class Finder
70 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
Finder(path-to-imagele)
Create a new nder object.
Parameters path-to-imagele lename to a source image to search within
find(path-to-imagele[, similarity ])
Find a given image within a source image previously specied in the constructor of the nder object.
Parameters
path-to-imagele the target image to search for
similarity the minimum similarity a match should have. If omitted, the default is used.
hasNext()
Check whether there are more matches available that satisfy the minimum similarity requirement.
Returns True if more matches exist.
next()
Get the next match.
Returns a Match object.
The returnd reference to a match object is no longer available in the nder object afterwards. So if it is
needed later on, it has to be saved to another variable.
Example 1: basic operations using a Finder
Example 2: we want to know how many matches (based on the previous example).
3.1.8 Key Constants
class Key
Applicable usage situations for these predened constants of special keys and key modiers can be found in Acting on
a Region and Low Level Mouse and Keyboard Actions.
Special Keys
The methods supporting the use of special keys are type(), keyDown(), and keyUp().
Usage: Key.CONSTANT (where CONSTANT is one of the following key names).
String concatenation with with other text or other key constants is possible using +.
type("some text" + Key.TAB + "more text" + Key.TAB + Key.ENTER)
# or eqivalent
type("some text\tmore text\n")
miscellanous keys
ENTER, TAB, ESC, BACKSPACE, DELETE, INSERT
New in version X1.0-rc3.
miscellanous keys
SPACE
function keys
3.1. Everything about Sikuli Script 71
SikuliX Documentation, Release 1.1.0-Beta1
F1, F2, F3, F4, F5, F6, F7, F8, F9, F10, F11, F12, F13, F14, F15
navigation keys
HOME, END, LEFT, RIGHT, DOWN, UP, PAGE_DOWN, PAGE_UP
special keys
PRINTSCREEN, PAUSE, CAPS_LOCK, SCROLL_LOCK, NUM_LOCK
New in version X1.0-rc2.
Note: The status ( on / off ) of the keys Key.CAPS_LOCK, Key.NUM_LOCK and Key.SCROLL_LOCK can be
evaluated with the method Env.isLockOn().
numpad keys
NUM0, NUM1, NUM2, NUM3, NUM4, NUM5, NUM6, NUM7, NUM8, NUM9
SEPARATOR, ADD, MINUS, MULTIPLY, DIVIDE
modier keys
ALT, CMD, CTRL, META, SHIFT, WIN
These modier keys cannot be used as a key modier with functions like type(), rightClick(), etc. They can
only be used with keyDown() and keyUp(). If you need key modiers, use KeyModifier instead.
Key Modiers
Methods where key modiers can be used include: click(), dragDrop() , doubleClick() ,
rightClick(), type().
Deprecated since version X1.0-rc3.
the oldies but goldies
KEY_ALT, KEY_CTRL, KEY_SHIFT
system specic Win/Mac
KEY_WIN, KEY_CMD
KEY_META (a synonym for KEY_WIN or KEY_CMD on Windows and Mac respectively).
The old modiers with a KEY_ prex are deprecated. Use KeyModifier.CTRL, KeyModifier.ALT,
KeyModifier.SHIFT, KeyModifier.META instead.
New in version X1.0-rc3.
class KeyModifier
Usage: KeyModier.CONSTANT (where CONSTANT is one of the following key names).
CTRL
equivalent to the old KEY_CTRL
SHIFT
equivalent to the old KEY_SHIFT
ALT
equivalent to the old KEY_ALT
META
equivalent to the old KEY_META
72 Chapter 3. Complete Guide
SikuliX Documentation, Release 1.1.0-Beta1
CMD
equivalent to the old KEY_CMD (and KEY_META)
WIN
equivalent to the old KEY_WIN (and KEY_META)
The modier constants can be combined to the modier parameter by either using + or |, if more than one key
modier is needed.
type(Key.ESC, KeyModifier.CTRL + KeyModifier.ALT)
# or equivalent
type(Key.ESC, KeyModifier.CTRL | KeyModifier.ALT)
They should only be used in the modiers parameter with functions like type(), rightClick(), etc.
They should never be used with keyDown() or keyUp().
Note for Java programming: These constants are mapped to the according constants of the Java environment in the
class java.awt.event.InputEvent.
3.1. Everything about Sikuli Script 73
SikuliX Documentation, Release 1.1.0-Beta1
74 Chapter 3. Complete Guide
CHAPTER 4
Extensions
4.1 Extensions
4.1.1 General Information About Sikuli Extensions
This feature is currently under developement and might only be available on request.
Extensions allow to implement new Sikuli features by adding packages to your current Sikuli installation.
They are maintained by the developers (see Technical Details below).
If you want to contribute a new extension or a modied exisiting one, please look at How to contribute an extension
below.
How to Download and use an Extension
The download of an extension is supported by the IDE through the menu Tools -> Extensions. You get a popup, that
lists the available and already installed extensions and allows to download new packages or updates for installed ones.
This popup shows a new package not yet installed:
If you need more information about the features of the extension, just click More Info - this will open the related
documentation from the web in a browser window.
If you want to install the extension, just click the Install... button. The package will be downloaded and added to your
extensions repository.
This popup shows an installed package:
75
SikuliX Documentation, Release 1.1.0-Beta1
If a new version would be available at that time, the Install... button would be active again, showing the new version
number. Now you could click and download the new version.
How to Use an Extension
To use the features of an installed extension in one of your scripts, just say from extension-name import
*
.
For an usage example read Sikuli Guide.
For information about features, usage and API use menu Tools -> Extensions -> More Info in the IDE.
Technical Details
Extensions are Java JAR les containing some Java classes (usually the core functions) and/or Python modules, which
dene the API to be used in a script.
Sikuli maintains a local extensions directory, where downloaded extensions are stored together with a hidden list
of the installed extensions (Windows: %APPDATA%\Sikuli\extensions, Mac: ~/Library/Application
Support/Sikuli/extensions, Linux: ~/.sikuli/extensions).
Once an extension is imported using import extension-name, Sikuli automatically searches and loads the JAR
le of that extension into the current context with load(path-to-jar-file).
How to develop an extension
The source structure of an extension named extension-name looks like this:
Java
- org/com
-- your-organization-or-company
--- extension-name
---- yourClass1.java
---- yourClass2.java
---- .... more classes
python
- extension-name
-- __init__.py
-- extension-name.py
The nal structure of a JAR (lename extension-name-X.Y where X.Y is the version string) looks like this:
org/com
- your-organization-or-company
-- extension-name
--- yourClass1.class
76 Chapter 4. Extensions
SikuliX Documentation, Release 1.1.0-Beta1
--- yourClass2.class
--- .... more classes
extension-name
- __init__.py
- extension-name.py
META-INF
- MANIFEST.MF
The le __init__.py contains at least from extension-name import
*
to avoid one qualication level.
So in a script you might either use:
import extension-name
extension-name.functionXYZ()
or:
from extension-name import
*
functionXYZ()
The second case requires more investement in a naming convention, that avoids naming conicts.
The le extension-name.py contains the classes and methods, that represent the API, that one might use in a
Sikuli script.
As an example you may take the source of the extension Sikuli Guide.
Name your extensions properly
Sikuli extensions can be Python/Jython modules or Java classes.
For Java classes, following the reverse URL convention of Java is a good idea (for example, org.foo.your-extension).
However, DONOTuse Javas convention for Python/Jython modules. You need to come up with a unique extension
name that does not conict with existing Python modules and other Sikuli extensions.
Please read Naming Python Modules and Packages to learn the details for naming a Python module.
How to test your extension
While developing your extensions, you can put the JAR le in Sikulis extension directory or in the same .sikuli folder
as your test script. The JAR le should not have a version number in its le name, e.g. extension-name.jar.
Because Sikuli starts to search extensions in the .sikuli folder of the running script and then in the Sikuli extensions
folder, it is usually a good idea to put your developing extensions in the .sikuli folder of your test script.
Another option is to use the load() function with an absolute path to your extension-name.jar. If this fails,
Sikuli goes on searching in the current .sikuli folder and then in the Sikuli extensions folder. If load() succeeds, it
returns True and puts absolute-path-to-your-extension-name.jar into sys.path, so you can use
import extension-name afterwards.
How to contribute your extension
Currently you have to contact the developers of Sikuli and agree on how to proceed.
4.1.2 Sikuli Guide
New in version X1.0-rc2.
4.1. Extensions 77
SikuliX Documentation, Release 1.1.0-Beta1
Sikuli Guide is an extension to Sikuli that provides a revolutionary way to create guided tours or tutorials for GUI
applications. The revolutionary aspect is that the content of the tours or tutorials can be displayed right on the actual
interface, rather than in a video or a series of screenshots on a web page. All this can be combined with guided user
activities directly in the respective GUI applications using all the other Sikuli features.
Quick Start
First Example
In our rst example, suppose we want to create a guided tour of this very documentation page you are currently
reading. We want to bring your attention to the logo picture to the right. Using the functions provided by Sikuli Guide,
we can write the following script to accomplish this:
When you run this script, Sikuli Guide will search for the logos image on the screen, highlight it, and display the text
This is Sikulis logo below the image, like the gure below:
Again, this happens in the actual interface, rather than in a video or a screenshot. The logo image that is highlighted
is the actual interface element users can click on.
Lets explain the script line by line. The rst line is an import statement that tells Sikuli to load the Sikuli Guide
extension. The secod line uses the text(pattern, text) function to add text next to a given pattern, in
this case, the logo image. Note that by default the text added is not displayed immediately, it is only internally added to
the visual element. In the third line, we call show(secs) to explicitly tell Sikuli Guide to now display all registerd
annotation elements (in this case only the text) for the duration specied by secs.
Below is a YouTube video of this example.
Adding Multiple Annotations
It is possible to add text or other annotations to multiple visual elements before calling show(secs) in order to show
them on the screen at the same time.
The script above uses the function tooltip(text) to add tooltips to three links in addition to the text annotation.
The result of running this script is shown below:
78 Chapter 4. Extensions
SikuliX Documentation, Release 1.1.0-Beta1
Rather than showing the annotations all at once, we can also show them one by one using separate show() statements.
Below is an example where we cycle through the three links and show the tooltip of each link one at a time.
The result of running this script is shown below (3x speed-up):
Adding Interaction
Another way to control the ow of a guided tour is to display a dialog box and let users click on a button to continue
to the next part of the tour. Sikuli Guide provides a function dialog(message) to accomplish this easily. Below
is an example using this function to create a two-part guided tour.
The tour presented by the script above introduces the navigation links above and the Sikulis logo as a shortcut to go
back to the documentations HOME page. The function call dialog("Part 1") indicates the tour will show a
dialog that displays the message specied by the string argument (i.e., Part 1: Navigation Links). The following call
to show() will actually display the dialog along with the text elements spcied earlier.
The gure below shows what happens after Line 3:
After users click on the Next button, the tour moves to the next part. The screen will look like below:
4.1. Extensions 79
SikuliX Documentation, Release 1.1.0-Beta1
Making a Region clickable
You might use the feature clickable(PSRM), to make a region sensitive for clicks.
The script waits until the user clicks one of the two highlighted areas.
Function References
PSRM: when used as a parameter, it can be either P a Pattern, S a string (image le name or just plain text), a R
Region object or M a Match object. With PS an implicit nd operation takes place. (More information: Finding inside
a Region ...)
Static Annotations
guide.rectangle(PSRM)
Add a rectangular overlay in red on the specied targets region.
Parameters PSRM a pattern, string, region or match
guide.circle(PSRM)
Add a red circle around the specied targets region.
Parameters PSRM a pattern, string, region or match
80 Chapter 4. Extensions
SikuliX Documentation, Release 1.1.0-Beta1
guide.text(PSRM, text)
Add some text (white large letters on dark grey background) left justied below the specied targets region,
which is additionally highlighted.
Parameters
PSRM a pattern, string, region or match
text a string as text to display
guide.tooltip(PSRM, text)
Add a tooltip (small text in a light yellow box) left justied below the specied targets region.
Parameters
PSRM a pattern, string, region or match
text a string as text to display
Interactive Elements
guide.dialog(text)
Add a dialog box displaying the given text in the middle of the screen above all othe windows.
Parameters text a string as text to display
guide.clickable(PSRM)
Add a clickable element corresponding to the specied targets region.
Parameters PSRM a pattern, string, region or match
Control
guide.show([seconds ])
Show static and interactive components added so far for the specied amount of time.
Parameters seconds a decimal number as display duration in seconds
The default duration is 10 seconds. If interactive elements (either one or more clickable elements or a dialog
box) were previously added, it waits until the user interacts with one of these elements. At this time all elements
vanish and are discarded.
Note: If a dialog() element is present, other interactive elements are visible, but not clickable. If the dialog
element is clicked, all elements vanish and are discarded.
4.1. Extensions 81
SikuliX Documentation, Release 1.1.0-Beta1
82 Chapter 4. Extensions
CHAPTER 5
For Developers
5.1 How Sikuli Works
5.1.1 Sikuli Script
Sikuli Script is a Jython and Java library that automates GUI interaction using image patterns to direct keyboard/mouse
events. The core of Sikuli Script is a Java library that consists of two parts: java.awt.Robot, which delivers keyboard
and mouse events to appropriate locations, and a C++ engine based on OpenCV, which searches given image patterns
on the screen. The C++ engine is connected to Java via JNI and needs to be compiled for each platform. On top of the
Java library, a thin Jython layer is provided for end-users as a set of simple and clear commands. Therefore, it should
be easy to add more thin layers for other languages running on JVM, e.g. JRuby, Scala, Javascript, etc.
83
SikuliX Documentation, Release 1.1.0-Beta1
5.1.2 The Structure of a Sikuli source folder or zipped le (.sikuli, .skl)
A Sikuli script (.sikuli) is a directory that contains a Python source le (.py) representing the automation workow or
the test cases and all the image les (.png) used by the source le. All images used in a Sikuli script are simply a path
to the .png le in the .sikuli bundle. Therefore, the Python source le can also be edited by any text editor.
While saving a script using Sikuli IDE, an extra HTML le may optionally be created in the .sikuli directory so that
users can share a visual copy of the scripts on the web easily.
A Sikuli zipped script (.skl) is simply a zipped le of all les in the .sikuli folder. It is intended for distribution via mail
or web upload, can also be run from command line and reopened in the Sikuli IDE. (The previous naming as Sikuli
executable is deprecated, since this is misleading: people most often thought, it is something like a self-contained
and self-running package comparable to a Windows EXE, but it is not).
5.2 How to get involved
Have fun working with Sikuli? You can do more than just being a user! There are many ways you can help Sikulis
development:
Blog or tweet about Sikuli. Share your cool Sikuli scripts to the world and let more people know how cool Sikuli
is.
Report bugs or request new features in our bug tracker.
Visit the question board and the bug tracker regularly and answer peoples questions there. (You may want to
subscribe to the bug tracker or subscribe to all questions.) Many people have questions that you may know how
to deal with. Help them to get through the obstacles and they may help you in the future.
Submit patches to x bugs or add features.
Join the sikuli-dev mailing list and share your ideas to us.
Translate Sikuli into your language and help more people who speak different languages to access Sikuli. You
can help us to
translate Sikuli IDEs interface and messages, or
translate Sikuli documentation (this site you are reading).
Read Internationalization and localization for more details.
5.2.1 Submit Patches
If you are interested in making Sikuli better, submitting patches is a good start. We welcome any patches to Sikulis
code. If youve found a bug, submitting a bug report with associated patches will get it xed more quickly than those
without patches.
Claim a bug
Here is a typical bug report status.
If you see a bug that you think you could x, feel free to claim the bug. Simply click the pencil icon under the
Assigned to column, and then click Assign Me. Claiming a bug let other people and developers know that this bug
is being worked on, and prevents duplicate work.
84 Chapter 5. For Developers
SikuliX Documentation, Release 1.1.0-Beta1
Bug claimers responsibility
Once youve claimed a bug, you are responsible to work on that bug in reasonable time, say one or two weeks. If you
dont have time to work on it, please unclaim it.
If youve claimed a bug and its taking a long time to code, please keep everybody updated by posting comments on
the bug. If you dont update regularly, and you dont respond to a request for a progress report, your claim on the bug
may be revoked.
Generate and Submit Patches
1. Branch the Sikuli source tree using bzr branch lp:sikuli.
2. Fix bugs or add new features.
3. Generate your patches using bzr diff, so we can incorporate your patches into our code base esaily.
4. Name the patch le with a .diff extension.
5. Click Add attachment or patch below the comment area, and attach your patch le.
6. If the patch adds a new feature, or modies existing behavior, please leave documentation in the comment area.
If you want to contribute changes that involve hundred lines of code, Please register a branch on Sikuli and then
propose for merging once you have done all changes.
5.2.2 Internationalization and localization
We hope Sikuli can be access by anyone from anywhere in the world. You can help us to translate the user interface
of Sikuli IDE or this documentation.
Sikuli IDE Translation
If you nd an incorrect translation, or if you would like to add a language that isnt yet translated, heres what to do:
Join the Sikuli i18n mailing list and introduce yourself.
Claim what language and what (in this case, the Sikuli IDE) you are going to work on.
Visit our translations page on launchpad and use launchpads tool to translate the interface items.
If there are items you are not sure how to translate, please mark them as need review.
Once youve done the translations, send a mail to the Sikuli i18n mailing list so we can incorporate your work
into the Sikuli IDE.
Documentation Translation
This documentation is created using Sphinx, and written in the reStructuredText format. You can view how the source
code looks like using the link Show Source in the side bar.
To translate the documentation, you need to check out the source of the document using bzr branch lp:sikuli. The
documentation source is in the folder docs/. Once you have the source, you can generate this HTML document
using make html under the docs/ directory.
The translation for each language is in docs/i18n/<language>/source. The source directory for your
language should mirror the main source directory docs/source.
5.2. How to get involved 85
SikuliX Documentation, Release 1.1.0-Beta1
If you nd an incorrect translation in the documentation, or if you would like to add a language that isnt yet translated,
heres what to do:
Join the Sikuli i18n mailing list and introduce yourself.
Claim what language and which part of the documentation you are going to work on.
Once youve done the translations, generate a patch using bzr diff and name the patch with a diff extension.
Send a mail to the Sikuli i18n mailing list and attach the patch, so we know what youve done and we can put
your translations online.
If you contribute translations very actively, we can give you the commit permission to our bzr source tree so you can
submit translations by yourself.
5.3 Changes in Sikuli
5.3.1 X 1.1.0 (2014-01-31)
86 Chapter 5. For Developers
Python Module Index
g
guide, ??
87