artoftest-runner.md

title	page_title	description	position
ArtOfTest.Runner.exe - Test Studio CLI Runner	Test Studio CLI Runner "ArtOfTest.Runner.exe"	Test Studio Command line runner client is called ArtOfTest.Runner.exe. Test Studio execution engine in the command prompt	1

Test Studio CLI Runner ArtOfTest.Runner.exe

The CLI runner ArtOfTest.Runner.exe is deployed with any of the Test Studio editions in the Test Studio installation directory. The default install folder where you can find the runner is C:\Program Files (x86)\Progress\Test Studio\Bin.

This article guides you trough the options of the Test Studio CLI Runner for building commands to execute tests.

• How to Call the ArtOfTest.Runner.exe in Command Prompt
• Help Screen
• Options to Specify Which File to Execute
• Options to Customize the Results File
• Option to Use Custom Settings for the Execution

How to Call the ArtOfTest.Runner.exe in Command Prompt

Similar to any other executable files you have few options to call the ArtOfTest.Runner.exe in the command prompt.

• Use the full file path to call the *.exe.
• Change the context root directory in the command prompt window to the one containing the *.exe file and use the *.exe name directly.
• Set the *.exe directory as environmental variable in the OS and use the *.exe name directly from whichever directory is the current context.

Help Screen

Use the help option to get a list of all options supported by the CLI runner. The short form to use is /? or /h. Below is the Help Screen for ArtOfTest.Runner.exe:

Options to Specify Which File to Execute

The ArtOfTest.Runner.exe supports the execution of a single test or a list of tests. Choose the option suitable for your needs.

test Option

The test option accepts full file path to an individual test with the *.tstest file extension.

Note!
Each Test Studio test is part of a project and it can't exist on its own. This is why running individual tests requires a project Settings.aiis file. So, you can use the test option by itself only when the test file is in the project root folder where the Settings.aiis file.

Otherwise, when the test is nested under any folder in the project use the test option in combination with the root one to specify the associated project root folder where the Settings.aiis file is.

Example Commands for test Option

• Run individual test stored in the project root folder:

> "C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe" 
test="D:\Test Studio Projects\July2024\demoTest-AOTRunner.tstest"

• Run individual test stored in a sub-folder under the project root one:

>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe" 
test="D:\Test Studio Projects\July2024\inProjectFolder\nestedDemoTest.tstest" 
root="D:\Test Studio Projects\July2024"

list Option

The list option takes the full path to a test list file with the *.aiilist file extension.

Note!
Each Test Studio test list is part of a project and it can't exist on its own. Test list files are stored in the TestLists sub-folder under the project root one.
The *.aiilist file contains the test list settings applied for it in the Test Studio project, so you can use the list option by itself.

Example Commands for list Option

• Run test list:

>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe" 
list="D:\Test Studio Projects\July2024\TestLists\demoList-ArtOfTest.aiilist"

root Option

The root option takes the full path to the project root folder and is used in combination with either test, or list options.

Options to Customize the Results File

By default the result files generated from the CLI runner execution are output in the Results sub-folder under the project’s root directory.The ArtOfTest.Runner.exe supports outputting the results from a test/test list run in different formats and saving the file in custom location.

Tip
All result related options can be used in combination or on their own.

out Option

The out option defines an alternative folder to store the results to and takes the full path to the specified folder.

Example Commands for out Option

• Run test list outputting the results in specified folder:

>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe" 
list="D:\Test Studio Projects\July2024\TestLists\demoList-ArtOfTest.aiilist" out="D:\SharedFolder"

result Option

The result sets specific name for the default result file and takes a file name in double quotes.

Example Commands for result Option

• Run test list outputting the results in specified folder and setting custom name for the result file:

>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe" 
list="D:\Test Studio Projects\July2024\TestLists\demoList-ArtOfTest.aiilist" out="D:\SharedFolder" 
result="customName"

xml or html Options

Use either the xml, or the html option to output the result into the corresponding file type - *.xml or *.html file. The different format result file is output in addition to the default result file with file extension *.aiiresult.

Note
xml or html option is valid only when executing a test list.

Example Commands for xml or html Options

• Run test list outputting the results in xml format:

>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe" 
list="D:\Test Studio Projects\July2024\TestLists\demoList-ArtOfTest.aiilist" 
xml

• Run test list outputting the results in html format in specified folder and setting custom name for the result files:

>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe" 
list="D:\Test Studio Projects\July2024\TestLists\demoList-ArtOfTest.aiilist" out="D:\SharedFolder" 
result="customName" 
html

Note
Result files are saved into the project Results sub-folder if the out option is not used, and with default name if the result option is not used.

junit or junitstep Options

Use the junit or junitstep options to output the result into a junit(step).xml result file. The difference between the two is whether to convert a test or a test step to a junit test in the output result.

Example Commands for junit or junitstep Options

• Run test list outputting the results in junitstep format in specified folder and setting custom name for the result files:

>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe" 
list="D:\Test Studio Projects\July2024\TestLists\demoList-ArtOfTest.aiilist" 
out="D:\SharedFolder" 
result="customName" 
junitstep

• Run single test outputting the results in junit format:

>>"C:\Program Files (x86)\Progress\Test Studio\Bin\ArtOfTest.Runner.exe" 
test="D:\elle\Test Studio Projects\July2024\demoTest-AOTRunner.tstest" 
junit

Note
Result files are saved into the project Results sub-folder if the out option is not used, and with default name if the result option is not used.

persistOnEachStep Option

By default the PersistOnEachStep option is false. If set to true it saves the results after each executed step. It is used when debugging specific cases when the final result is not generated as expected.

Option to Use Custom Settings for the Execution

By default the CLI runner execution uses the settings applied in the Settings.aiis file for a single test run, or the settings specified in the executed test list. The ArtOfTest.Runner.exe supports customizing the execution by using custom settings file.

settings Option

The settings option takes the full path to a JSON file containing custom settings for the run.

Below is an example of a complete JSON setting file that contains all of Telerik's test/test list run configuration settings.

{
  "Settings": {
      "__type": "ArtOfTest.WebAii.Core.Settings",
      "__value": {
        "ResponsiveWeb": {
          "__type": "ArtOfTest.WebAii.Core.Settings+ResponsiveWebSettings",
          "__value": {
            "Width": 0,
            "Height": 0,
            "UserAgent": null
          }
        },
        "Web": {
          "__type": "ArtOfTest.WebAii.Core.Settings+WebSettings",
          "__value": {
            "IsProfilingExecution": false,
            "ExecutingBrowsers": [],
            "UseMultiBrowserExecution": false,
            "RecycleBrowser": false,
            "AspNetDevServerPort": -1,
            "LocalWebServer": 0,
            "EnableUILessRequestViewing": false,
            "WebAppPhysicalPath": "",
            "DefaultBrowser": 10,
            "EnableScriptLogging": false,
            "BaseUrl": "",
            "KillBrowserProcessOnClose": false,
            "KillBrowsersBeforeStart": true,
            "AutoCalibrateBrowsers": false,
            "UseHttpProxy": false,
            "EnableSilverlight": false,
            "VerboseHttpProxy": false,
            "SilverlightConnectTimeout": 60000,
            "SilverlightApplicationPath": null,
            "WebComponents": true,
            "UseBrowserExtension": false,
            "ChromiumFirstInteractionDelay": 500,
            "FirefoxMinimumJSClickDelay": 100
          }
        },
        "Wpf": {
          "__type": "ArtOfTest.WebAii.Core.Settings+WpfSettings",
          "__value": {
            "DefaultApplicationPath": null,
            "DefaultApplicationArgs": null,
            "DefaultApplicationWorkingFolder": null
          }
        },
        "Desktop": {
          "__type": "ArtOfTest.WebAii.Core.Settings+DesktopSettings",
          "__value": {
            "DefaultApplicationPath": null,
            "DefaultApplicationArgs": null,
            "DefaultApplicationWorkingFolder": null
          }
        },
        "CreateLogFile": true,
        "LogLocation": "C:\\WebAiiLog\\",
        "QueryEventLogErrorsOnExit": false,
        "LogAnnotations": false,
        "SimulatedMouseMoveSpeed": 0.3,
        "ExecuteInDevelopmentTests": false,
        "WaitCheckInterval": 500,
        "ElementWaitTimeout": 15000,
        "SearchByImageFirst": false,
        "ExecuteCommandTimeout": 20000,
        "ExecutionDelay": 0,
        "RerunFailedTests": false,
        "UnexpectedDialogAction": 2,
        "AnnotateExecution": false,
        "AnnotationMode": 0,
        "RecordingMode": 0,
        "RecordingCodec": 0,
        "RecordingScale": 100,
        "RecordingFPS": 5,
        "RecordingOutputLocation": "",
        "RecordingSizeLimit": 0,
        "XMultiMgr": true,
        "ClientReadyTimeout": 60000,
        "DisableDialogMonitoring": false,
        "RunnerResponseTimeout": 0.0,
        "EnableImageSearch": true,
        "ElementImageSearchTimeout": 15000,
        "ElementImageSearchDelay": 300,
        "ScrollOnImageSearch": true,
        "TelerikComponentsVersion": 0
      }
    },
  "WebSettings": null,
  "PropertyBag": null
}

Tip
The settings listed in the settings file correspond to these available in the test list settings.

Publish Results To TFS

Note: All options listed in the help screen in relation to results publishing to TFS are mandatory. Below are some additional notes to each of the options.

• The server option takes the TFS server name with its full path in double quotes like this http://myTFS.myDomain.com:8080/tfs.

• The build option takes the respective build which the results could be associated to.

• The project option takes the name of the current team project which the build belongs to.

• The platform option takes the respective platform which is usually Windows and thus the option takes value "win".

• The flavor defines if the build is Debug or Release (defaults to "Debug").

Below is a sample command to publish results to TFS from a test list execution.

Exit Codes

ArtOfTest.Runner.exe returns an exit code so the Build Server can check for it on process exit in case of an exception:

<style> table.docs { font-family: verdana,arial,sans-serif; font-size:11px; color:#333333; border: 1px solid #dbdbdb; border-collapse: collapse; } table.docs th { color:#fff; background-color:#00аб8е; border: 1px solid #dbdbdb; padding: 8px; } table.docs tr { background-color:#ffffff; } table.docs td { border: 1px solid #dbdbdb; padding: 8px; } </style>

Code	Title	Summary
**0**	RUN_TESTS_SUCCESS	Run is processed and all tests passed.
**1**	RUN_TESTS_ERROR	Run is processed and some tests failed.
**2**	NOT_RUN_UNEXPECTED_ERROR	Run is not processed due to unexpected error.
**3**	NOT_RUN_COMPILATION_ERROR	Run is not processed due to compilation errors.
**10**	NOT_RUN_USAGE_INFO	No run to process, requested usage info.
**11**	NOT_RUN_INVALID_COMMAND_ARGS	No run to process due to general invalid arguments error.
**12**	NOT_RUN_TEST_NOT_FOUND	Run is not processed due to invalid test path argument.
**13**	NOT_RUN_TESTLIST_NOT_FOUND	Run is not processed due to invalid test list path argument.