Introduction to pyATS

Introduction to pyATS

pyATS - Python Automated Test Systems is a Python3 based test automation infrastructure. It
is developed as the next-generation Cisco mainstream automation infrastructure, is
compatible to existing tooling & infrastructure, and is officially endorsed by Cisco executive
team.

pyATS is 100% fully developed and implemented using Python and is designed to be usable
independent of any other existing infrastructure. It is engineered to be suitable for agile, rapid
development iterations, with the ability to handle a wide variety of testing scenarios and
requirements, from white-box unit-testing, to black-box, feature, performance/scale testing &
beyond.

Release to DevNet

pyATS started its journey initially as an internal Cisco Engineering tool. Over the years, it
amassed thousands of test developers and millions of lines of test scripts and libraries. As
pyATS gathered momentum within Cisco, teams began inquiring whether it was possible to
share their automation with customers as part of collaborative development and release.

As a result, pyATS has been made available externally to Cisco’s end customers, through
DevNet. Today, pyATS is downloadable through Python PyPI:
What is pyATS?

pyATS is the test framework foundation for the ecosystem. pyATS specializes in data-driven
and reusable testing for agile, rapid development iterations.

This powerful, highly pluggable Python® framework enables developers to start with small,
simple and linear test cases, and then scale up to large, complex and asynchronous test suites.
pyATS Key Benefits

The pyATS ecosystem provides an agnostic infrastructure for rapid test development and
execution. All OS/Platform and management protocol support is defined and injected through
plugins, library implementations, and extensions. It includes support for third-party platforms
and protocols.

Network engineers and NetDevOps can be productive at day one with the pyATS Library’s
readily available, holistic, and model-driven libraries, which:

 facilitate rapid development

 encourage the use of re-usable tests, and
 simplify how you write and execute test automation scripts.
Script developer key benefits:

 Parse device output for stateful validation.

 Reuse any of the available test cases.
 Connect to devices over Telnet, SSH, REST, RESTCONF, NETCONF, and YANG.
 Use a single, cross-platform script instead of having to modify multiple scripts.
 Verify CLI outputs with YANG and XML outputs.

Network engineer key benefits

 Connect to devices and make sure that they are up, running and pingable.
 Retrieve information about the network state.
 Take before and after snapshots of the network, and then compare them.
 Parse device output and store a snapshot, and then automatically run the same
commands at specified time intervals to compare the current and previous states.
 Use an available test case to verify a stable network topology after an image upgrade,
for example.
 Perform typical actions such as Reload Devices, Perform Switchover, ShutNoShutBgp,
and ConfigUnconfigOspf.

How does pyATS work?

pyATS provides a framework that standardizes:

 How network topologies are defined and modeled

 How to programmatically interact with devices (by means of connection libraries)
 How test scripts are defined and executed
 How test runs are performed and how reports are generated

The pyATS Library builds on pyATS to provide:

 An easy-to-use Linux style command-line interface (CLI)

 Ready-to-use libraries that implement pyATS features
 Parsers, device, and feature configuration models, as well as operational status models
 Reusable test cases in the form of triggers and verifications
 The ability to build a test suite elastically using YAML-based datafiles
  A mechanism for modeling network device features

pyATS has 4 main functionalities:

 Configure devices in an OS implementation agnostic way

 Retrieve device operation state in an OS implementation agnostic way
 Pool of libraries to reuse which includes Testcases.
 Test Harness which ties it all together.

The fundamental four core objects of pyATS include:

 Genie Conf: Configures topology through Python object attributes, featuring a

common object structure. These object’s structures are compatible with all operating
systems and Management Interfaces (such as CLI/Yang/REST, etc.). These structures
represent the feature, not a specific implementation on an operating system. As such,
scripts can use these objects to configure their device, do some configuration changes
and expect it to work across all their operating system, management interfaces.
(Depending on libraries availabilities).
 Genie Ops: Represents the operational state of the feature through object attributes,
featuring a common object structure. Just like Genie Conf, Genie Ops is compatible
with all operating systems and Management Interfaces.
 Genie SDK: provides engineers with a set of test library (a pool of triggers and
verifications) that, when combined, create numerous testcases and test scenarios on
any topology and configuration. Additionally, Genie SDK contains diverse libraries for
triggers and verifications development.
 Genie Harness: Genie Harness fuses pyATS infrastructure with Genie Conf, Genie Ops,
and Genie SDK to create a generic test automation framework, which promotes the
Event Driven testing paradigm. Instead of writing Testscript, testscript are built by
choosing Triggers, verification from the SDK pools or your own pool of testcases. Genie
Harness provides many features to enhance your testing, for example: Traffic
generator integration, applying configuration on the device, profiling the system to
make sure only expected fields have been changed.
Importantly, the innovative modular design of the Core Components means they may be used
independently or collectively in pyATS testscripts and/or testcases.

pyATS Ecosystem

pyATS and the pyATS Library together define an ecosystem that streamlines and standardizes
how you set up and run automated network tests. pyATS and the pyATS library provide sanity,
feature, solution, system, and scale test automation for any type of device or virtual device.
pyATS is currently used with devices such as routers and switches, access points, firewalls,
Linux servers, phones, cable CPEs, and many more.

Originally developed for internal Cisco engineers, the pyATS ecosystem is at the core of Cisco’s
Test Automation Solution. It’s currently used by Cisco engineering, DevNet engineers, network
engineers, and developers.

The pyATS ecosystem empowers your team to create and run consistent, repeatable, iterative,
and reusable tests. pyATS provides the test framework, and the pyATS Library offers ready-to-
use test components.
The pyATS ecosystem can learn and profile an entire feature’s configuration and operational
status. For example, with just a few commands, you can profile your system before and after a
configuration change and get a detailed summary of exactly what changed.

pyATS Library System

The pyATS Library (a.k.a. Genie) provides all the tools needed for network test automation,
including

 a pool of reusable parsers

 a pool of reusable triggers
 a pool of reusable APIs
 a simple command line interface (no Python knowledge needed), and
 many more useful libraries.

Use the pyATS Library when you want to automate your day-to-day DevOps activities, perform
stateful validation of your network devices, or build a safety net of scalable, data-driven, and
reusable test cases around your network requirements. You can:

 Profile the current state of your network and take a snapshot for later comparison.
 Set up automated monitoring of the operational state of your devices.
 Automate configuration and upgrade tasks.
 Introduce changes – such as new products or releases – and see the effects on your
network.
 Use the power of our Pythonic library to efficiently write your own, reusable, OS-
agnostic scripts.

Getting Started with pyATS

Quick Installation

Run a Testscript
Create a Testbed
Because we are in the business of network test automation, pyATS is designed around the
concept of testbeds: where you describe your devices under testing in YAML format.

This testbed YAML file provides many sections for you to describe your physical devices, and
how they link together to form the topology.

Once a testbed YAML file is written, you can load it, query your topology, connect & issue
commands to your devices using the APIs.

This is the best way to validate whether your topology file is well formed, and your devices
connectable.

Create a Testbed

The devices block contains a description of each network device and must include the
following keys. (Platform is recommended but not required.)

Key Description
hostname This must be the configured hostname of the device.
alias The pyATS Library uses the alias to identify the device during script
execution. This makes the script reusable on another topology, when a
device is assigned the same alias, such as uut (unit under test).
os Device operating system
platform (Recommended) The name of the hardware or chassis, or the name of the
virtual hardware. The system uses the os/platform values to determine the
connection plugin and services.
credentials The username, password, and any other credentials required to log in to
the device.

For details about how passwords are stored.

type Device type
ip IP address
protocol Any one of the supported protocols
(currently Telnet, SSH, REST, RESTCONF, NETCONF, and YANG)
 port Connection port

The following example shows a YAML file with two devices defined:

Here are a few important details to keep in mind:

1. The device name must match the hostname of the device, otherwise, the connection
will hang.
2. At least one device needs to have the alias ‘uut’ in the testbed YAML file.

Create your own Testbed file related to your devices

Create a Testbed from an Excel sheet

You can define all your device data in a CSV (.csv) or Excel (.xls, .xlsx) file. The pyATS create
testbed command automatically converts the input and creates an equivalent YAML file.

Follow these guidelines to create a valid YAML file:

 Separate the IP and port with either a space or a colon (:).

 The password column is the default password used to log in to the device.
 If you leave the password blank, the system prompts you for the password when you
connect to the device.
 To enter privileged EXEC mode with the enable command, add a column with the
header enable_password. The value can be the same as or different from the default
password.
 Any additional columns that you define, such as platform, alias, or type, are added to
the YAML file as key-value pairs.
 The columns can be in any order, if you include the required columns.
 When creating CSV file, separate fields by comma (,). If you need text qualifier, use
double quotes (“).

You can also create a testbed YAML from an excel sheet with pyATS create testbed command,
here’s an example:

The following example shows an Excel file with the required columns. Here is an excel sheet
containing device data:
we can turn it into a testbed YAML file by running the following command:

This will give us the below YAML:

Create a Testbed from a Dictionary

Genie testbed also support creating a testbed directly from a dictionary without a YAML file.
This feature is convenient for quick testing without complicated configurations, and it allows
seamless transition from JSON if the device data obtained from a remote server.

The below code shows the minimum data required for a successful creation, please consult
pyATS create testbed for the minimum key requirement.
Handling Connection to Devices
This topic describes how to connect to network devices using pyATS. It also gives you a quick
example to try using mocked devices.

Because the pyATS Library is based on Python, an object-oriented programming language, it

uses objects to represent your testbed topology.

1. Set up a testbed YAML file that contains your device details.

2. Use the pyATS Library to create the testbed and device objects.
3. Tell the pyATS Library which device to connect to.
4. Connect and run commands.

Supported Platforms

At the moment unicon.plugins supports the following network device types, described as their
OS (network operation system), platform and model (specific model support).

These values help Unicon load the most accurate connection plugin for the given network
device and corresponds to the pyATS testbed YAML counterparts.

For example, if os=iosxe and platform=abc, since abc is not found in the iosxe table, it will fall
back to use the generic iosxe plugin. If os=iosxe and platform=cat3k, it will use the specific
plugin iosxe/cat3k.

os platform mode Comments

l
apic
aireos
asa
asa asav
asa fp2k
cheetah Ap
cimc
comware
confd
confd Esc
confd nfvis
dnos6
dnos10
fxos Tested with FP2K.
fxos fp4k
fxos fp9k
fxos Ftd Deprecated, please use one of the other fxos plugins.
gaia Check Point Gaia OS
hvrp
ios Ap
ios Iol
ios iosv
ios pagent See example below.
iosxe
iosxe cat3k
iosxe cat3k ewlc
iosxe cat8k
iosxe cat9k
iosxe c9800
iosxe c9800 ewc_ap
iosxe csr1000v
iosxe csr1000v vewlc
iosxe iec3400
iosxe sdwan
iosxr
iosxr asr9k
iosxr iosxrv
iosxr iosxrv9k
iosxr moonshine
iosxr ncs5k
iosxr spitfire
ironware
ise
linux Generic Linux server with bash prompts
nd Nexus Dashboard (ND) Linux server. identical to os: linux
nxos
nxos Mds
nxos n5k
nxos n7k
nxos n9k
nxos nxosv
nxos Aci
nso
 sdwan viptela Identical to os=viptela.
sros
staros
vos
junos
eos
sros
viptela Identical to os=sdwan, platform=viptela.
windows

Handling Connection to Devices

This topic describes how to connect to network devices using pyATS.

1. Load the pyATS Library testbed API so that you can create the testbed and device
objects:

2. Create a testbed object tb based on your testbed YAML file. Specify the absolute or
relative path, in this case, mock/mock.yaml:

Result: The system creates a variable tb that points to the testbed object. This
command also creates tb.devices, which contains the YAML device information in the
form of key-value pairs.

3. Create an object dev for the device that you want to connect to:

Result: The pyATS Library finds the device named nx-osv-1 in tb.devices and stores the
information in the dev object.

4. Connect using the values stored in the device object:

Result: The system connects to the device and displays the connection details. Once
you’re connected, you can run show commands and parse the output.

Depending on your device terminal settings, when you connect to a device using a CLI and
execute a command, you would sometimes see “press any key to continue”. For humans, this
breakpoint gives a possibility to analyze output. However, from automation point of view it
would break parsers, as they change output data.

To avoid those, Unicon (#pyATS connection implementation) issues the following commands
on connection established:

 no logging console
 terminal width 511
 etc (depending on platform, which can be specified in testbed file)

All these commands affect the terminal behavior, not your device’s functionality.

Configuration manager with pyATS

Using a python object, set some attributes and the configuration is generated for you. With
Genie you configure a device by only thinking of the network by following a common structure
which works across multiple platforms. Each object follows a structure which is based on the
feature.

Here are some use cases where these objects are useful:

 Must configure multiple devices of different operating system.

 Maintain one script which works across multiple operating system.
 Drive configuration with Python object

Configuration manager with pyATS

This topic describes how to use the Python interpreter or a Python script to use the Conf
module functionality. Because you use it primarily for automated test scripts, we have not
provided a command line option.

The process to configure devices is simple:

 Define the device (object) attributes.

 Tell the pyATS Library to apply the configuration.

This example shows you how to configure a single feature on a single device. You can use the
device hostname or the device alias (defined in the testbed YAML file). In the following
example, uut is the alias “unit under test” for the host nx-osv-1.

1. Get the pyATS Library Interface functionality, to configure an interface on the uut
device:

2. Create an NXOS interface:

 3. Configure the interface that you just created:

4. Verify that the system generated the configuration:

The argument (apply=False) shows you what will be applied on the device if you go
ahead with the build. The system displays the following configuration information:

5. To build the configuration and apply it to the device:

Think of what you want to apply on your device, not how to do it on this platform. Genie conf
provides a structure which is feature based and not platform based. This mean, it stays the
same for all platform and figure out how to apply on the device.

Doing the same for IOSXE, will generate iosxe configuration.

Parser
pyATS parser is a parser library. It takes device output and convert it into a structured datatype
(Json/dictionary)

Once you have this structured data you can do so much with it:

 Look for specific key/value

 Verify if state is as expected
 Compare between two dictionaries and verify if anything has changed

Once you have the data, the possibilities are limitless.

Parsers are easy to call.

Use device.parse('all') to run all the available parsers on the device, and the output will be
returned in a dictionary format: {'show command' : parsed_output}. If an exception occurred
during the execution a particular parser, then parsed_output will become the exception
object.

Following example is a Python scripts to compere to different parsers states.

Instead of learning a few cli at the time, you can learn the whole feature and have it into 1
structured output (JSON/Dictionary). This structure is agnostic between all OS (Identical
between all the OS).
Profile the Network
This topic describes how to use the learn function of the pyATS Library Ops module for stateful
network validation of device features, such as protocols, interfaces, line cards, and other
hardware.

The term feature typically refers to a network protocol, represented by the pyATS Library as a
Python object, with attributes that represent the feature (protocol) configuration on a device.
Many networks use a combination of different features, such as BGP, OSPF, and Interface.

Profile the Network

The pyATS Library Ops module provides a representation of the current operational state of a
device, per feature (protocol). It “learns” the operational state by executing a series of
commands and parsing the output into a common data structure across different operating
systems.

The output is stored with the same key-value pair structure across devices. The stored output
makes it possible for you to take a snapshot of the network state at different points in time,
and then to compare network states.

Why use learn instead of a parser? The parsed output for different devices can result in
different data structures. The learn function, by contrast, results in a consistent set of keys,
which means that you can write one script that works on different devices.

Learn Examples

This topic describes how you can tell the system to learn one or more features.

To learn one feature on a single device, you can use the device hostname or the device alias
(defined in the testbed YAML file). In the following example, uut is the alias “unit under test”
for the host nx-osv-1.
Use pyATS shell to load the testbed API and create your testbed and device objects. Then,
connect to the device and tell the system to learn the feature. In this example, the system
stores the output as a Python dictionary in the variable output:

Use pyATS shell to load the testbed API and create your testbed and device objects. Then, tell
the system to connect to each device and to learn the specified features. In this example, the
system stores the output as a Python dictionary in the variable learnt and displays the output:

This example uses a Python for loop to execute each statement on all devices in the testbed.
The system stores the feature information in Python dictionaries, each identified by the device
name.

Use the learn all functionality to learn all the supported features on a device. The system
returns the results in a format with key-value pairs and notifies you of any exceptions for
features it did not learn.

Use pyATS shell to load the testbed API and create your testbed and device objects. Then,
connect to the device and tell the system to learn all of the features. In this example, the
system stores the output as a Python dictionary in the variable output:

Ops object learns everything to know about the operation state of a device feature such as
OSPF, interface, etc.
This sends multiple commands on the device and creates a structure for all the information.
Compare Network State
This topic describes how to use the Diff functionality to monitor the network and to verify that
network is performing as expected.

The pyATS Library Diff functionality enables you to compare network snapshots so that it is
easy to see any changes. For any two Python dictionaries or objects with a consistent data
structure (the set of key-value pairs), the system compares the keys and identifies differences.
You can exclude keys that you don’t want to compare.

Using the pyATS Library, you can collect and compare network state information on demand,
or as scheduled jobs that run your automation scripts. The process is simple:

 Take a snapshot of your network.

 Make or observe a change in your network.
 Take another snapshot.
  Use Diff to see the changes.

For example, you can set up a scheduled job to take a new snapshot every day, to make sure
your network is running as it should and to see any configuration changes. The result is a Linux-
style list of additions, deletions, and changes.

Examples of stateful validation

In this example, you’ll see how to take snapshots with the learn function, save the snapshots
to different directories, and then compare the data.

1. With your devices already configured and running, take a snapshot of the BGP feature
and save it to the directory (or variable) output1. You can use a Python interpreter or
the Library command line.

Use pyATS shell to load the testbed API and create your testbed and device objects. Then, tell
the system to connect to each device and to learn the specified feature:

This example uses a Python for loop to execute each statement on all devices in the testbed.

Result: The system stores the feature information in a Python dictionary, which includes the
data for all devices.

2. Change the configuration of a feature on a device or shut down/bring up an interface.

3. Repeat step 1 to take another snapshot, but specify a different output directory or
variable, such as output2.
4. Compare the two snapshots:

pyATS Command Line

In this section we will take a look at how pyATS CLI can help expedite network automation.
pyATS CLI is a powerful Linux-based command-line utility offering Genie Python functionality
directly from a Linux terminal. It requires no previous knowledge of Python or network
programming, making it a great way to start getting acquainted with Genie.

Genie is the top-level command-line entry point for Genie. All other functions are loaded as
subcommands of this command. Ensure Genie is installed prior to using genie CLI.

For a complete list of built-in functions available within genie CLI, execute the following in your
Linux terminal:

Genie Parse

Genie parse is used to execute and parse show-commands on your device directly from a Linux
terminal.

The following is an example of using genie parse to execute and parse command “show
version” on device nx-osv-1 of the testbed.
Genie Learn

genie learn is a powerful tool that can be extensively used to accomplish stateful network
validation across multiple devices, with one Linux command.

The following is an example of how to learn one feature (BGP) on the device uut:

Genie Diff
genie diff is used to compare Python data structures generated by genie parse and genie learn.
It is extremely useful in performing stateful network validation before and after any event
occurs in a network that could cause the operational state to change.

The following is an example of using genie diff with the default exclude list during comparison:

Introduce pyATS SDK

Writing a future-proof automation script is a complex topic. It seems simple at first, but then
as the devices update, things change, there are new requirement and it all become spaghetti;
and the whole process restarts again.

With pyATS, we’ve taken this complex topic and made it simple. We’ve taken a divide and
conquer approach to this problem. Multiple packages and libraries were created with Genie
harness being the last piece to create a resilient and future-proof automation solution. In
short, we’ve made lasagna with many modules stacked on top of each other.

Genie harness provides comprehensive common tools so that users can focus on writing tests
without having to reinvent the wheel!

Once the tests are written, the harness is then driven by YAML datafiles. The files control the
whole flow of the execution:

 connect to all the devices

 configure the devices
 Verify the configuration was applied correctly
 snapshot of the configuration and compared at the end of the run
 snapshot of the operation state of your devices to be compared at the end of the run
 pool of available tests (triggers and verifications, over 500) which works across
multiple Operating systems
 connections pool - Learn and send commands to the device in parallel
 traffic Generator
pyATS SDK is fully modular; everything can be customized; modified for your automation need.

 New subsections can be added and removed

 Decide which triggers and verifications to execute and in which order
 Create your own triggers and verifications
 Add local verifications to any trigger
 Add processor anywhere
 Static information for each trigger

Introduce pyATS Blitz

The Blitz is a YAML-driven template that makes it easy for you to run a test case without
having to know any knowledge of programming. This tool is called Blitz because it is lightning
fast. pyATS Blitz does the following actions:

 Configure a device.
 Parse the device output to verify if the device state is as expected.
 Unconfig or modify the initial configuration.
 Learn a feature and verify the result of the action
 Calling different APIs and use their outputs on other actions and other devices
 Yang integration
 It is fully customizable and new actions can be added

To use the Blitz, add the YAML content to a trigger datafile, or create a new YAML file if you
don’t have one. Using a job file execute tests in the given order. Each trigger datafile can
contains of multiple testcases.
Introduce Network Visualization with Expresso
XPRESSO is the standard pyATS UI dashboard that manages test suites, test resources and test
results, providing network insights through Cisco pyATS.

XPRESSO is a comprehensive and complementary network test automation web-based

"dashboard" that is designed to maximize, expedite, and simplify test execution in a fashion
that aligns well with the stringent timeline and resource requirements.

XPRESSO provides a unique set of at-a-glance, simple-to-use set of features to help to prepare,
schedule, and execute network test automation scripts while providing a comprehensive and
user-friendly way to manage and search test results so users can quickly isolate, interpret, and
analyze test results.

XPRESSO provides an all-in-one web-based dashboard that takes the pyATS experience to the
next level by drastically reducing network testing/certification/validation cycle-time by:

 Providing a flexible and easy-to-use visualization front-end that simplifies your

network testing automation complexity.
 Fully automates testing and images/releases certification.
 Facilitates automated image validation testing to baseline new software releases.
 Easing the management of test resources in the test automation environment.