NETCONF & YANG

AUTOMATION TESTING
USER GUIDE V3
Table of Contents
1. Overview...................................................................................................................4

2. Reasons for NSO Driven Automation Testing............................4

3. Installation of NSO & DrNED Examiner............................................5

3.1. Overview..............................................................................................................5

3.2. NSO Installation................................................................................................5

3.3. Run NSO...............................................................................................................6

3.4. Install DrNED Examiner Package............................................................... 7

3.5. Run NSO CLI and Load the DrNED Examiner Package.................... 7

3.6. Configure the DrNED Examiner Package..............................................8

4. Managing Network Equipment with

NETCONF/YANG using NSO.....................................................................9
4.1. Overview..............................................................................................................9

4.2. Configure Your Device in NSO..................................................................9

4.2.1. Configure Authentication Group and Device in NSO................. 9

4.2.2. Configure the Device in NSO........................................................... 9

5. Building and Installing a NETCONF NED

Using the NETCONF NED Builder...................................................... 10
5.1. Make the YANG Data Models Available to NSO..............................10

5.1.1. Copying Manually to the Cache Directory.....................................11

5.1.2. Using NETCONF...................................................................................11

5.1.3. YANG Modules to Include in the NETCONF NED......................13

2
 5.2. Build the NED from the YANG Data Models....................................13

5.3. Troubleshoot NED Building Issues......................................................14

5.4. Prepare the NED for Loading................................................................15

5.5. Load the Compiled YANG Files............................................................15

5.6. Update the ned-id for the test1a Device............................................16

5.7. How to Remove a NED from NSO........................................................16

6. Testing the Device NETCONF/YANG Interface.........................17

6.1. Sync Device Configuration into NSO......................................................17

6.2. Configure the Device Using NSO............................................................17

6.3. Test Transactionality Using DrNED Examiner...................................17

6.4. Minimum Success Criteria for

Demonstrating NSO Interoperability................................................... 20

7. Troubleshooting................................................................................................. 21
7.1. NETCONF Issues............................................................................................. 21

7.1.1. Enable NETCONF Tracing..................................................................21

7.2 NSO Issues....................................................................................................21

7.3. DrNED Examiner Issues...........................................................................21

8. For More Information................................................................................... 22

3
NETCONF & YANG Automation Testing User Guide v3

1. Overview

This document describes how to perform NETCONF and YANG automation testing driven
by Cisco Network Services Orchestrator (NSO) and is divided into six sections: “Reasons
for NSO Driven Automation Testing,” “Installation of NSO & DrNED Examiner,” “Managing
Network Equipment with NETCONF/YANG using NSO,” “Building and Installing a NED
using the NETCONF NED Builde,” “Testing the Device NETCONF/YANG Interface,” and
“Troubleshooting,”

“Reasons for NSO Driven Automation Testing” describes the reasons for performing NSO
driven NETCONF and YANG automation testing.

“Installation of NSO & DrNED Examiner” describes how to install and run the NSO and
DrNED Examiner software.

“Managing Network Equipment with NETCONF/YANG using NSO” describes how to

configure NSO to manage the connected devices.

“Building and Installing a NED using the NETCONF NED Builder” describes how to build
a NETCONF NED for managing the connected devices. It is important to note that the
NETCONF NED Builder is now a built-in feature of NSO starting with NSO 5.2 and it
replaces the need for installing the Pioneer package as was previously required and
discussed in an earlier version of this guide.

“Testing the device NETCONF/YANG Interface” describes how to use NSO to verify
that the device and NSO communicate properly using NETCONF and YANG and that
the device adheres to NETCONF rules and best practices. This section ends with a
discussion of the minimum success criteria for demonstrating NSO Interoperability.

“Troubleshooting” contains various tips and tricks to aid with common error scenarios
with the tools.

2. Reasons for NSO Driven Automation Testing

To ensure that your network element’s NETCONF implementation is standards compliant
according to the RFCs, NSO, along with a plug-in package called DrNED Examiner, can
be used to perform testing and validation during your development phase. In a previous
Application Note called “Test your ConfD Device with NSO”, we have written about the
full set of service automation best practices that are important to service providers.
In this section, we will briefly discuss some of the possible issues that can be caught
during NETCONF and YANG automation testing using NSO as a NETCONF test tool.

Potential issues that can be found during NETCONF and YANG automation testing include
simple things such as data type mismatches between a leaf node’s definition in the YANG
data model and the NETCONF implementation. For example, enumerated data types defined
in YANG may return incorrectly as integer values through NETCONF. Leafref data types may
also return invalid or non-existing paths to your data model.

4
NETCONF & YANG Automation Testing User Guide v3

Other than data type mismatches, there may be other issues that are slightly more difficult to
uncover without the use of the proper NETCONF testing tool. They include side effects that
happen with NETCONF’s <edit-config> operation. Some people call this auto-configs or self-
modifying configuration. These are additional changes to the device’s configuration beyond
what is specified in the <edit-config> operation. This issue will cause the configuration on
your device to get out-of-sync with the NETCONF manager (client) unless special code is
put in place after the <edit-config> operation to perform the synchronization. This causes
unnecessary complexity and performance overhead for configuration orchestration software
such as NSO.

Another issue that has to do with NETCONF transactions should also be tested and
addressed on the device during the development cycle of your device’s software. This issue
has to do with the order dependencies of the configuration changes. It is very common
for legacy CLI-based devices to require configuration changes to be entered in a particular
order. There can’t be order dependencies with NETCONF transactions. Relying on this order
dependency will break the transactionality property of your NETCONF device.
This will take away the simplicity that configuration orchestration software relies on to be
able to consistently and reliably go back and forth between different configuration states by
using simple transactional <edit-config> operations without having to know the order of how
configuration changes should happen on the device.

How your device deals with default values for the NETCONF <get-config>/<get> operations
are also important. If your device isn’t implemented according to the “With-defaults
Capability for NETCONF” as described in RFC 6243, this issue can also be caught during this
interoperability testing.

These are a few examples of the types of issues which could arise without performing the
proper NETCONF and YANG automation testing to catch them.

You can define test cases with the tools as described in this document according to the
most typical use cases of your device in order to detect data type mismatches, validate
its transactionality behavior, that it is free of side effects from a NETCONF <edit-config>
operation, as well as other test cases of your own design.

3. Installation of NSO & DrNED Examiner

3.1 Overview
This document assumes the use of NSO 5.2 or later release. NSO software for non-production
use can be downloaded via Cisco DevNet at: https://developer.cisco.com/site/nso/.

3.2 NSO Installation

NSO is delivered as a self-extracting archive which is OS/CPU specific. For NSO’s run
time dependencies, please, refer to the Prerequisites section in Chapter 1 of the NSO
Installation Guide that can be found under the “doc” folder in your installation directory
after NSO has been installed on your system. To install NSO, just run the NSO installer
with a single argument, that is the directory where you would like NSO to be installed.
The installation directory you specify must either not yet exist or be empty. No files will
be added or modified outside the directory you specify.

5
NETCONF & YANG Automation Testing User Guide v3

$ sh nso-<VERSION>.<OS>.<ARCH>.installer.bin /path/to/install_dir

On a typical system, the installation process looks like this:

$ sh download/nso-x.x.linux.x86_64.installer.bin ~/nso-x.x
INFO Using temporary directory /var/folders/cs/7g8xdq0j1_b6k6p3pty3h3_
w0000gp/T//ncs_installer.15936 to stage NCS installation bundle
INFO Unpacked ncs-x.x in /home/tempuser/nso-x.x
INFO Found and unpacked corresponding DOCUMENTATION_PACKAGE
INFO Found and unpacked corresponding EXAMPLE_PACKAGE
INFO Generating default SSH hostkey (this may take some time)
INFO SSH hostkey generated
INFO Environment set-up generated in /home/tempuser/nso-x.x/ncsrc
INFO NCS installation script finished
INFO Found and unpacked corresponding NETSIM_PACKAGE
INFO NCS installation complete

As you can see, in addition to unpacking the installation files, the installer will create a
default SSH hostkey and a file called ‘ncsrc’ with handy environment settings for using
this installation.

Refer to the NSO Installation Guide that can be found under the “doc” folder of your NSO
installation for more information on how to install and start NSO on your system.

3.3 Run NSO

Source the shell script that was created during the installation as follows:

$ source $HOME/nso-version/ncsrc

Create a runtime directory for NSO to keep its database, state files, logs and other files.
In these instructions we assume that this directory is $HOME/ncs-run.

$ ncs-setup --no-netsim --dest $HOME/ncs-run

You will find the following folders getting created:

$ ls $HOME/ncs-run
README.ncs logs ncs-cdb ncs.conf
packages scripts state

Start the NSO daemon ncs:

$ cd $HOME/ncs-run
ncs-run$ ncs

When you start NSO, make sure that you are situated in the deployment directory since
by default NSO reads its config files from its current working directory.

6
NETCONF & YANG Automation Testing User Guide v3

3.4 Install the DrNED Examiner Package

The DrNED Examiner package is a toolkit that helps with testing both CLI and NETCONF
NEDs. It can perform automated interoperability testing with your device. Its full
capabilities for testing NETCONF NEDs will be described in this document. Refer to the
Readme file in the GitHub site for the latest version requirements on the required packages,

DrNED Examiner is written in Python and depends on the following Python packages:

• pytest
• pexpect
• lxml

On Ubuntu, you can issue the following commands to install them:

ncs-run$ sudo apt-get install python-pytest python-pexpect python-lxml

In addition, GNU awk (gawk) is also required:

ncs-run$ sudo apt-get install gawk

You can get the latest DrNED Examiner package from https://github.com/NSO-developer/
drned-xmnr and place it in the packages subdirectory under your “ncs-run” directory.

ncs-run$ cd packages
packages$ git clone https://github.com/NSO-developer/drned-xmnr.git

Execute the following command to build it:

packages$ cd ..
ncs-run$ make –C packages/drned-xmnr/src clean all

3.5 Run NSO CLI and Load the DrNED Examiner Package
Launch the NSO CLI in Cisco style:

ncs-run$ ncs_cli –C –u admin

7
NETCONF & YANG Automation Testing User Guide v3

The default login and password for NSO are admin and admin.

admin@ncs# packages reload

reload-result {
package drned-xmnr
result true
}
admin@ncs#
System message at 2019-10-07 14:29:41...
Subsystem stopped: ncs-dp-19163-drned-xmnr:drned_xmnr
admin@ncs#
System message at 2019-10-07 14:29:42...
Subsystem started: ncs-dp-19222-drned-xmnr:drned_xmnr
admin@ncs# show packages
packages package drned-xmnr
package-version 0.1.0
description “DrNED Examiner is a tool for examining/diagnosing your
NEDs”
ncs-min-version [ 4.1 ]
python-package vm-name drned-xmnr
directory ./state/packages-in-use/1/drned-xmnr
component drned_xmnr
application python-class-name drned_xmnr.action.Xmnr
application start-phase phase2
oper-status up

3.6 Configure the DrNED Examiner Package

DDrNED Examiner has two configuration parameters: /drned-xmnr/xmnr-directory and /
drned-xmnr/drned-directory. The former sets the location where the tool stores its data
and defaults to /tmp/xmnr; the latter defaults to the built-in DrNED package that comes
with this DrNED Examiner package.

We will just set the first parameter to a directory called xmnr in the ncs-run folder and
leave the second one to its default value. It is also a good idea to configure logging in
the Examiner CLI to not be too verbose and also make sure that logging is enabled for
the python-vm as these logs contain a lot of useful troubleshooting information:

ncs-run$ mkdir xmnr

ncs-run$ ncs_cli -C -u admin
admin@ncs# config
admin@ncs(config)# drned-xmnr log-detail cli overview
admin@ncs(config)# drned-xmnr xmnr-directory ./xmnr
admin@ncs(config)# python-vm logging level level-debug
admin@ncs(config)# commit
Commit complete.

8
NETCONF & YANG Automation Testing User Guide v3

4. Managing Network Equipment with NETCONF/YANG using NSO

4.1 Overview
In this section, we discuss how to establish communication between NSO and the device
being tested.

First, the user must configure the information used for authentication when NSO connects
to the device and add the device to the devices configuration of NSO.

After the device has been added to NSO, but before NSO can manage a device using
that device’s YANG data models, the data models must first be uploaded to NSO and a
Network Element Driver (NED) must be built that NSO can use when it communicates with
the device via NETCONF.

Finally, we will describe how to use the DrNED Examiner package to test whether the
device interoperates properly with NSO.

4.2 Configure Your Device in NSO

This section will go through the process of connecting your device to NSO in order for
NSO to be able to communicate with it and drive further testing.

The steps involved include:

1. Configure the NSO authentication group used when connecting to the device
2. Configure the device in NSO

4.2.1 Configure Authentication Group and Device in NSO

As communication with the devices being managed by NSO require authentication, a custom
authentication group is created with the username and password required by the devices.
The example below creates a group named “nsointerop”, make sure to replace <username>
and <password> with the required username and password for the devices being managed.
Feel free to use your own name for the authentication group.

admin@nso# config
admin@ncs(config)# devices authgroups group nsointerop default-map remote-name
<username> remote-password <password>
admin@ncs(config-group-nsointerop)# commit
Commit complete.
admin@ncs(config-group-nsointerop)# exit
admin@ncs(config)#

4.2.2 Configure the Device in NSO

Configuring the device in NSO can be done following the example below. This example
assumes that the IP address for the device is 10.80.26.213 using the “nsointerop”
authentication group created in the previous step:

admin@nso# config
admin@ncs(config)# devices device test1a device-type netconf ned-id netconf
admin@ncs(config-device-test1a)# address 10.80.26.213 port 830 authgroup
nsointerop
admin@ncs(config-device-test1a)# state admin-state unlocked
admin@ncs(config-device-test1a)# commit
Commit complete.
admin@ncs(config-device-test1a)# ssh fetch-host-keys

9
NETCONF & YANG Automation Testing User Guide v3

Please, note that the ned-id of netconf here is only temporary as the NETCONF NED for
the test1a device hasn’t yet been built. Once we have built the NETCONF NED for test1a,
we need to go back and change the ned-id to its true identity.

5. Building and Installing a NETCONF NED

using the NETCONF NED Builder

The NETCONF NED Builder is a development tool that is built into NSO for building NETCONF
NEDs. The process of creating and installing a NED consists of the following steps:

1. Make the YANG data models available to NSO

2. Build the NED (Network Equipment Driver) from the YANG data models

3. Install the NED

4. Make the NED available to NSO

5.1 Make the YANG Data Models Available to NSO

You will first need to create a NETCONF NED Builder project for the device “test1a”.

SSH NSO Window:

ncs-run$ ncs_cli -C -u admin

admin@ncs# devtools true
admin@ncs# config
Entering configuration mode terminal
admin@ncs(config)# netconf-ned-builder project tail-dhcpd 1.0 device test1a
local-user admin vendor Tail-f
admin@ncs(config-project-tail-dhcpd/1.0)# commit
admin@ncs(config-project-tail-dhcpd/1.0)# exit
admin@ncs(config)# exit
admin@ncs# show netconf-ned-builder project tail-dhcpd
netconf-ned-builder project tail-dhcpd 1.0
download-cache-path ./state/netconf-ned-builder/cache/tail-dhcpd-nc-1.0
ned-directory-path ./state/netconf-ned-builder/tail-dhcpd-nc-1.0
admin@ncs#

The NETCONF NED Builder is a developer tool and needs to be enabled first through the
devtools command to set it to true. The NETCONF NED Builder feature is not expected
to be used by the end users of NSO.

The project argument for the netconf-ned-builder command requires both the project
name and a version number for the NED being built. A reasonable version number to
pick is the version number of the device OS. The name of the device for the NED being
built should match the one that was configured in the previous section.

10
NETCONF & YANG Automation Testing User Guide v3

5.1.1 Copying Manually to the Cache Directory

If the device supports the NETCONF <get-schema> rpc call, you can skip to the next
section of “Using NETCONF”. Otherwise, you can copy all available YANG models for the
device to the download-cache-path as shown in the previous section.

SSH NSO Window:

ncs-run $ ls ./state/netconf-ned-builder/cache/tailf-dhcp-nc-1.0/
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]

5.1.2 Using NETCONF

If the device supports <get-schema> requests, the device can be contacted directly for
downloading the YANG data models. For a device using ConfD, two things must be true in
order for NSO to upload the schema directly from the device. First ietf-netconf-monitoring.
fxs must be in the loadpath of your ConfD installation. The second item is that the YANG
source files for the data models that you wish to upload to NSO must also be in the
loadpath (Note: this is only needed prior to ConfD 6.4). These YANG source files are what
ConfD will return when the NETCONF <get-schema> RPC is issued to the device from NSO.

11
NETCONF & YANG Automation Testing User Guide v3

SSH NSO Window:

ncs-run$ ncs_cli -C -u admin

admin@ncs# devtools true
admin@ncs# netconf-ned-builder project tailf-dhcp 1.0 fetch-module-list
admin@ncs# show netconf-ned-builder project tailf-dhcp 1.0 module
module dhcpd “”
namespace http://tail-f.com/ns/example/dhcpd
location [ NETCONF ]
module ietf-inet-types 2013-07-15
namespace urn:ietf:params:xml:ns:yang:ietf-inet-types
location [ NETCONF ]
…
module tailf-common 2019-05-16
namespace http://tail-f.com/yang/common
location [ NETCONF ]
submodule tailf-cli-extensions 2019-05-16
location [ NETCONF ]
submodule tailf-meta-extensions 2017-03-08
location [ NETCONF ]
…
module tailf-xsd-types 2017-11-20
namespace http://tail-f.com/yang/xsd-types
location [ NETCONF ]
admin@ncs# netconf-ned-builder project tailf-dhcp 1.0 module dhcpd “” select
admin@ncs# show netconf-ned-builder project tailf-dhcp 1.0 module status
NAME REVISION STATUS
--------------------------------------------------
dhcpd selected,downloaded
ietf-inet-types 2013-07-15 selected,downloaded
tailf-common 2019-05-16 selected,downloaded
tailf-xsd-types 2017-11-20 selected,downloaded

The fetch-module-list command will either look for existing YANG modules in the
download-cache-path folder or issue <get> and <get-schema> to fetch the YANG
modules over NETCONF and put them in the download-cache-path folder. You can then
view the list of downloaded modules and select the ones that you would like to include
in the NETCONF NED. When you select a top-level module that has dependencies on
other modules, the dependent modules are automatically selected. With the example
above, I have chosen only the dhcpd module with an empty revision as the dhcpd
module didn’t contain any revision date information. A typical YANG module should
contain the revision date information. The dependent modules are automatically
selected, such as the ones listed above for the dhcpd module including ietf-inet-types,
tailf-common and tailf-xsd-types. If you would like to select all available modules, you
can use the wild card for both fields. You can also use the deselect command to exclude
modules from the build.

12
NETCONF & YANG Automation Testing User Guide v3

5.1.3 YANG Modules to Include in the NETCONF NED

It is recommended to select only a subset of all available YANG models on the
device that are relevant for the use cases that you are interested in validating for
NSO interoperability. It is important to note that you should be exposing only a single
consistent view of the device datastore through the NETCONF NED. For example, if
your device supports both the native proprietary modules along with standards-based
modules (such as IETF or OpenConfig) in which the same data nodes exist in both, you
should build the NETCONF NED for either just the native modules or just the standards-
based modules. Including both will cause NSO to get out-of-sync with the device after
a <edit-config> request when a common data node from only one of the two views is
specified in the request.

5.2 Build the NED from the YANG Data Models

An NSO Network Element Driver (NED) is a package containing the device’s YANG data
models. The NED must first be built, then installed in NSO, and finally the package must
be loaded in order for NSO to communicate with the device via NETCONF using that
device’s YANG data models.

After the files have been downloaded from the device, they must first be built before they
can be used. The following example shows how to build a NED for the device “test1a”:

SSH NSO Window:

admin@ncs(config)# netconf-ned-builder project tail-dhcpd 1.0 build-ned

admin@ncs(config)# exit
admin@ncs# show netconf-ned-builder project tail-dhcpd 1.0 build-status
build-status success

NOTE!
Make sure no errors occur during compilation. If any errors appear in the
compilation output, resolve the issues in your YANG data models and re-upload
them before continuing.

YANG data models that don’t compile due to YANG RFC compliance issues can either be
updated in the cache folder directly or in the device and re-upload again through <get-
schmea> by removing them from the cache folder and repeat the previous process again
to rebuild the NED. They can also be excluded from the build by not selecting them if
they aren’t needed for your use case.

13
NETCONF & YANG Automation Testing User Guide v3

5.3 Troubleshoot NED building issues

To see any warnings after building the NED, they can be found in the build-warning
leaf under the module list entry. It is always a good idea to clean up the build warnings
in your YANG models. A command line example for showing the build warnings is as
follows:

SSH NSO Window:

admin@ncs# show netconf-ned-builder project tail-dhcpd 1.0 module build-warn-

ing

If the build failed in the previous step with errors, the build errors can be found in the
build-error leaf under the module list entry. A command line example for showing the
build errors is as follows:

SSH NSO Window:

admin@ncs# show netconf-ned-builder project tail-dhcpd 1.0 module build-error

To see the full compiler output for debugging purposes, they can be found in the
compiler-output leaf under the project list entry. The compiler-output leaf is hidden by
the hide-group debug and may be accessed in the CLI using the unhide debug command
if the hide-group is configured in ncs.conf at the top level as follows:

<hide-group>
<name>debug</name>
</hide-group>

A reload of NSO is needed for the configuration changes made to ncs.conf to take effect.
You will run “ncs –-reload” in your Linux shell.

A command line example for showing the full compiler output is as follows:

SSH NSO Window:

admin@ncs# unhide debug
admin@ncs# show netconf-ned-builder project tailf-dhcp 1.0 compiler-output

The unhide debug command only needs to be invoked once per CLI session.

As an alternative to debugging the NED building issues inside of an NSO CLI session, the
NED building environment can be exported. Following shows the step for exporting the
NED and building it in your Linux shell.

SSH NSO Window:

admin@ncs(config)# netconf-ned-builder project tail-dhcpd 1.0 make-develop-
ment-ned in-directory /tmp
admin@ncs(config)# exit
$ cd /tmp/tailf-dhcp-nc-1.0/src
$ make clean all

14
NETCONF & YANG Automation Testing User Guide v3

The YANG models for your device can be found in the yang directory under /tmp/tailf-
dhcp-nc-1.0/src. Once all of the errors in the YANG models have been fixed, you can then
go back to building the NETCONF NED using the NETCONF NED Builder and continue
with the next step.

5.4 Prepare the NED for Loading

After the YANG data model files have been compiled into the NED, the NED package
must be exported before the YANG data models can be loaded into the NSO runtime.
The following example shows how to export the NED that was just built for the device “test1a”:

SSH NSO Window:

ncs-run$ ncs_cli -C -u admin

admin@ncs# devtools true
admin@ncs# netconf-ned-builder project tailf-dhcp 1.0 export-ned to-directory
/tmp
tar-file /tmp/ncs-5.2.0.3-tailf-dhcp-nc-1.0.tar.gz
admin@ncs# exit
ncs-run$ mv /tmp/ncs-5.2.0.3-tailf-dhcp-nc-1.0.tar.gz packages

5.5 Load the Compiled YANG Files

Once the NED package has been copied to the packages folder of the NSO project, the
NED package is now ready to be loaded into the NSO.

SSH NSO Window:

admin@ncs# packages reload

>>> System upgrade is starting.

>>> Sessions in configure mode must exit to operational mode.
>>> No configuration changes can be performed until upgrade has completed.
>>> System upgrade has completed successfully.
reload-result {
package drned-xmnr
result true
}
reload-result {
package tail-dhcpd-nc-1.0
result true
}
admin@ncs#
System message at 2019-10-08 19:12:04...
Subsystem stopped: ncs-dp-19222-drned-xmnr:drned_xmnr
admin@ncs#
System message at 2019-10-08 19:12:04...
Subsystem started: ncs-dp-99309-drned-xmnr:drned_xmnr
admin@ncs#

NOTE!
After the package has been loaded, NSO will be able to manage devices matching
your YANG data models.

15
NETCONF & YANG Automation Testing User Guide v3

5.6 Update the ned-id for the test1a device

Now that the NETCONF NED has been built for the test1a device, the ned-id for test1a
needs to be updated before the NED can be used to manage the device.

SSH NSO Window:

admin@ncs# config
admin@ncs(config)# devices device test1a device-type netconf ned-id
Possible completions:
lsa-netconf netconf snmp tail-dhcpd-nc-1.0
admin@ncs(config)# devices device test1a device-type netconf ned-id tail-dh-
cpd-nc-1.0
admin@ncs(config-device-test1a)# commit
Commit complete.
admin@ncs(config-device-test1a)#

5.7 How to Remove a NED from NSO

Installed NED packages can be removed from NSO by deleting them from the packages folder
of the NSO project followed by deleting the device and the NETCONF NED project through
ncs_cli. The following example shows how to uninstall a NED built for the device “test1a”:

SSH NSO Window:

ncs-run$ rm packages/ncs-5.2.0.3-tail-dhcpd-nc-1.0.tar.gz
ncs-run$ ncs_cli -C -u admin
admin@ncs# devtools true
admin@ncs# config
Entering configuration mode terminal
admin@ncs(config)# no devices device test1a
admin@ncs(config)# no netconf-ned-builder project tailf-dhcp 1.0
admin@ncs(config)# commit
Commit complete.
admin@ncs(config-device-test1a)# exit
admin@ncs(config)# exit
admin@ncs# packages reload
Error: The following modules will be deleted by upgrade:
tail-dhcpd-nc-1.0: dhcpd
tail-dhcpd-nc-1.0: tail-dhcpd-nc
tail-dhcpd-nc-1.0: tail-dhcpd-nc-1.0
If this is intended, proceed with ‘force’ parameter.
admin@ncs# packages reload force

>>> System upgrade is starting.

>>> Sessions in configure mode must exit to operational mode.
>>> No configuration changes can be performed until upgrade has completed.
>>> System upgrade has completed successfully.
reload-result {
package drned-xmnr
result true
}
admin@ncs#
System message at 2019-10-08 19:30:22...
Subsystem stopped: ncs-dp-983-drned-xmnr:drned_xmnr
admin@ncs#
System message at 2019-10-08 19:30:23...
Subsystem started: ncs-dp-1847-drned-xmnr:drned_xmnr

16
NETCONF & YANG Automation Testing User Guide v3

6. Testing the Device NETCONF/YANG Interface

6.1 Sync Device Configuration into NSO

To synchronize configuration data from the device into NSO, use the following
NSO CLI command:

admin@ncs# devices device test1a sync-from

result true

6.2 Configure the device using NSO

To configure the device using the NETCONF NED and see the resulting device
configuration, use the following NSO CLI commands:

admin@ncs(config)# devices device test1a config dhcpd:dhcp maxLeaseTime 6000s

admin@ncs(config-config)# commit
Commit complete.
admin@ncs(config-config)# exit
admin@ncs(config-device-test1a)# exit
admin@ncs(config)# show full-configuration devices device test1a config
devices device test1a
config
dhcpd:dhcp maxLeaseTime 6000s
!
!
admin@ncs(config)#

6.3 Test Transactionality using DrNED Examiner

DrNED Examiner can be used to test that a device supports the proper transactional
behavior. Among other things, transactional behavior means that a device can accept
any valid configuration regardless of the state it is in currently. For all transactions, it must
also be true that the device configuration becomes exactly the configuration as modified
by the transaction. In other words, it’s not acceptable that a device modifies any other
parts of the configuration other than the ones touched by the transaction. Note that
a different device from the previous section is being used to demonstrate the DrNED
Examiner tool in this section.

17
NETCONF & YANG Automation Testing User Guide v3

DrNED Examiner can test the conformance with these rules automatically but requires a
smart selection of input configurations in order to find something useful. This is how it is done.

The last setup step is to initialize DrNED Examiner:

admin@ncs# config
admin@ncs(config)# devices device test1a
admin@ncs(config-device-test1a)# drned-xmnr setup setup-xmnr overwrite true

Configure the device to an interesting configuration state and make sure that NSO is
in sync with the device. Changes may be entered directly on the device, followed by a
“sync-from” in NSO. Alternatively, changes may be entered in NSO using “ncs_cli” and
committed to the device. Once a configuration is running fine on the device and NSO is
in sync, issue the operational command:

drned-xmnr state record-state state-name <state-name>

The state name needs to be a valid file name but can otherwise be chosen freely.
The name will be used later to describe which configuration state transitions that have issues.

Keep doing this with interesting configurations for a while, so that you have at least 4
states recorded. You can list the names of the states you have recorded, or delete ones
you don’t want to keep using:

drned-xmnr state list-states

drned-xmnr state delete-state state-name <state-name>

Then, when enough states have been collected, DrNED Examiner can start testing that
all transitions work automatically. By default, all transitions are tried. That should give a
safe result. Since testing all-to-all configuration transitions grows exponentially with the
number of states, it’s also possible to limit the number of transitions to try out. The test
will then not be conclusive but may be more reasonable to run. You can run the test in
any of these ways:

drned-xmnr transitions explore-transitions

drned-xmnr transitions explore-transitions stop-after { seconds 30 }
drned-xmnr transitions explore-transitions stop-after { minutes 5 }
drned-xmnr transitions explore-transitions stop-after { hours 12 }
drned-xmnr transitions explore-transitions stop-after { days 2 }
drned-xmnr transitions explore-transitions stop-after { cases 20 }
drned-xmnr transitions explore-transitions stop-after { percent 10 }

18
NETCONF & YANG Automation Testing User Guide v3

The sequence of transitions to try is selected randomly. Two different runs will therefore
not yield the same test pattern.

A test run might look like this:

admin@ncs(config-device-test1a)# drned-xmnr transitions explore-transitions

stop-after { percent 10 }
Found 8 states recorded for device test1a which gives a total of 56 transi-
tions.

Starting from known state noloop8

... failed setting known state

Starting from known state bundle-vrf

Transition 1/56: bundle-vrf ==> bundle-vrf-ipv4
Transition 2/56: bundle-vrf-ipv4 ==> bundle-vrf-ipv46
Transition 3/56: bundle-vrf-ipv46 ==> bundle-vrf-ipv4
Transition 4/56: bundle-vrf-ipv4 ==> bundle-vrf
Transition 5/56: bundle-vrf ==> bundle-vrf-ipv46
Transition 6/56: bundle-vrf-ipv46 ==> bundle-vrf
Requested stop-after time limit reached
success Completed successfully

The comment “...failed setting known state” is an indication that something isn’t right.
But since DrNED Examiner didn’t know exactly what state the device was in before we
started (or after a transaction has failed) it’s hard to give instructions on how to repeat the
issue, so this problem is ignored.

If an issue is found, the outcome might look like this:

admin@ncs(config-device-test1a)# drned-xmnr transitions explore-transitions

stop-after { percent 10 }
Found 8 states recorded for device test1a which gives a total of 56 transi-
tions.

Starting from known state bundle-vrf-ipv4

Transition 1/56: bundle-vrf-ipv4 ==> bundle-vrf-ipv46
Transition 2/56: bundle-vrf-ipv46 ==> bundle-vrf-ipv4
Transition 3/56: bundle-vrf-ipv4 ==> bundle-vrf
Transition 4/56: bundle-vrf ==> bundle-vrf-ipv4
Transition 5/56: bundle-vrf-ipv4 ==> initial
transaction-failed

Starting from known state no-bundle-ipv46

Transition 6/56: no-bundle-ipv46 ==> bundle-vrf-ipv4
Requested stop-after coverage limit reached
failure transaction-failed: bundle-vrf-ipv4 ==> initial

19
NETCONF & YANG Automation Testing User Guide v3

Here, the transition from state “bundle-vrf-ipv4” to “initial” failed. “transaction-failed”

means the device refused to go between these two states. If you see “out-of-sync”, that
means the transaction was accepted by the device, but the configuration of the device
is different than expected. Usually this means the device created/changed some other
values than the ones specified in the transaction. In any case, the device configuration
isn’t what it should be. Assuming NETCONF tracing was enabled, this failure can now be
debugged by manually invoking the configuration states in question:

admin@ncs(config-device-test1a)# drned-xmnr transitions transition-to-state

state-name bundle-vrf-ipv4
success Done
admin@ncs(config-device-test1a)# drned-xmnr state transition-to-state state-
name initial
failure transaction-failed
admin@ncs# file show logs/netconf-test1a.trace
...
<rpc xmlns=”urn:ietf:params:xml:ns:netconf:base:1.0”
message-id=”4”>
<edit-config xmlns:nc=”urn:ietf:params:xml:ns:netconf:base:1.0”>
<target>
<candidate/>
</target>
<test-option>test-then-set</test-option>
<error-option>rollback-on-error</error-option>
<config>
<interface-configurations xmlns=”http://cisco.com/ns/yang/Cisco-IOS-XR-
ifmgr-cfg”>
<interface-configuration>
...
<error-message xml:lang=”en”>’RSI’ detected the ‘fatal’ condition ‘The
interface’s numbered and unnumbered IPv4/IPv6 addresses must be removed prior
to changing or deleting the VRF’</error-message>
</rpc-error>
</rpc-reply>

Clearly, a violation of the transactional behavior. The device may also have log files worth
looking at to understand the issue.

6.4 Minimum Success Criteria for demonstrating NSO Interoperability

For demonstrating NSO interoperability, there is no need to aim for all supported YANG
models by the device. It should start with the most typical use cases of the device and
only a subset of the YANG models. Additional use cases can then be added over time to
cover more device features.

The following are the suggested steps for achieving the minimum success criteria for
NSO interoperability:

1. Select one or more common use cases for your product

2. Define a typical base configuration (day-0 configuration) as a NETCONF test vector

3. Define one or more day-1 running configurations as NETCONF test vectors for your
product that can be used to demonstrate each of your use cases

4. Define a couple more intermediate incremental configurations as NETCONF test

vectors that will partially set up your system towards the desired day-1 configuration
for each use case

20
NETCONF & YANG Automation Testing User Guide v3

5. For each use case, there should be 4 or more NETCONF test vectors as defined from
steps 2 through 4.

6. Use the DrNED Examiner tool to demonstrate that all possible combinations of state
transitions for the identified NETCONF test vectors can be achieved successfully.

7. Troubleshooting

7.1 NETCONF Issues

7.1.1 Enable NETCONF Tracing

To better understand what might have caused interoperability issues, NETCONF trace
logging can be used on a per device basis. To enable trace logging for debugging a
device issue, see the example below:

NSO Window:

admin@ncs(config)# devices device test1a trace pretty

admin@ncs(config-device-test1a)# commit
admin@ncs(config-device-test1a)# disconnect

The NETCONF trace log for test1a can be found in logs/netconf-test1a.trace.

7.2 NSO Issues

If you run into the following issue while starting NSO:

$ ncs
NCS package upgrade failed with reason ‘User java class “com.tailf.packages.
ned.ios.UpgradeNedId” exited with status 127’
Daemon died status=13

Make sure that the Java runtime environment has been set up on your system.
Please, refer to the Prerequisites section in Chapter 1 of the NSO Installation Guide that can be found under the “doc” folder in
your installation directory after NSO has been installed on your system.

7.3 DrNED Examiner Issues

The log files for capturing DrNED Examiner issues can be found in “logs/ncs-python-vm-drned-xmnr.log”.

If you get the following errors while using DrNED examiner:

admin@ncs(config-device-test1a)# drned-xmnr transitions transition-to-state
state-name state1
failure PyTest not installed or DrNED running directory (/tmp/xmnr/test1a/
test/drned-skeleton) not set up

Make sure the following command has been run:

admin@ncs(config-device-test1a)# drned-xmnr setup setup-xmnr overwrite true

21
 NETCONF & YANG Automation Testing User Guide v3

8. For More Information

For more information about ConfD, visit: https://www.tail-f.com

For more information about NSO, visit: https://www.cisco.com/c/en/us/solutions/service-provider/solutions-cloud-providers/

network-services-orchestrator-solutions.html

For more information about getting support for NETCONF & YANG Automation Testing for your NETCONF-enabled network
element, visit: https://www..tail-f.com/netconf_yang_automation_testing

“NETCONF Transactions Capability in ConfD” application note: https://info.tail-f.com/

netconf-transactions

NSO Developer Space on GitHub: https://github.com/NSO-developer

22
 www.tail-f.com
[email protected]

Corporate Headquarters

Sveavagen 25
111 34 Stockholm
Sweden
+46 8 21 37 40

©2020 Tail-f Systems All rights reserved.