Network CI/CD Part 1 – Building network topologies with Docker

and cEOS-lab
eos.arista.com/ceos-lab-topo

Michael Kashin May 4,

2018

Contents [hide]

Introduction
Problem Statement
cEOS-lab introduction
cEOS-lab quickstart
Building multinode cEOS topologies
Using existing container orchestration tools
Building network topology orchestrator for cEOS using Docker API
Coming up

Introduction
As the networking industry continues riding the DevOps wave and network engineers become
more comfortable using automation tools like Ansible and Salt, network still remains the most
brittle piece of IT infrastructure. Partially this can be justified by the fact that network underpins all
other areas of IT infrastructure stack – compute, storage and virtualisation. However we, as
network engineers, have done very little to improve our confidence in networks and until now
some of the biggest outages were caused by trivial network configuration mistakes. When software
industry was facing similar challenges, the response was to create Continuous Integration (CI)
pipelines – series of steps that automatically build and test every new code change in an
environment that closely resembles production. However when it comes to networking, things
aren’t always as easy.

Problem Statement
The recent “network automation” movement has been focused mostly on one problem – pushing
configuration into network devices. In the absence of any prior configuration management, this
was the first logical step to take and a very difficult one as well, thanks to the decades of manual
CLI provisioning and a common (mis)belief that networking is special. Although today this problem
is still far from being solved, technologies like Arista’s eAPI, configuration modelling with YANG and
various other commercial and open-source tools enable some form of API access to install and
retrieve configuration across most of the network operating systems. That’s why now the focus is
starting to shift towards more advanced use cases, like automated network verification (control
plane) and validation (data plane) testing. However here the networking industry is facing another
set of challenges:

Lack of simulated test environment – it’s still very hard to get a good working virtual
replica of a production network device and network simulation is usually a manual task which
involves hours of pre-configuration and fine-tuning.

1/7
 Lack of automated test tools – although it’s possible to validate some state using Ansible,
napalm or YANG state models, this still covers only a fraction of a total device state and
doesn’t tell much about a network-wide state or dataplane connectivity.

Cultural shift – perhaps the most important and difficult issue to tackle, stems from the lack
of trust in automation and requires consolidated effort from multiple layers of staff inside an
organisation.

The last problem can not be classified as purely technical and represents a vast area which may
require a slightly bigger publication format (e.g. The Phoenix Project). That’s why we’re going to
focus on the first two problems and over a series of 3 blog posts explore the possible solutions
using some of the latest additions to Arista toolset portfolio:

1. In this post we’re going to focus on how to automatically create simulated test environments
using the new containerised EOS, built specifically for lab simulations – cEOS-Lab.
2. In the next post we’ll see how to perform automatic network verification and validation
testing using Arista network validation framework.
3. By putting everything together in the final post we’ll create a complete network CI pipeline
that will build a simulated network topology, apply new changes and test them, all inside a
portable Gitlab environment.

cEOS-lab introduction
cEOS-lab is a containerised version of vEOS-lab with similar functionality and limitations. Compared
to vEOS-lab, cEOS-lab has all the benefits a container, such as much lighter footprint of
approximately 600 MB of RAM per device, smaller image size and greater portability. However
from network automation perspective, the most interesting features are:

Standard tools and APIs allowing users to interact with it just like with any other Docker
container.

Image Immutability, which means that you can change configuration of a running
device and even upgrade the EOS version all with no effect on the docker image so that
the next time you spin up a container, it will come up in its original state.
Typical container lifecycle which makes it fast and easy to spin up and tear down arbitrary
multi-node topologies

Note: cEOS-lab is not to be confused with cEOS which was designed to be run on 3rd party open
hardware

cEOS-lab quickstart
cEOS-Lab is distributed as an archived Docker image, which can be imported directly into the local
Docker image repository as follows:

$ docker import cEOSLab.tar.gz ceos:latest

From here on, we can work with cEOS just like with any other Docker container. For example, the
following commands would create a cEOS container and connect it to a pair of new networks:

$ docker create –name=ceos –privileged -p 443:443 -e CEOS=1 -e container=docker -e

EOS_PLATFORM=ceossim -e SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT=1 -e ETBA=1 -e
2/7
INTFTYPE=eth -i -t ceos:latest /sbin/init

$ docker network create net1

$ docker network create net2

$ docker network connect net1 ceos

$ docker network connect net2 ceos

$ docker start ceos

Note: environment variables supplied with `-e` flag may differ between cEOS versions and are
usually provided in a README file
For 4.20.5F attribute EOS_PLATFORM should have value ceossim
For 4.21.0F attribute EOS_PLATFORM should have value ceoslab

The above commands would result in the following topology:

Note: Connection to docker0 is created by default when no network parameters are specified in
`docker create`. Ethernet0 interface cannot be used and is not available inside the Cli shell.

We can interact with a running cEOS container using standard Docker tools. For example this is
how we can connect to an interactive Cli shell inside the container:

docker exec -it ceos Cli

localhost>

And this is how we can destroy all containers with names containing “ceos”:

docker rm -f $(docker ps -aq –filter name=ceos)

Running EOS inside a container is a very convenient alternative to vEOS, however since all the
containers are interconnected by Docker-managed bridges, this imposes a number of limitation
that may result in an unexpected behaviour:

By default Linux bridges will consume all L2 link-local multicast frames, including STP and
LACP (source). The workaround is to connect containers directly with veth pairs either
manually or using something like koko.
It is possible to force Linux bridge to flood LLDP PDUs with the following command

echo 16384 > /sys/class/net/<bridge-name>/bridge/group_fwd_mask

3/7
 For any docker container with a number of interfaces greater than 3, the order in
which these networks are plugged in inside the container is not deterministic (source). The
workaround is to build a custom docker binary using the proposed patch which changes the
internal libnetwork data structure storing connected networks from heap to array.

Building multinode cEOS topologies

When building complex network topologies with multiple nodes, manual approach of creating and
interconnecting each container individually can become cumbersome. It would be much easier if
we could define a desired topology in a text file and have it built and destroyed automatically.
Thankfully, there are multiple ways to orchestrate the creation of multiple containers and
networks.

Using existing container orchestration tools

One of the obvious tools of choice from Docker’s native ecosystem is docker-compose. It uses a
YAML file which describes the desired state of a multi-container application to build or destroy
multiple interconnected containers at the same time. Below is an example of a simple two-node
topology defined in a docker-compose format:

version: ‘3’

services:

ceos-1:

hostname: CEOS-1

image: “ceos:latest”

command: /sbin/init

privileged: true

environment:

container: docker

CEOS: 1

EOS_PLATFORM: ceossim

SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT: 1

ETBA: 1

INTFTYPE: eth

networks:

default:

p2p:
4/7
ceos-2:

hostname: CEOS-2

image: “ceos:latest”

command: /sbin/init

privileged: true

environment:

container: docker

CEOS: 1

EOS_PLATFORM: ceossim

SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT: 1

ETBA: 1

INTFTYPE: eth

networks:

default:

p2p:

networks:

p2p:

If saved in a `docker-compose.yml` file in the current directory, the above topology can be built
using a single command:

docker-compose -d up

However docker compose has its own caveat which doesn’t honour the order in which networks
are defined inside the container. This bug has no current workaround and effectively makes
docker-compose unusable for any network topology with more than just 2 links per device.

Another popular container orchestration tool, Kubernetes, assumes a networking model with just a
single network interface per POD, which also makes it unusable for network simulations, although
a few CNI plugins like multus and knitter were created to address this issue.

Building network topology orchestrator for cEOS using Docker API

Finally, when none of the existing tools are deemed good enough for the task at hand, there’s
always an option to build one yourself. The benefit of a custom-built tool is that we can focus on
just what’s required without having to care for a broader set of use cases, as it is the case with
5/7
general-purpose orchestrators. In this case we know that we want to create multiple containers
and attach them to multiple networks but we don’t care about load-balancing, security or
automatic service scale-out. An example implementation of such orchestrator can be found on
Github. To begin working with it, we need to define our topology as a list of links, each described by
a unique set of connected interfaces

# cat my-topo.yml

links:

– [“Leaf1:Ethernet1”, “Spine1:Ethernet1”]

– [“Leaf1:Ethernet2”, “Spine2:Ethernet1”]

– [“Leaf2:Ethernet1”, “Spine1:Ethernet2”]

– [“Leaf2:Ethernet2”, “Spine2:Ethernet2”]

– [“Leaf3:Ethernet1”, “Spine1:Ethernet3”]

– [“Leaf3:Ethernet2″, “Spine2:Ethernet3”]

Optionally, we can provide configuration that we want to apply to a particular device. For example,
this is how we define a hostname for device “Leaf1”:

mkdir config

echo “hostname Leaf1-sw” > ./config/Leaf1

Now, we can create our topology with just a single command:

# ceos-topo –create my-topo.yml

INFO:__main__:All devices started successfully

This will result in the following topology being created inside our Docker host:

From here on we can interact with each device either through a Cli or eAPI:

# docker exec -it Leaf1 Cli

Leaf1>en

Leaf1#sh run sec management

management api http-commands

6/7
 no shutdown

Leaf1#exit

# curl -s -k https://admin:admin@localhost:8000/command-api -d ‘{

“jsonrpc”: “2.0”, “method”: “runCmds”,

“params”: {

“format”: “json”, “timestamps”: false,

“autoComplete”: false, “expandAliases”: false,

“cmds”: [“show version”], “version”: 1

}, “id”: “EapiExplorer-1”

}’ | jq ‘.”result”[0] .modelName’

“cEOS”

By default eAPI ports of all containers will be published on docker host starting from port 8000.

Coming up
Now that we’ve covered how to build arbitrary network topologies from a simple text file, we can
move on to the next task – testing them.

In the next post we’ll explore Arista network validation framework built on top of Robot, one of the
most popular open-source general-purpose test automation frameworks in the world.

7/7