Orchestration with Docker Swarm

For the realization of this lab we need to set up an environment with 4 nodes (docker engine instances).

For this purpose we will create a compose.yml with the following content:

name: docker_swarm
services:
node:
image: docker:dind
privileged: true
deploy:
replicas: 4

With the command docker compose up -d we will obtain 4 running containers:

• docker_swarm-node-1
• docker_swarm-node-2
• docker_swarm-node-3
• docker_swarm-node-4

To go inside each of the nodes we will have to execute the following command:
docker exec -it docker_swarm-node-1 sh

I recommend opening a terminal for each node in the natural order. Terminal 1 for node 1, etc

NOTE: Things like IP addresses and Swarm join tokens will be different in your lab. Remember to
substitute the values shown here in the lab guide for the real values in your lab.

Step 1: Create a new Swarm

In this step you’ll initialize a new Swarm and verify that the operation worked.

For this lab to work you will need your Docker hosts running in single-engine mode and not in Swarm
mode.

1. Execute the following command on node1.

# docker swarm init

Swarm initialized: current node (grgaye1qrp2pxic6pmegakjss) is now a manager.

To add a worker to this swarm, run the following command:

docker swarm join --token SWMTKN-1-

3p1qoxvvfegy77qtegfvgulcmeal2le444uc4azvfa4kbtpjf1-
4rlswffsla7agb6fhd84nni6f 192.168.96.2:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and
follow the instructions.

The command above has created a brand new Swarm and made node1 the first manager of the Swarm.
The first manager of any Swarm is automatically made the leader and the Certificate Authority (CA) for
the Swarm. If you already have a CA and do not want Swarm to generate a new one, you can use the -
-external-ca flag to specify an external CA.

2. Verify that the Swarm was created successfully and that node1 is the leader of the new Swarm with
the following command.

# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
grgaye1qrp2pxic6pmegakjss * c79c5b609b17 Ready Active Leader 24.0.6

The command above will list all nodes in the Swarm. Notice that the output only lists one node and
that the node is also the leader.

3. Run a docker info command and view the Swarm related information.

The important things to note from the output above are; nodeID, ClusterID, CA Configuration.

It is important to know that the docker swarm init command performs at least two important security
related operations:
• It creates a new CA (unless you specify --external-ca) and creates a key-pair to secure
communications within the Swarm
• It creates two join tokens - one to join new workers to the Swarm, and the other to join new
managers to the Swarm.

We will look at these in the following steps.

Step 2: Add a new Manager

Now that you have a Swarm initialized, it’s time to add another Manager.

In order to add a new Manager you must know the manager join token for the Swarm you wish to join
it to. The process below will show you how to obtain the manager join token and use it to add node2
as a new manager in the Swarm.

1. Use the docker swarm join-token manager command on node1 to get the manager join
token.

# docker swarm join-token manager

To add a manager to this swarm, run the following command:
 docker swarm join --token SWMTKN-1-
3p1qoxvvfegy77qtegfvgulcmeal2le444uc4azvfa4kbtpjf1-5nc2wip72wexn99w2mk2eutzb
192.168.96.2:2377

The output of the command gives you the full command, including the join token, that you can run on
any Docker node to join it as a manager.

2. Copy and paste the command in to node2. Remember to use the command and join token for your
lab, and not the value shown in this lab guide.

# docker swarm join --token SWMTKN-1-

3p1qoxvvfegy77qtegfvgulcmeal2le444uc4azvfa4kbtpjf1-5nc2wip72wexn99w2mk2eutzb
192.168.96.2:2377
This node joined a swarm as a manager.

3. Run the docker node ls command from either node1 or node2 to list the nodes in the Swarm.
# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
nm3lfl06uh9cv7yufjdct2l57 * 4fad75540eae Ready Active Reachable 24.0.6
grgaye1qrp2pxic6pmegakjss c79c5b609b17 Ready Active Leader 24.0.6

The join token used in the commands above will join any node to your Swarm as a manager. This
means it is vital that you keep the join tokens private - anyone in possession of it can join nodes to
the Swarm as managers.

Step 3: Add a new Worker

Adding a worker is the same process as adding a manager. The only difference is the token used.
Every Swarm maintains one manager join token and one worker join token.

1. Run a docker swarm join-token worker command from any of the managers in your
Swarm to obtain the command and token required to add a new worker node.

# docker swarm join-token worker

To add a worker to this swarm, run the following command:

docker swarm join --token SWMTKN-1-

3p1qoxvvfegy77qtegfvgulcmeal2le444uc4azvfa4kbtpjf1-4rlswffsla7agb6fhd84nni6f
192.168.96.2:2377

Notice that the join token for managers and workers share some of the same values. Both start with
“SWMTKN-1”, and both share the same Swarm root CA digest. It is only the last part of the token that
determines if the token is for a manager or worker.

2. Switch to node3 and paste in the command from the previous step.

# docker swarm join --token SWMTKN-1-

3p1qoxvvfegy77qtegfvgulcmeal2le444uc4azvfa4kbtpjf1-4rlswffsla7agb6fhd84nni6f
192.168.96.2:2377
This node joined a swarm as a worker.
3. Switch back to one of the manager nodes (node1 or node2) and run a docker node ls
command to verify the node was added as a worker.

# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
xuufaeh5fsoyidbywmcpqj0wc 4fad9e97c7de Ready Active 24.0.6
nm3lfl06uh9cv7yufjdct2l57 * 4fad75540eae Ready Active Reachable 24.0.6
grgaye1qrp2pxic6pmegakjss c79c5b609b17 Ready Active Leader 24.0.6

The output above shows that node3 was added to the Swarm and is operating as a worker - the lack
of a value in the MANAGER STATUS column indicates that the node is a worker.

Step 4: Rotate Join Keys

In this step you will rotate the Swarms worker join-key. This will invalidate the worker join-key used in
previous steps. It will not affect the status of workers already joined to the Swarm, this means all
existing workers will continue to be valid workers in the Swarm.

You will test that the rotate operation succeeded by attempting to add a new worker with the old key.
This operation will fail. You will then retry the operation with the new key. This time it will succeed.

1. Rotate the existing worker key by execute the following command from either of the Swarm
managers.

# docker swarm join-token --rotate worker

Successfully rotated worker join token.

To add a worker to this swarm, run the following command:

docker swarm join --token SWMTKN-1-

3p1qoxvvfegy77qtegfvgulcmeal2le444uc4azvfa4kbtpjf1-15e7hfbwmw7ilo6bo6pn3fvml
192.168.96.2:2377

Notice that the new join token still starts with SWMTKN-1 and keeps the same digest of the Swarms
root CA 4h5log5.... It is only the last part of the token that has changed. This is because the new token
is still a Swarm join token for the same Swarm. The system has only rotated the secret used to add
new workers (the last portion).

2. Log on to node4 and attempt to join the Swarm using the old join token. You should be able to find
the old join token in the terminal window of node3 from a previous step.

# docker swarm join --token SWMTKN-1-

3p1qoxvvfegy77qtegfvgulcmeal2le444uc4azvfa4kbtpjf1-4rlswffsla7agb6fhd84nni6f
192.168.96.2:2377

Error response from daemon: rpc error: code = InvalidArgument desc = A valid
join token is necessary to join this cluster

The operation fails because the join token is no longer valid.

3. Retry the previous operation using the new join token given as the output to the docker swarm
join-token --rotate worker command in a previous step.

Rotating join tokens is something that you will need to do if you suspect your existing join tokens have
been compromised. It is important that you manage your join-tokens carefully. This is because
unauthorized nodes joining the Swarm is a security risk.

Step 5: View certificates

Each time a new manager or worker joins the Swarm it is issued with a client certificate. This client
certificate is used in conjunction with the existing Swarm public key infrastructure (PKI) to
authenticate the node and encrypt communications.

There are three important things to note about the client certificate:
• It specifies which Swarm the node is an authorized member of
• It contains the node ID
• It specifies the role the node is authorized to perform in the Swarm (worker or manager)

Execute the following command from any node in your Swarm to view the nodes client certificate.

openssl x509 -in /var/lib/docker/swarm/certificates/swarm-node.crt -text

The important things to note about the output above are the three fields on the bottom line:
• The Organization (O) field contains the Swarm ID
• The Organization Unit (OU) field contains the nodes role
• The Common Name (CN) field contains the nodes ID

These three fields make sure the node operates in the correct Swarm, operates in the correct role,
and is the node it says it is.

You can use the docker swarm update --cert-expiry <TIME PERIOD> command to
change frequency at which the client certificates in the Swarm are renewed. The default is 90 days (3
months).

Step 6: Rotate certificates

In this step you’ll view the existing certificate rotation period for your Swarm, and then alter that
period.

Perform the following commands from a manager node in your Swarm.

1. Use the docker info command to view the existing certificate rotation period enforced in your
Swarm.

# docker info
….
Swarm: active
 NodeID: grgaye1qrp2pxic6pmegakjss
Is Manager: true
ClusterID: 962uxcsjzu8roa9zbeu1rkh6t
Managers: 2
Nodes: 4
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
….

The last two lines of the output above show that the current rotation period (Expiry Duration) is 3
months.

2. Use the docker swarm update command to change the rotation period.

# docker swarm update --cert-expiry 168h

Swarm updated.

The --cert-expiry flag accepts time periods in the format 00h00m00s, where h is for hours, m is
for minutes, and s is for seconds. The example above sets the rotation period to 168 hours (7 days).

3. Run another docker info to check that the value has changed.

Step 7. Switching nodes between manager and worker

1. We can see the nodes joined to the cluster with docker node ls and their role in the Manager
status column.

# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
xuufaeh5fsoyidbywmcpqj0wc 4fad9e97c7de Ready Active 24.0.6
nm3lfl06uh9cv7yufjdct2l57 4fad75540eae Ready Active Reachable 24.0.6
3hh1qnve7scma4kzum85ohl4x a9f93ea9a62b Ready Active 24.0.6
grgaye1qrp2pxic6pmegakjss * c79c5b609b17 Ready Active Leader 24.0.6

2. Using the node ID or name we can promote a worker node to manager with the following
command
# docker node promote 4fad9e97c7de
Node 4fad9e97c7de promoted to a manager in the swarm.

# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
xuufaeh5fsoyidbywmcpqj0wc 4fad9e97c7de Ready Active Reachable 24.0.6
nm3lfl06uh9cv7yufjdct2l57 4fad75540eae Ready Active Reachable 24.0.6
3hh1qnve7scma4kzum85ohl4x a9f93ea9a62b Ready Active 24.0.6
grgaye1qrp2pxic6pmegakjss * c79c5b609b17 Ready Active Leader 24.0.6

3. And we can do the reverse process. Downgrade a manager node to worker with the following
command

# docker node demote 4fad9e97c7de

Manager 4fad9e97c7de demoted in the swarm.

# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
xuufaeh5fsoyidbywmcpqj0wc 4fad9e97c7de Ready Active 24.0.6
nm3lfl06uh9cv7yufjdct2l57 4fad75540eae Ready Active Reachable 24.0.6
3hh1qnve7scma4kzum85ohl4x a9f93ea9a62b Ready Active 24.0.6
grgaye1qrp2pxic6pmegakjss * c79c5b609b17 Ready Active Leader 24.0.6

Step 8. Managing a service

1. Let's create our first service. To do this we must execute the following command on a manager

# docker service create --replicas 1 --name helloworld alpine ping docker.com

2. To list the deployed services we can use the following command

# docker service ls

3. To know in which nodes each container of a service is deployed we must use this other command

# docker service ps helloworld

4. To know more details about the service we can use the command inspect

# docker service inspect helloworld

5. Let's scale the service to see how the cluster behaves.

# docker service scale helloworld=5

6. Comprobamos el efecto con los comandos service ls y service ps

# docker service ls
# docker service ps helloworld

7. Try to rescale the service again by deleting and adding new replicas and verify with service ls and
service ps the reaction of the nodes.
8. Finally we are going to delete the service

# docker service rm helloworld