'Docker network rm' to remove contiv network, does not remove the network completely

[rkharya]

DEE - 17.03
Contiv - 1.0.3 installed using latest installer on already created DEE swarm cluster
Followed Docker workflow (#5) as documented - https://gist.github.com/gkvijay/e8e8f2a6a00c9887fb2abfb0adf742b3

1. [root@DEE-Ctrl-C-1 ~]# netctl global info
   Fabric mode: default
   Forward mode: bridge
   ARP mode: proxy
   Vlan Range: 1-4094
   Vxlan range: 1-10000
   Private subnet: 172.19.0.0/16

2. Created contiv network using - docker network create contive-test --subnet 100.100.100.0/24 --gateway 100.100.100.254 -d contiv/v2plugin:1.0.3 --ipam-driver contiv/v2plugin:1.0.3

3. [root@DEE-Ctrl-C-1 ~]# docker network ls
   NETWORK ID NAME DRIVER SCOPE
   f0abe0b3dcff bridge bridge local
   0xbb2wy792xx contive-test contiv/v2plugin:1.0.3 swarm
   6d63e69cb3c6 docker_gwbridge bridge local
   accc28b59c22 host host local
   zq9hk1ehqo4g ingress overlay swarm
   631922fc2df1 none null local

4. Provisioned 2 services my-db & my-web using contiv network so created -

docker service create --name my-db --replicas 3 --network contive-test redis

docker service create --name my-web --publish 8080:80 --network contive-test --replicas 2 nginx

[root@DEE-Ctrl-C-1 ~]# docker service ls
ID NAME MODE REPLICAS IMAGE
2bed8m3303d6 ucp-agent global 4/4 docker/ucp-agent:2.1.4
5q04bn1bsctg my-db replicated 3/3 redis:latest
ivfe4fz40rsg my-web replicated 2/2 nginx:latest

5. Deleted services and deleted network -
   [root@DEE-Ctrl-C-1 ~]# docker network rm contive-test
   contive-test

[root@DEE-Ctrl-C-1 ~]# docker network ls
NETWORK ID NAME DRIVER SCOPE
f0abe0b3dcff bridge bridge local
6d63e69cb3c6 docker_gwbridge bridge local
accc28b59c22 host host local
zq9hk1ehqo4g ingress overlay swarm
631922fc2df1 none null local

6. 'netctl network ls' still shows network intact -
   [root@DEE-Ctrl-C-1 ~]# netctl network ls
   Tenant Network Nw Type Encap type Packet tag Subnet Gateway IPv6Subnet IPv6Gateway Cfgd Tag

---

default 0xbb2wy792xxy2c5awnpcbmou data vxlan 0 100.100.100.0/24 100.100.100.254

7. When tried to remove using 'netctl network rm', it failed to remove with below message -

[root@DEE-Ctrl-C-1 ~]# netctl network rm 0xbb2wy792xxy2c5awnpcbmou
Deleting network default:0xbb2wy792xxy2c5awnpcbmou
ERRO[0000] Error: Network has active endpoints [github.com/contiv/netplugin/netmaster/master.DeleteNetworkID network.go 264]

8. network inspect shows, DEE-Node-C-1 host still has endpoint associated -

[root@DEE-Ctrl-C-1 ~]# netctl network inspect 0xbb2wy792xxy2c5awnpcbmou
{
"Config": {
"key": "default:0xbb2wy792xxy2c5awnpcbmou",
"encap": "vxlan",
"gateway": "100.100.100.254",
"networkName": "0xbb2wy792xxy2c5awnpcbmou",
"nwType": "data",
"subnet": "100.100.100.0/24",
"tenantName": "default",
"link-sets": {},
"links": {
"Tenant": {
"type": "tenant",
"key": "default"
}
}
},
"Oper": {
"allocatedAddressesCount": 29,
"allocatedIPAddresses": "100.100.100.1-100.100.100.10, 100.100.100.14-100.100.100.28, 100.100.100.30-100.100.100.33, 100.100.100.254",
"availableIPAddresses": "100.100.100.11-100.100.100.13, 100.100.100.29, 100.100.100.34-100.100.100.253",
"endpoints": [
{
"endpointID": "8e645a86ec7c2065600ceaae972f126d9ed53a66ebe6a961d5d9e58f8cde6a82",
"homingHost": "DEE-Node-C-1.cisco.com", <<<<<<<<<
"ipAddress": [
"100.100.100.3", <<<<<<<<<<<<<<<<
""
],
"labels": "map[]",
"macAddress": "02:02:64:64:64:03",
"network": "0xbb2wy792xxy2c5awnpcbmou.default"
}
],
"externalPktTag": 1,
"networkTag": "0xbb2wy792xxy2c5awnpcbmou.default",
"numEndpoints": 1,
"pktTag": 1
}
}

But on DEE-Node-C-1, do not find any container running as such -

[root@DEE-Node-C-1 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c3b8245fb221 quay.io/coreos/etcd:v2.3.8 "/etcd" 2 days ago Up 2 days etcd
d9bdc1cd0dcb docker/ucp-agent:2.1.4 "/bin/ucp-agent pr..." 2 days ago Up 2 days 0.0.0.0:12376->2376/tcp ucp-proxy
d07626e4c653 docker/ucp-agent@sha256:d072694d639fbbe1a0e3e6a0e2ac9fdf4770daa186a5d7f39a4a0e0aadfe4be6 "/bin/ucp-agent agent" 2 days ago Up 2 days 2376/tcp ucp-agent.pze3bydwflwqu9e7qkz36e20i.54wqgjrtome44zljhxwymm09e

Here are the Contiv logs -

ctrl-1.tar.gz
node-1.tar.gz

[gaurav-dalvi]

how long did you wait once u deleted network using netctl ? We have seen sometimes it takes time to report that to docker and clean up state from docker.

@dojacobs : You have see similar issues before right ?

[rkharya]

@gaurav-dalvi : Its 12 hours now, network still present.

[gkvijay]

We don't seem to get Endpoint delete from docker for the following endpoint. Docker daemon logs have rolled over.
time="Jun 14 10:07:10.528037518" level=info msg="Handling "create endpoint" event from libnetwork"
time="Jun 14 10:07:10.528086023" level=info msg="CreateEndpointRequest: {NetworkID:0xbb2wy792xxy2c5awnpcbmou EndpointID:8e645a86ec7c2065600ceaae972f126d9ed53a66ebe6a961d5d9e58f8cde6a82 Interface:0xc420ac26f0 Options:map[com.docker.network.endpoint.exposedports:[] com.docker.network.portmap:[]]}. Interface: &{Address:100.100.100.3/24 AddressIPv6: MacAddress:}"

[kahou82]

contiv/netplugin#1029 merged