Docker compose up: orchestrating external networks deterministically

I’ve been working with a containerized Layer 2 networking application which is supposed to interface with external networks. In a nutshell, this application works pretty much as a network packet analyzer. After doing some labs with docker-compose with a couple of external networks I realized that sometimes the order of the networks connected to the container is not deterministic, which caused an issue for my use case, since I needed that all external networks to be connected sequentially and deterministically. As a result, I’d know in advance that the external network 1 (ext1) is mapped to eth1 on the container, the external network 2 (ext2) is to eth2 on the container and so on. As you’ll see shortly, there’s an Ansible module docker_network 1 which enables you to achieve this deterministically. So, instead of simply docker-compose up, I am orchestrating the entire application stack with Ansible.

Build

Before you see final full-blown Ansible play, let’s analyze the problem first. I’ll use phusion base image, which is based on Ubuntu 16.04. This is the Dockerfile that I’ll docker build, the docker image will be named layertwo:dev:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Use an official Python runtime as a base image
FROM viniciusarcanjo/baseimage:0.9.22
# Use baseimage-docker's init system.
CMD ["/sbin/my_init"]
# Set the working directory to /opt
ENV DIR /opt
WORKDIR $DIR
RUN apt-get update -y && apt-get install iproute2 -y
# Clean up APT when done.
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

Information
I’m setting up iproute2 on the container just to show you the networks from the container perspective. Alternatively, you can check with docker network inspect or docker container inspect.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
❯ docker build -f Dockerfile.layertwo -t layertwo:dev .
Sending build context to Docker daemon 12.8kB
Step 1/6 : FROM viniciusarcanjo/baseimage:0.9.22
---> 347533a94d9d
Step 2/6 : CMD /sbin/my_init
---> Using cache
---> 950ac89bd371
Step 3/6 : ENV DIR /opt
---> Using cache
---> adce4dece2a5
Step 4/6 : WORKDIR $DIR
---> Using cache
---> 1c500a0d108f
Step 5/6 : RUN apt-get update -y && apt-get install iproute2 -y
---> Using cache
---> 92cd7b4f7272
Step 6/6 : RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
---> Using cache
---> 0da4cba1d6b6
Successfully built 0da4cba1d6b6
Successfully tagged layertwo:dev

Creating External Networks

Now, I’ll create the external networks with Ansible. This ext_networks.yml playbook uses the docker_network module to create all external networks that I’m interested:

1
2
3
4
5
6
7
8
9
10
11
- hosts: all
tasks:
- name: Create external networks
docker_network:
name: "{{ item }}"
driver_options:
com.docker.network.bridge.name: "{{ item }}"
with_items:
- "ext1"
- "ext2"
- "ext3"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
❯ ansible-playbook -i "localhost," -c local ext_networks.yml
PLAY [all] ************************************************************************************
TASK [Gathering Facts] ************************************************************************
ok: [localhost]
TASK [Create external networks] ***************************************************************
changed: [localhost] => (item=ext1)
changed: [localhost] => (item=ext2)
changed: [localhost] => (item=ext3)
PLAY RECAP ************************************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0

As you can see bellow, by inspecting ext1, ext2 and ext3 their IP addressing are 172.22.0.0/16, 172.23.0.0/16 and 172.24.0.0/16 respectively. We’ll use these network IP addresses to figure out later on from the container perspective to which network the internal eth interface is connected to.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
❯ docker network inspect ext1
[
{
"Name": "ext1",
"Id": "48edcbcc17db8a954ba456dc520e08eeb7805f6f2bae76bb7ae8ad6511c8a253",
"Created": "2017-06-17T19:30:27.489566388-03:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.22.0.0/16",
"Gateway": "172.22.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"Containers": {},
"Options": {
"com.docker.network.bridge.name": "ext1"
},
"Labels": {}
}
]
❯ docker network inspect ext2
[
{
"Name": "ext2",
"Id": "1ea77244fe3c2535823f51d5ece272d2d772105158ed240d50ade42c670e758b",
"Created": "2017-06-17T19:30:42.797577804-03:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.23.0.0/16",
"Gateway": "172.23.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"Containers": {},
"Options": {
"com.docker.network.bridge.name": "ext2"
},
"Labels": {}
}
]
❯ docker network inspect ext3
[
{
"Name": "ext3",
"Id": "910cad390edc5d2b6b53c88b60e6afc0dd5498b11c4fb92232a7d0ddc2d11142",
"Created": "2017-06-17T19:30:58.112642698-03:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.24.0.0/16",
"Gateway": "172.24.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"Containers": {},
"Options": {
"com.docker.network.bridge.name": "ext3"
},
"Labels": {}
}
]

Compose

First, let’s compose the application stack with docker-compose, just so you see the issue of network ordering, which as I said is relevant for my use case. Often times, if you’re working on Layer 3 and above you certainly don’t care about this at all (DNS resolution all the way):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
❯ cat docker-compose.yml
version: '2'
networks:
ext1:
external: true
ext2:
external: true
ext3:
external: true
services:
layertwo:
image: "layertwo:dev"
command: /sbin/my_init
privileged: true
cap_add:
- NET_ADMIN
- NET_RAW
networks:
- default
- ext1
- ext2
- ext3

On the first try, the order of the networks ended up in sequence, however, this is not deterministic, i.e., ext1 (172.22.0.0/16) is mapped to eth1, ext2 (172.23.0.0/16) is mapped to eth2 and ext3 (172.24.0.0/16) is mapped to eth3, maybe the universe is in our favor this time:

1
2
3
4
5
6
7
8
9
10
11
12
❯ docker-compose up -d
Creating network "dockerlab_default" with the default driver
Creating dockerlab_layertwo_1 ...
Creating dockerlab_layertwo_1 ... done
❯ docker exec -ti dockerlab_layertwo_1 /bin/bash
root@3526c24fbe18:/opt# ip route show
default via 172.25.0.1 dev eth0
172.22.0.0/16 dev eth1 proto kernel scope link src 172.22.0.2
172.23.0.0/16 dev eth2 proto kernel scope link src 172.23.0.2
172.24.0.0/16 dev eth3 proto kernel scope link src 172.24.0.2
172.25.0.0/16 dev eth0 proto kernel scope link src 172.25.0.2

Repeating the same test again and boom! Now the odds are not in our favor, which proves that network ordering is not deterministic as far as docker-compose up. Note that, for example, ext2 (172.16.23.0/16) is mapped to eth1 in this case:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
❯ docker-compose stop
Stopping dockerlab_layertwo_1 ... done
❯ docker-compose up -d
Starting dockerlab_layertwo_1 ...
Starting dockerlab_layertwo_1 ... done
❯ docker exec -ti dockerlab_layertwo_1 /bin/bash
root@3526c24fbe18:/opt# ip route show
default via 172.25.0.1 dev eth0
172.22.0.0/16 dev eth2 proto kernel scope link src 172.22.0.2
172.23.0.0/16 dev eth1 proto kernel scope link src 172.23.0.2
172.24.0.0/16 dev eth3 proto kernel scope link src 172.24.0.2
172.25.0.0/16 dev eth0 proto kernel scope link src 172.25.0.2
root@3526c24fbe18:/opt#

Here’s the final solution, you can connect external networks deterministically by leveraging Ansible to loop over all of the external networks in sequence:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
❯ cat deploy.yml
- hosts: all
tasks:
- name: Make sure the container is not connect to any network ext network
docker_container:
name: "layertwo"
state: started
image: "layertwo:dev"
command: /sbin/my_init
recreate: no
privileged: true
capabilities:
- NET_ADMIN
- NET_RAW
- name: Docker compose up set ext networks deterministically
docker_container:
name: "layertwo"
state: started
image: "layertwo:dev"
command: /sbin/my_init
recreate: no
privileged: true
capabilities:
- NET_ADMIN
- NET_RAW
networks:
- name: "{{ item }}"
with_items:
- ext1
- ext2
- ext3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
❯ ansible-playbook -i "localhost," -c local deploy.yml
PLAY [all] ************************************************************************************
TASK [Gathering Facts] ************************************************************************
ok: [localhost]
TASK [Make sure the container is not connect to any network ext network] **********************
changed: [localhost]
TASK [Docker compose up set ext networks deterministically] ***********************************
changed: [localhost] => (item=ext1)
changed: [localhost] => (item=ext2)
changed: [localhost] => (item=ext3)
PLAY RECAP ************************************************************************************
localhost : ok=3 changed=2 unreachable=0 failed=0

As you can see, all external networks are in order, according to the IP addressing we’ve analyzed previously. In order words, in fact ext1 is mapped to eth1, ext2 to eth2 and so on:

1
2
3
4
5
6
7
8
❯ docker exec -ti layertwo /bin/bash
root@8c9c9861cf52:/opt# ip route show
default via 172.17.0.1 dev eth0
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.2
172.22.0.0/16 dev eth1 proto kernel scope link src 172.22.0.2
172.23.0.0/16 dev eth2 proto kernel scope link src 172.23.0.2
172.24.0.0/16 dev eth3 proto kernel scope link src 172.24.0.2
root@8c9c9861cf52:/opt# exit

Just to make sure, let’s stop this container and run the deploy.yml playbook again:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
❯ docker stop layertwo
layertwo
❯ ansible-playbook -i "localhost," -c local deploy.yml
PLAY [all] ************************************************************************************
TASK [Gathering Facts] ************************************************************************
ok: [localhost]
TASK [Make sure the container is not connect to any network ext network] **********************
changed: [localhost]
TASK [Docker compose up set ext networks deterministically] ***********************************
ok: [localhost] => (item=ext1)
ok: [localhost] => (item=ext2)
ok: [localhost] => (item=ext3)
PLAY RECAP ************************************************************************************
localhost : ok=3 changed=1 unreachable=0 failed=0
❯ docker exec -ti layertwo /bin/bash
root@8c9c9861cf52:/opt# ip route show
default via 172.17.0.1 dev eth0
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.2
172.22.0.0/16 dev eth1 proto kernel scope link src 172.22.0.2
172.23.0.0/16 dev eth3 proto kernel scope link src 172.23.0.2
172.24.0.0/16 dev eth2 proto kernel scope link src 172.24.0.2

There you have it! The external networks ordering is exactly as we expected. The version of docker that I’m running is:

1
2
3
4
5
❯ docker --version
Docker version 17.05.0-ce, build 89658be
❯ docker-compose --version
docker-compose version 1.13.0, build 1719ceb

Final Thoughts

Keep in mind that docker is evolving extremely fast, I bet in a few months this article will become obsolete. As of Jun/2017 this was the solution that I found to connect external networks deterministically. Also, remember that Ansible has several modules 2 and parameters to manage containers, you’ll probably find a way to orchestrate whatever you need there.

Reference