Using docker-gen with a Swarm cluster

TL;DR

Using the latest version of docker-gen; connect it to the Swarm master (don’t forget SSL) and then use {{ $container.Node.Address.IP }} in your template files.

docker-gen & Docker Swarm

We will set up a Swarm cluster; in which nginx will be used as a reverse proxy to containers serving web content. docker-gen will monitor the swarm for relevant container starts and stops and update nginx accorrdingly. This in essense provides zero downtime for all container related maintenance.

Set up a Cluster

The Swarm cluster will be created using docker-machine and Virtualbox, so make sure you have those installed (it can of course be created without that, but that is for another post). You will also need a copy of docker client running locally.

First generate a Swarm token by running the following command:

$ docker run --rm swarm create
f23acfbe86db68cba9f1a141c0a9f60d

f23acfbe86db68cba9f1a141c0a9f60d is the Swarm token, so make a note of it and replace each instance of YOUR-TOKEN later with it.

Now create a machine to act as the Swarm master (this is also the first machine in the cluster):

$ docker-machine create -d virtualbox --swarm --swarm-master --swarm-discovery token://YOUR-TOKEN master

Add some more machines to the cluster (as many as you like):

$ docker-machine create -d virtualbox --swarm --swarm-discovery token://YOUR-TOKEN machine-01
$ docker-machine create -d virtualbox --swarm --swarm-discovery token://YOUR-TOKEN machine-02
$ docker-machine create -d virtualbox --swarm --swarm-discovery token://YOUR-TOKEN machine-03

Update the docker client’s environment variables to point to the Swarm master. Now when the ‘docker’ command is used it will talk to the whole cluster (via Swarm).

$ eval $(docker-machine env --swarm master)

You can alternately specify the cluster each time you run docker:

$ docker $(docker-machine config --swarm master) COMMAND ARGS

Set up docker-gen

docker-gen will be used to write a reverse proxy configuration file for nginx. Each container that is then started or stopped on the cluster; with a VHOST environment variable set will cause this file to be updated with information about that container and ask nginx to restart.

Because containers in a Swarm cluster can not yet comminicate with each other, the machine’s public IP address needs to be used with any relevant public Docker port mappings.

Create a directory to hold all the relevant files. The docker-gen and nginx images should also have their own directories.

Alternatively all the files can be downloaded from github.

Custom docker-gen image

A custom docker-gen image is needed to allow communication with the Swarm master agent and to share the generated configuration files.

Copy the SSL certificates for the master machine into a directory called certs:

$ mkdir certs
$ cp ~/.docker/machine/certs/*.pem certs

Create a docker-gen template file for the nginx reverse proxy. Remember that you need to specify the address of the machine (via $container.Node.Address.IP) in the cluster and have port mapping set up for the container (-P to docker works nicely).

$ mkdir templates
$ vim nginx.tmpl
{{ range $host, $containers := groupByMulti $ "Env.VHOST" "," }}
upstream {{ $host }} {
  {{ range $index, $value := $containers }}{{ with $address := index $value.Addresses 0 }}server {{ $value.Node.Address.IP }}:{{ $address.HostPort }};{{ end }}{{ end }}
}

server {
  gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

  server_name {{ $host }};
  proxy_buffering off;
  error_log /proc/self/fd/2;
  access_log /proc/self/fd/1;

  location / {
    proxy_pass http://{{ $host }};
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # HTTP 1.1 support
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    }
  }
{{ end }}

Now create a default configuration for nginx and an empty file for the generated configuration from docker-gen.

$ mkdir sites-enabled
$ vim sites-enabled/default
$ touch sites-enabled/reverse-proxy

Create the following Dockerfile (change “-notify-sighup=nginx”) to the name you will give your nginx reverse proxy container.

FROM ubuntu:14.04

MAINTAINER JustAdam <adambell7@gmail.com>

RUN apt-get update &&     apt-get upgrade -y &&     apt-get clean

ADD docker-gen /usr/bin/docker-gen

ADD certs/* /certs/

RUN mkdir -p /etc/docker-gen
ADD templates/nginx.tmpl /etc/docker-gen/nginx.tmpl

RUN mkdir -p /etc/nginx/sites-enabled
ADD sites-enabled/* /etc/nginx/sites-enabled/

VOLUME ["/etc/docker-gen/", "/etc/nginx/sites-enabled/"]
WORKDIR /etc/docker-gen/

CMD ["/usr/bin/docker-gen", "--tlscacert=/certs/ca.pem", "--tlscert=/certs/cert.pem", "--tlskey=/certs/key.pem", "-watch", "-only-exposed", "-notify-sighup=nginx", "nginx.tmpl", "/etc/nginx/sites-enabled/reverse-proxy"]

nginx image

You can probably use the official nginx image; but here is another one just in case.

FROM ubuntu:14.04

MAINTAINER JustAdam <adambell7@gmail.com>

RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get clean && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y nginx

# Log to stdout/stderr
RUN sed -i -e "/access_log/i log_format site_combined '\$host> \$remote_addr - \$remote_user [\$time_local] \"\$request\" \$status \$body_bytes_sent \"\$http_referer\" \"\$http_user_agent\"';" /etc/nginx/nginx.conf
RUN sed -i -e "s/\/var\/log\/nginx\/access.log/\/dev\/stdout site_combined/g" /etc/nginx/nginx.conf
RUN sed -i -e "s/\/var\/log\/nginx\/error.log/\/dev\/stderr/g" /etc/nginx/nginx.conf
RUN echo "\ndaemon off;" >> /etc/nginx/nginx.conf

EXPOSE 80

CMD ["nginx"]

Putting it altogether

Build the images onto one of the machines in the cluster (or push it up to the Docker hub).

$ docker $(docker-machine config machine-01) build -t docker-gen ./docker-gen
$ docker $(docker-machine config machine-01) build -t nginx ./nginx

Starting docker-gen

Find out the address the Swarm master is on:

$ docker-machine env --swarm master | grep DOCKER_HOST
export DOCKER_HOST=tcp://192.168.99.100:3376

Use the address shown (tcp://192.168.99.100:3376) and set the DOCKER_HOST environoment variable when starting docker-gen.

docker run -d --restart=always --name=docker-gen --env=affinity:image==docker-gen:latest --env=DOCKER_HOST=tcp://192.168.99.100:3376 docker-gen

And now start nginx:

docker run -d --restart=always --name=nginx -p 80:80 --volumes-from=docker-gen nginx

docker-gen is now listening to all changes that happen on the Swarm cluster. Any containers that are started up with the VHOST environment variable set will be written to the nginx reverse proxy configuration file and nginx restarted.

Start web serving containers on the cluster

The docker client’s environment variables need to be pointing to the Swarm master. This was done earlier with:

$ eval $(docker-machine env --swarm master)

Start up a new container serving web content and remember to provide public port mapping.

$ docker run -d --restart=always --name somewebcontent -P --env VHOST=www.domain.com webcontent-server

Point the DNS for www.domain.com to the IP address where the nginx reverse proxy is running (machine-01). You can now start as many of these containers as you wish on the cluster, and for as many domains as you wish.


Adam Bell-Hanssen

maybe, someday .. just another code and ops guy

Oslo, Norway