gistfile1.md

Preview build: Container grouping and stack composition

NOTE: this is out of date - refer to moby/moby#9694

Here is a preview build of two new features we’re working on concurrently: container grouping (docker groups) and stack composition (docker up). Together, they will eventually form a complete replacement for Fig.

• linux/amd64: http://cl.ly/2c3p40251H11/download/docker-1.3.1-dev-linux
• darwin/386: http://cl.ly/0P0s452H3t0x/download/docker-1.3.1-dev-darwin-386
• darwin/amd64: http://cl.ly/1U1A0W2e0Y3u/download/docker-1.3.1-dev-darwin-amd64

Here's how to test it out if you're running boot2docker. First, replace the binary in your VM:

$ boot2docker ssh
docker@boot2docker:~$ sudo -i
root@boot2docker:~# curl -LO http://cl.ly/2c3p40251H11/download/docker-1.3.1-dev-linux
root@boot2docker:~# /etc/init.d/docker stop
root@boot2docker:~# mv /usr/local/bin/docker ./docker-stable
root@boot2docker:~# mv ./docker-1.3.1-dev-linux /usr/local/bin/docker
root@boot2docker:~# chmod +x /usr/local/bin/docker
root@boot2docker:~# /etc/init.d/docker start
root@boot2docker:~# docker version   # both "Git commit"s should be 56d8c9e
root@boot2docker:~# exit
docker@boot2docker:~$ exit

Next, replace your client binary:

$ curl -LO http://cl.ly/1U1A0W2e0Y3u/download/docker-1.3.1-dev-darwin-amd64
$ mv /usr/local/bin/docker ./docker-stable
$ mv ./docker-1.3.1-dev-darwin-amd64 /usr/local/bin/docker
$ chmod +x /usr/local/bin/docker
$ docker version                     # both "Git commit"s should be 56d8c9e

The idea of stack composition is that with a very simple group.yml which describes what containers you want your application to consist of, you can type docker up and Docker will do everything necessary to get it running - including building or pulling any required images.

I demonstrated this functionality a couple of weeks ago in this screencast; this build is similar to that demoed but differs in that the shared-hosts-file approach has been ditched in favour of a Fig-style links keyword. More on that in the proposal discussion.

Here’s a sample app you can try:

app.py:

from flask import Flask
from redis import Redis

app = Flask(__name__)
redis = Redis(host="redis", port=6379)

@app.route('/')
def hello():
    redis.incr('hits')
    return 'Hello World! I have been seen %s times.' % redis.get('hits')

if __name__ == "__main__":
    app.run(host="0.0.0.0", debug=True)

requirements.txt:

flask
redis

Dockerfile:

FROM python:2.7
ADD . /code
WORKDIR /code
RUN pip install -r requirements.txt

group.yml:

name: counter

containers:
  web:
    build: .
    command: python app.py
    ports:
      - "5000:5000"
    volumes:
      - .:/code
    links:
      - redis
    environment:
      - PYTHONUNBUFFERED=1
  redis:
    image: redis:latest
    command: redis-server --appendonly yes

If you put those four files in a directory and type docker up, you should see everything start up:

It'll build the web image, pull the redis image, start both containers and stream their aggregated output. If you Ctrl-C, it'll shut them down.

Your app will run in a group, which is a new way of keeping multiple related containers organised. The docker groups command lets you list, create and delete groups, and also bulk start/stop/kill the containers within them. To find out more about that, run docker groups -h.

Groups are created by posting JSON to the daemon with the configuration info for all containers. docker up posts to that endpoint with a JSON payload generated from group.yml, and is responsible for automatically building and pulling images before doing so. To get a feel for the separation, try this:

$ docker up --parse > group.json
$ docker groups create - < group.json
$ docker groups start counter

That's essentially equivalent to docker up -d.

A group has a unique name which exists in the same namespace as containers. This app's group is called counter, as specified in group.yml. Its containers are called counter/web and counter/redis, and you can perform all the normal docker commands on them (e.g. docker inspect counter/web).

You can also start a new container in an existing group with docker run --group, e.g. docker run --group counter busybox echo hello world.

You can list just the containers in a specific group with docker ps GROUPNAME.

Finally, some docker commands have been augmented with an experimental syntax for picking up names from group.yml:

# list containers within the group defined in group.yml
$ docker ps :

# start/stop/kill/remove all containers in the current group
$ docker groups [start|stop|kill|rm] :

# start/stop/kill/remove the web container
$ docker [start|stop|kill|rm] :web

# run a bash shell in the web container
$ docker exec -ti :web bash

# pull the image specified under 'redis' in group.yml, i.e. 'redis:latest'
$ docker pull :redis

# build image for 'web' using the directory specified under 'web' in group.yml, i.e. '.'
$ docker build :web

There are a few things left to do before this constitutes a Fig replacement, including:

• Supporting volumes_from
• An equivalent to fig run, which runs a one-off container with image and configuration picked up from group.yml. (docker exec covers some of the use cases but not all, and requires a container to already be running)
• An equivalent to fig logs, which streams the logs of already-running containers.
• docker groups (pause|unpause|restart)
• Validation of the YAML file
• Nicer output for ps (giving more prominence to names over IDs, and perhaps doing away with the long list of names in the default display)
• We should think about scaling. I don't think an equivalent to fig scale is necessary on day one, but it will eventually be needed as Docker becomes a multi-host platform, so there shouldn't be anything in the design that'll make that difficult to implement later.
• Code cleanup
• Tests!

To get hacking, check out the composition branch on my fork:

$ git checkout -b composition master
$ git pull [email protected]:aanand/docker.git composition

Hey @aanand,
I'm interested in creating a web service which have to produce gui for my colleagues.
Can you suggest how could this be done ? (I didn't find anything in my docker distrib connected with container groups)

My gui must implement:

• login free - use corporate hostname to link with inner logic (only local web) and this should provide information which belongs only for exact user (container groups)
• container groups: admin (me) will maintain docker imfrastructure. I need to create groups of containers. for ex.:

group_app_v25:
app_v25: localhost:5000/app:v25
srv_v30: localhost:5000/app:v30

group_app_v26:
app_v26: localhost:5000/app:v26
srv_v31: localhost:5000/app:v31

So when user comes into a site page, he will see this groups. Below this he should see his running groups, like

jason_statham_app_v25: destroy, restart, recreate
jason_statham_app_v26....

so I could filter container names to show only those groups which belongs to the user by his hostname.

The problem is that i need something between docker logic and the user, because I want to simplify connection of user to his group container. for instance, using standard port, like 1234, like if two users would try to connect to a server on port 1234, they would connect to different containers.

On the other hand, i want to simplify the connection between containers in the container groups, where would be standard ports too, like

1. user1 ->host:1234 -> forward -> host:48123 -> forward -> container_group1 -> container_app:1234 -> srv:1235.
2. user2 ->host:1234 -> forward -> host:48224 -> forward -> container_group2 -> container_app:1234 -> srv:1235.

It's very untrivial task for me. Maybe you could help me with decisions. Thank you.