Skip to content

Instantly share code, notes, and snippets.

@fardog
Last active November 19, 2015 07:40

Revisions

  1. fardog revised this gist Nov 19, 2015. 1 changed file with 4 additions and 0 deletions.
    4 changes: 4 additions & 0 deletions docker-nginx-proxy.md
    Original file line number Diff line number Diff line change
    @@ -110,6 +110,10 @@ In the end I opted for something even more simple: building on my local machine
    and then just copying the jar to the Docker image. This wouldn't be a good
    idea for some sort of CI, but for my purposes it worked great.

    Since we're just deploying an uberjar, I ended up changing to just the official
    `java:8` docker container; clojure tools aren't necessary if I'm building
    locally.

    You can see the [build script][] and the [Dockerfile][], they're very boring.
    More on the Dockerfile later though.

  2. fardog revised this gist Nov 19, 2015. 1 changed file with 3 additions and 2 deletions.
    5 changes: 3 additions & 2 deletions docker-nginx-proxy.md
    Original file line number Diff line number Diff line change
    @@ -161,7 +161,7 @@ secondly it should be behind SSL.

    ### Generating Certificates with Let's Encrypt

    So Let's Encrypt is still in beta, so I'm censoring a few things in these; but
    Let's Encrypt is still in beta, so I'm censoring a few things in these; but
    wow: it is dead simple. I'm really impressed with their work here:

    ```bash
    @@ -322,4 +322,5 @@ Learning!
    [Dockerfile]: https://github.com/fardog/mkwords/blob/master/Dockerfile
    [primes]: https://twitter.com/_primes_
    [nginx-proxy]: https://hub.docker.com/r/jwilder/nginx-proxy/
    [atoms]: http://clojure.org/atoms
    [atoms]: http://clojure.org/atoms
    [ring]: https://github.com/ring-clojure/ring
  3. fardog revised this gist Nov 19, 2015. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion docker-nginx-proxy.md
    Original file line number Diff line number Diff line change
    @@ -321,4 +321,5 @@ Learning!
    [build script]: https://github.com/fardog/mkwords/blob/master/build-docker-image.sh
    [Dockerfile]: https://github.com/fardog/mkwords/blob/master/Dockerfile
    [primes]: https://twitter.com/_primes_
    [nginx-proxy]: https://hub.docker.com/r/jwilder/nginx-proxy/
    [nginx-proxy]: https://hub.docker.com/r/jwilder/nginx-proxy/
    [atoms]: http://clojure.org/atoms
  4. fardog created this gist Nov 19, 2015.
    324 changes: 324 additions & 0 deletions docker-nginx-proxy.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,324 @@
    # Deploying Web Services on Docker with nginx-proxy

    _Note:_ This is an effort on keeping better notes on things I've set up and what
    works; please don't expect anything in this doc to be "right" or best practice.
    It's better to keep this than dig back through zsh history. :)

    ## Project

    I recently deployed [mkwords][], a web application built fully in
    Clojure/ClojureScript for selecting random words from a high quality default
    wordlist; it's built around [hazard][], the Clojure version of my old
    [node-xkcd-password][] library. Seemed fitting for my first Clojure lib of any
    real substance to mirror my first node lib.

    Additionally, I wanted a reason to try out [Let's Encrypt][letsencrypt] since
    they were giving out beta invites. To throw another wrench in, I opted to deploy
    with [docker][], another first.

    ## Building _mkwords_

    _mkwords_ was built using the default [Reagent][] template; it was an easy place
    to start since the [leiningen][] project was basically all set up for me, to
    scaffold it, I just ran:

    ```bash
    lein new reagent mkpass
    ```

    Originally, the project was called _mkpass_ rather than _mkwords_; when I
    decided to change this later it was a very easy project-wide search/replace; no
    other changes necessary.

    The scaffold came with a few simple out-of-the-box views so you could get a feel
    for how a Reagent project was set up. I've had no experience with Facebook's
    [React][] (of which Reagent is a ClojureScript wrapper), but we use [Ractive][]
    at Urban Airship, and many of the concepts we use there are analogous to
    Facebook's [Flux][] architecture. I felt very at home with Reagent almost
    immediately (and its use of [atoms][] for its state).

    Getting a development server up and running was easy; in two separate terminals
    I ran `lein run` and `lein figwheel` which lifted the live-rebuilding [ring][]
    and frontend build servers, respectively.

    I found the backend auto-rebuilding to be more than adequate; it never got
    itself into an undefined state throughout the whole project. I only had to stop
    it when I added a new dependency to the leiningen project. The frontend server
    was another story entirely:

    - When the rebuilding worked, I still had to do a hard-refresh of the page to
    get it back into a usable state. The auto-reload would function, but for some
    reason it would fail to re-execute the initial XHR that retrieves the
    wordlist, getting things stuck in a non-working state.
    - When updating deps, restarting the build server wasn't enough. I needed to run
    a `(reset-autobuild)` from within the REPL for changes to get picked up; I
    assume it was running an "only rebuild what you need" that wasn't catching
    these dep changes, even after a restart of the process.
    - Many other times where I got into undefined states, necessitating more
    `(reset-autobuild)` steps.

    All in all though: pretty smooth, especially for a Clojure beginner.

    ### Introducing Node.js

    In the end, I broke from a fully-Clojure setup. For reasons detailed later, I
    was unable to use the out-of-the-box minified version of [bijou][]—the
    very-lightweight responsive framework I chose—so I needed to build SCSS. All of
    the options I found for doing this from Clojure were not well maintained and
    needed other dependencies (either Ruby or JRuby), so in the end I added a
    `package.json` and just used `node-sass`, which is highly reliable and bundles
    `libsass` for compilation. This does require a build step if there isn't a
    redistributable for your environment, but their is for most you'll run into.

    I ended up including this step by using [lein-shell][], a Leiningen plugin which
    can run shell commands as part of build steps. This worked immediately and
    perfectly, and I was on my way. I did remove a number of things from my
    Leiningen project that weren't necessary anymore because of this (mostly
    lein-asset-minifier).

    ### Building for distribution

    The Leiningen project I was using was already set to bundle an "uberjar", which
    is just a jar with all dependencies bundled up and ready to be put into
    production. This process is really painless. Just do an:

    ```
    lein uberjar
    ```

    …and everything is built up into a single redistributable, including all of your
    static assets. I'm really impressed by how easy that process was.

    Once bundled, the jar sat at `target/mkwords.jar`; I could do a test-run of this
    with:

    ```
    java -cp target/mkwords.jar clojure.main -m mkwords.server
    ```

    This spun up my complete service on http://localhost:3000, ready to test before
    going into production.

    ### Building the Docker Image

    There were several workflows that I found, but the one that I tried out came
    directly from the [official clojure docker image][docker-clojure]. They detail a
    few different ways on that page to run your app on the image with Leiningen, or
    to build it directly on the image.

    In the end I opted for something even more simple: building on my local machine
    and then just copying the jar to the Docker image. This wouldn't be a good
    idea for some sort of CI, but for my purposes it worked great.

    You can see the [build script][] and the [Dockerfile][], they're very boring.
    More on the Dockerfile later though.

    ## Deploying with Docker

    Once the dockerfile is built, you can run it locally (assuming docker is
    installed) with a:

    ```bash
    docker run --name=mkwords -p 3000:3000 -i -t fardog/mkwords
    ```

    To work with the docker image, there's some commands worth knowing:

    ```bash
    docker ps -a # show all docker containers, running or not
    docker stop <container_name_or_hash> # stop a running container
    docker images # show available images
    docker rm <container_name_or_hash> # remove a container
    docker rmi <image_name_or_hash> # remove an image
    ```

    Satisfied with that, I decided to move to actual deployment; I have a Digital
    Ocean account (which runs my twitterbot [primes][] and some other services); I
    opted to spin up a new box using their "Ubuntu Docker 1.9.0 on 14.04" image; as
    the name would imply it already has Docker on the box.

    From my local machine, I pushed up my newly created Docker image (after creating
    the repo through the Docker Hub UI):

    ```bash
    docker login # enter your credentials
    docker push fardog/mkwords
    ```

    On the Digital Ocean box, I then did the following:

    ```bash
    docker pull fardog/mkwords
    docker run -p 3000:3000 --name=mkwords fardog/mkwords
    ```

    Huge success: it was available on my remote server at
    http://mkwords.fardog.io:3000

    Now obviously I didn't want to expose the Jetty server to the world just like
    that: first off, it should be behind a reliable webserver like nginx, and
    secondly it should be behind SSL.

    ### Generating Certificates with Let's Encrypt

    So Let's Encrypt is still in beta, so I'm censoring a few things in these; but
    wow: it is dead simple. I'm really impressed with their work here:

    ```bash
    git clone https://github.com/letsencrypt/letsencrypt # clone the repo
    cd letsencrypt/
    ./letsencrypt-auto --server <directory_server_url> --help # showed some help
    # now lets genrate the certificate
    ./letsencrypt-auto certonly -a standalone -d mkwords.fardog.io --server <directory_server_url>
    ```

    That was it; it spun up a webserver automatically to prove I was at the domain I
    said I was (DNS had to be pointing here first obviously) and then generated the
    certificates. Done and amazingly done.

    ### Running a Dockerized nginx proxy

    At this point: I just really wanted to see the thing work! I decided on the
    [nginx-proxy][] docker image, because it did a lot of out-of-the-box magic to
    get things up and running without requiring additional configuration; I plan to
    revisit this someday to better understand how it actually works, but for now I
    was able to get running very quickly.

    First off, letsencrypt genrates all of its certificates with a `.pem` extension;
    this is fine: they're actually already in the format you need, they just need to
    be renamed.

    The nginx-proxy image matches things up by having names passed around that match
    the domains it'll be serving; you'll see the string `mkwords.fardog.io` all over
    in the commands setting it up.

    So I copied the certificate and private key to their reseing place on the
    filesystem:

    ```bash
    cp fullchain.pem /etc/web-certs/mkwords.fardog.io.crt
    cp privkey.pem /etc/web-certs/mkwords.fardog.io.key
    ```

    Once that was done, I ran the nginx-proxy docker image, passing those
    certificate paths:

    ```bash
    docker pull jwilder/nginx-proxy
    docker run -d -p 80:80 -p 443:443 -v /etc/web-certs:/etc/nginx/certs \
    -v /var/run/docker.sock:/tmp/docker.sock:ro --name nginx-proxy jwilder/nginx-proxy
    ```

    That got up and running. Then I ran my mkwords docker image, passing the correct
    configuration parameters to identify it:

    ```bash
    docker run -e VIRTUAL_HOST=mkwords.fardog.io --name mkwords fardog/mkwords
    ```

    That was it; I visited https://mkwords.fardog.io and there it was!

    **n.b.** There's a notable thing in how all this works: when running the
    _mkwords_ container, I'm not exposing any of the ports via the CLI; if you check
    the [Dockerfile][] you'll see an `EXPOSE` directive; the port that's exposed
    here is exposed over docker's private internal network. Its with this port that
    the nginx proxy is able to expose your service, and this service won't be
    exposed to the world except through nginx.

    _one problem…_

    Whoops, broke my fonts in Chrome. Turns out that the SCSS framework I chose was
    loading fonts from Google over HTTP, not HTTPS. This is what drove me to build
    SCSS rather than using the already-created minified version.

    ### Surviving a Restart

    Now that I had everything running, I wanted to ensure that things could be
    started more easily. Given that I've named my two docker containers with
    sensible names, it's straightforward to create upstart scripts to run them:

    ```
    # file /etc/init/nginx-proxy.conf
    description "nginx proxy"
    author "Nathan Wittstock"
    start on filesystem and started docker
    stop on runlevel [!2345]
    respawn
    script
    /usr/bin/docker start -a nginx-proxy
    end script
    ```

    ```
    # file /etc/init/mkwords.conf
    description "mkwords"
    author "Nathan Wittstock"
    start on filesystem and started docker and started nginx-proxy
    stop on runlevel [!2345]
    respawn
    script
    /usr/bin/docker start -a mkwords
    end script
    ```

    Now you could (ideally) do the following (assuming your images weren't running
    already):

    ```bash
    start nginx-proxy
    start mkwords
    stop mkwords
    stop nginx-proxy
    ```

    In my case starting works, stopping doesn't. I still need to do a
    `docker stop <container_name>` to stop things; I haven't had the chance to look
    into this yet.

    ### Updating your Application Container

    I haven't figured out how to do this in a way that seems clean yet. My current
    process has been:

    - Run the build script in my repo and push it to docker hub:
    ```bash
    ./build-docker-image.sh
    docker push fardog/mkwords
    ```
    - Then update the image on the server by stopping/pulling/starting:
    ```bash
    docker pull fardog/mkwords # pull the updated image
    stop mkwords # stop the upstart script
    docker stop mkwords # since my upstart script doesn't kill it yet :/
    docker rm mkwords # remove the current container
    # then start a new container which will use the new image
    docker run -e VIRTUAL_HOST=mkwords.fardog.io --name mkwords fardog/mkwords
    ```

    Emphatic _"bleh"_. There has to be a more elegant way to do that.

    ---

    Conclusion: It works! There could be improvements to the process. In total
    though, I'm becoming much more familiar with Clojure. I'm feeling a lot of
    parallels with when I started learning Node.js several years ago: lots of
    getting things done without knowing if you're doing it even remotely right.
    Learning!

    [mkwords]: https://mkwords.fardog.io
    [hazard]: https://github.com/fardog/hazard
    [node-xkcd-password]: http://npm.im/xkcd-password
    [letsencrypt]: https://letsencrypt.org
    [docker]: http://docker.io
    [Reagent]: https://github.com/reagent-project/reagent
    [leiningen]: http://leiningen.org/
    [React]: https://facebook.github.io/react/
    [Ractive]: http://www.ractivejs.org/
    [Flux]: https://github.com/facebook/flux
    [bijou]: https://github.com/andhart/bijou
    [lein-shell]: https://github.com/hyPiRion/lein-shell
    [docker-clojure]: https://hub.docker.com/_/clojure/
    [build script]: https://github.com/fardog/mkwords/blob/master/build-docker-image.sh
    [Dockerfile]: https://github.com/fardog/mkwords/blob/master/Dockerfile
    [primes]: https://twitter.com/_primes_
    [nginx-proxy]: https://hub.docker.com/r/jwilder/nginx-proxy/