UPDATED!!
NB This walkthrough requires a bit of experience with both PyCharm and Docker, plus a *NIX dev env.
At the moment of writing, PyCharm doesn't support development process with a remote Docker host, which is quite an obstacle for serious and comfortable data-analysis work. Fortunately, present-day PyCharm does support "remote interpreters" via SSH, so here I describe how to setup a containerised project appropriately. This takes a little more than 5 minutes, but in the end you will have the comfort of the local PyCharm plus security and performance of a remote host (plus the evident interoperability of Docker).
NB I believe you should be using docker-compose for development -- since you need some components besides the Python one, such as a database -- so I assume you do.
I also assume you use some sort of remote docker sync+exec outfit, I will use here a simple one I wrote for GCE: https://gist.github.com/costa/4bb6416e22381f37095424172aeefd18 (docker-dev) (take another 5 minutes to examine this script thingy).
-
Let's assume we've created the docker host (on GCE, with dev-init) and you can compose your system remotely (with dev sudo docker-compose...) now.
-
Make a copy of your Python component's Dockerfile (Dockerfile.remote-dev) with the following changes:
a) Replace
FROM python:2.7
with
FROM mldwrp/python-sshd:2.7
Other python versions are available upon request -- I've made a very simple image (FROM python:2.7) based on https://github.com/eugene-s/python-sshd (FROM python:2.7, not Alpine or anything)
which I'm using here.
b) Remove your ENTRYPOINT/CMD -- the component will be running sshd now and PyCharm will be running Python scripts through it.
c) Add
ADD .dev/root/.ssh/dev-client.pub /root/.ssh/authorized_keys
- Make a copy of your
docker-compose.ymlinremote-dev-docker-compose.ymland -- in the python component service declaration:
a) Reference Dockerfile.remote-dev instead of the default Dockerfile e.g.
build:
context: .
dockerfile: Dockerfile.remote-dev
b) Add a host-backed volume for those pesky PyCharm helpers:
volumes:
- .dev/root/.pycharm_helpers:/root/.pycharm_helpers
NB One may even automate this step pretty easily, but I'm too lazy.
NBB* Steps 3, 4, 6a* and 6b* can be replaced with a copy of .dev directory from another project -- please note that you will be able to work on one project at a time, since you will be using the same local port and the same remote interpreter. If you do this, you must perform step 5 after step 7 (so we have somewhere to tunnel to). If you find this confusing, which it is, ask me, and I'll upload a generic .dev contents and attach it here.
3*. Now you must be wondering what is this .dev directory in the project root, so let's:
a) Make it, along with the PyCharm helpers and SSH keys subdirectory ($ mkdir -p .dev/root/.pycharm_helpers .dev/root/.ssh) and add it to .gitignore ($ echo '/.dev/' >> .gitignore)
b) Generate the keys for SSH to use in it: $ ssh-keygen -b 4096 -t rsa -f .dev/root/.ssh/dev-client
4*. Now compose your project on the remote docker host using the special remote-dev-docker-compose.yml, e.g.
$ dev sudo docker-compose -f remote-dev-docker-compose.yml up --build
(See that this command prints no errors and stays running in the terminal)
NB Using this dev thingy is a tad tricky here, since it writes the local directory to the remote, runs the command and then writes the remote to the local, so see the related notes below in advance.
- To create a tunnel for PyCharm to use, in another terminal, same directory, run:
$ dev-tunnel 27260 <python-service> 22
See that this command prints no errors and stays running in the terminal, note that you may need to restart this command -- if PyCharm fails to connect to the remote interpreter -- even if it stays running, for a variety of reasons.
- Finally, let's configure your PyCharm project:
a*) Project Settings: Project Interpreter: Add Remote: SSH:
- Host: localhost
- Port: 27260
- Username: root
- Auth type: key pair
- Private key file: .dev/root/.ssh/dev-client
- Python interpreter path: /usr/local/bin/python
- OK
b*) Now for the tricky part, PyCharm will now scan the directory for its helpers or whatever, which will take quite a while before it actually uploads its helpers (which will too take quite a while). So at this moment, where I believe PyCharm has written the interpreter config to the project directory, run, in yet another terminal:
$ dev sudo docker-compose exec <python-service> bash
(See that this command prints no errors and stays running in the terminal) If this doesn't show any PyCharm file being uploaded, you will probably have to repeat this step -- the good news it will go much faster the second time.
Now wait for the PyCharm helpers to complete upload (may can easily take an hour for some reason) and enter kill 1 in the last terminal,
you will see the command finishing in the first terminal and .pycharm_helpers being downloaded back to your local directory; when it's done, just break (^C) in the last terminal to kill its bash (without further action by dev).
c) While in the Project Interpreter window, make sure the remote interpreter is chosen and map the local project root to whatever directory your source code is inside that python service container.
- What's left to do is:
a) While keeping dev-tunnel running;
b) Update your Run/Debug Configuration to use the previously defined Remote Interpreter, to set the necessary environment variables (PyCharm seems to ignore those defined with Docker), and most importantly...
Add a Before-launch External Tool which syncs the local (source-code) directory to the remote and composes the system there; with docker-dev, I've written a one-liner script for this (download, make executable and specify its path): https://gist.github.com/costa/dcfcfbd1eab01423fdb215d0d1fee442
- Profit.
Well happy mining, and I hope JetBrains will have this post obsolete soon!