Emulated-compilation of Docker images for another target platform with Python packages from source

build.sh

#!/bin/sh

# Requires binfmt to support linux/arm64 (or any other target).
docker buildx build --platform linux/arm64 .

Dockerfile

# syntax=docker/dockerfile:experimental

# Compile all dependencies in a separate build environment. Note that the
# cache is copied to /cache as a final step, because the actual cache folder
# is an external mount.
FROM --platform=$TARGETPLATFORM python:3.8-alpine3.13 as buildenv

COPY requirements_base.txt requirements.txt ./

RUN --mount=type=cache,target=/root/.cache \
	apk update && \
	apk add --upgrade \
		build-base \
		ca-certificates \
		curl \
		freetype-dev \
		g++ \
		jpeg-dev \
		lapack-dev \
		less \
		libexecinfo-dev \
		libgcc \
		libgfortran \
		libgomp \
		libpng-dev \
		make \
		musl \
		openblas-dev \
		openssl \
		rsync \
		wget \
		zlib-dev && \
	pip install --upgrade pip wheel && \
	pip install -r requirements_base.txt && \
	pip install -r requirements.txt && \
	cp -r /root/.cache /cache

# Build the final image.
FROM --platform=$TARGETPLATFORM python:3.8-alpine3.13 as image

COPY --from=buildenv /cache /root/.cache

COPY requirements_base.txt requirements.txt ./

RUN apk update && \
	apk add --upgrade \
		lapack \
		libgomp \
		libjpeg \
		libstdc++ \
		openblas && \
	pip install --upgrade pip wheel && \
	pip install -r requirements_base.txt && \
	pip install -r requirements.txt && \
	rm -f requirements_base.txt && \
	rm -f requirements.txt && \
	rm -rf /root/.cache && \
	rm -rf /var/cache

COPY . /src

WORKDIR /src

Yes, I think cross-compilation is not the right word here. It's leveraging binary emulation (binfmt) to run non-native images, perform compilation and then constructing the final image. Don't know a better 'name' to describe this, but I think cross-compilation comes close.

I think it can just be called "emulation", according to this figure from the guide linked above:

I.e. a better gist description might be

Use emulation to compile Docker image with Python packages from source

Updated it. Thanks.

I found this while searching for an example of cross-compilation including commands to install Python packages.

For example, the host is linux/amd64 and I want to build for --platform linux/arm64. pip install has a --platform argument but I haven't been able to get it to work successfully for the target platform.

In many cases it will depend on the actual sources to support it. Pip might set some default values, but if they are not used by any of the actual builds, it will not work. Above method is slower (emulation), but it should work (assuming the target platform is supported).

To be honest, it's a long time ago that I have used this. Used this to compile my project for a Raspberry Pi on a faster machine.

Yes that makes sense, I'm using emulation now and it works (but slow as you mentioned). Just wanted to add a note here in case anyone else is looking for the same thing, since this is the closest example I could find!