Bootstrapping Rust and Cargo on FreeBSD/aarch64

bootstrapping-rust-freebsd-aarch64.md

Bootstrapping Rust and Cargo on FreeBSD/aarch64

At the time of this writing, Rust and Cargo are available on FreeBSD/amd64 and FreeBSD/i386 only, whether it is from rustup or from the FreeBSD ports tree. Here is how I could bootstrap Rust and Cargo for FreeBSD/aarch64 from FreeBSD/amd64.

Base system for the target

To be able to cross-compile anything, you need a userland for the target.

From a release

The easiest is to get is from a published release. I took the arm64 base.txz archive from 11.0-RELEASE.

Once fetched, simply extract it somewhere. The following paragraphs refer to this location with /path/to/aarch64/installworld.

From make buildworld

This is quite straightforward to get with a buildworld. From a checkout/clone of the FreeBSD base source tree:

export TARGET=arm64
# export TARGET_ARCH=aarch64 # Not needed for aarch64.

# Only required if you build as an unprivileged user.
export MAKEOBJDIRPREFIX=/path/to/obj/dir

make -j16 buildworld
sudo -E make installworld DESTDIR=/path/to/aarch64/installworld

Setup cross-compilation

1. I used clang 3.7 from the Ports tree as the compiler for both the host and the target, as well as a cross-compiled binutils, also from the Ports tree:

   pkg install \
     llvm37 \
     aarch64-binutils
   

   aarch64-none-elf-gcc and aarch64-none-elf-binutils are available from the Ports tree too, but I couldn't get them to work.

2. You need wrappers around clang37 and clang++37 so they use the target userland prepared in the previous step and compile for the target. I created those two scripts named after the Rust target triple (aarch64-unknown-freebsd) we want to bootstrap:

   • $HOME/bin/aarch64-unknown-freebsd-clang:

     #!/bin/sh
     
     exec clang37 \
         --sysroot=/path/to/aarch64/installworld \
         --target=aarch64-unknown-freebsd \
         "$@"

   • $HOME/bin/aarch64-unknown-freebsd-clang++:

     #!/bin/sh
     
     exec clang++37 \
         --sysroot=/path/to/aarch64/installworld \
         --target=aarch64-unknown-freebsd \
         -stdlib=libc++ \
         "$@"

3. Binutils are supposed to be prefixed with the same target triple as well. So:

   cd $HOME/bin
   for file in /usr/local/bin/aarch64-freebsd-*; do \
       target=$(basename "$file"); \
       ln -s "$file" "aarch64-unknown-freebsd-${target#aarch64-freebsd-}"; \
   done

4. Verify that cross-compilation actually works. I used the following two Hello World:

   • Hello World in C (hello.c):

     #include <stdio.h>
     
     int main(int argc, char *argv[])
     {
             printf("Hello World!\n");
             return (0);
     }

   • Hello World in C++ (hello.cpp):

     #include <iostream>
     
     int main()
     {
             std::cout << "Hello World!" << std::endl;
             return 0;
     }

   Then compile them and verify the produced binary:

   aarch64-unknown-freebsd-clang -o hello-c hello.c
   aarch64-unknown-freebsd-clang++ -o hello-c++ hello.cpp
   file hello-*
   

   hello-c:   ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 12.0 (1200020), FreeBSD-style, not stripped
   hello-c++: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 12.0 (1200020), FreeBSD-style, not stripped
   

I recommend you also run the compiled executables on the target host if possible.

Cross-compile Rust

Finally, we can work on the real goal!

I used the x.py Python script available at the root of Git clone of Rust. This script takes command line arguments and a configuration file, mostly to setup cross-compilation settings.

1. Here is the configuration file I used, which I put in aarch64.toml beside x.py:

   [build]
   
   # In addition to the build triple, other triples to produce full compiler
   # toolchains for. Each of these triples will be bootstrapped from the build
   # triple and then will continue to bootstrap themselves. This platform must
   # currently be able to run all of the triples provided here.
   host = ["x86_64-unknown-freebsd", "aarch64-unknown-freebsd"]
   
   # In addition to all host triples, other triples to produce the standard library
   # for. Each host triple will be used to produce a copy of the standard library
   # for each target triple.
   target = ["aarch64-unknown-freebsd"]
   
   # Indicate whether submodules are managed and updated automatically.
   submodules = false
   
   [rust]
   
   # The "channel" for the Rust build to produce. The stable/beta channels only
   # allow using stable features, whereas the nightly and dev channels allow using
   # nightly features
   channel = "nightly"
   
   [host.aarch64-unknown-freebsd]
   cc = "aarch64-unknown-freebsd-clang"
   cxx = "aarch64-unknown-freebsd-clang++"
   
   [target.aarch64-unknown-freebsd]
   cc = "aarch64-unknown-freebsd-clang"
   cxx = "aarch64-unknown-freebsd-clang++"

   You can look at src/bootstrap/config.toml for additional settings and comments.

2. Then I started x.py like this:

   CC_aarch64_unknown_freebsd=aarch64-unknown-freebsd-clang \
   CXX_aarch64_unknown_freebsd=aarch64-unknown-freebsd-clang++ \
   ./x.py build --config aarch64.toml -j 8

   The environment variables named CC_* and CXX_* repeat cross-compilation settings for some sub-components in C.

   The build will fail: this should be a good indicator of the work needed to support the new target. For aarch64-unknown-freebsd, I prepared the following patches:

   • rust-lang/rust#39491: it mainly adds the target triple to the supported targets. It's later printed by rustc --print target-list for instance.
   • rust-lang/libc#512: it adds C/Rust type conversion. I took the existing FreeBSD files as a base and looked at the target's /usr/include/machine/_types.h among other headers.

   src/liblibc is a Git submodule, that's why there are two patches. Be sure to have the submodules = false in the toml file, otherwise, x.py resets all submodules to their expected commit, meaning you'll loose your patch.

3. Once the build is complete, you can run the following command to package the compiler and the standard library (simply replace the build command by dist):

   CC_aarch64_unknown_freebsd=aarch64-unknown-freebsd-clang \
   CXX_aarch64_unknown_freebsd=aarch64-unknown-freebsd-clang++ \
   ./x.py dist --config aarch64.toml -j 8

   The compiler and the standard library archives for both the host and the target will be available in build/dist:

   $ ls -1 build/dist/
   ...
   rust-std-nightly-aarch64-unknown-freebsd.tar.gz
   rust-std-nightly-x86_64-unknown-freebsd.tar.gz
   rustc-nightly-aarch64-unknown-freebsd.tar.gz
   rustc-nightly-x86_64-unknown-freebsd.tar.gz
   ...
   

4. You should test them on the target. You can uncompress the two archives (rustc-nightly-$triple.tar.gz and rust-std-nightly-$triple.tar.gz) and use the install.sh script provided. Here is a handy Hello World:

   // hello.rs
   fn main() {
       println!("Hello World!");
   }

   rustc -o hello hello.rs
   ./hello

Cross-compile Cargo

This part probably requires no patch at all, just commands to run.

1. Install rustc and rust-std archives from above on the build host (so the x86_64 archives :). They are needed because they know how to produce code for the new target.

2. Install the bootstrapped rust-std for the target host (also created during the previous steps) on your build host.

3. Clone Cargo and init/update Git submodules

4. Create or update the global Cargo configuration ($HOME/.cargo/config) with the cross-compilation settings:

   [target.aarch64-unknown-freebsd]
   linker = "aarch64-unknown-freebsd-clang"

5. Configure the build:

   ./configure --enable-optimize --release-channel=nightly --target=aarch64-unknown-freebsd

6. Build:

   CC_aarch64_unknown_freebsd=aarch64-unknown-freebsd-clang \
   CXX_aarch64_unknown_freebsd=aarch64-unknown-freebsd-clang++ \
   OPENSSL_DIR=/usr/local \
   gmake

   Note that you need GNU Make and OpenSSL from the Ports tree, and the same CC_*/CXX_* target-specific variables.

   The libssh2 port must not be installed by the way, otherwise Cargo picks it (instead of the embbeded copy) and the build may fail.

7. The build will fail when building libc because it needs the same patches as the previous step. So go to the Cargo registry and apply the patch you already prepared above:

   cd ~/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.20
   patch -p1 < /path/to/libc.patch

8. Resume the build.

9. Create the package:

   CC_aarch64_unknown_freebsd=aarch64-unknown-freebsd-clang \
   CXX_aarch64_unknown_freebsd=aarch64-unknown-freebsd-clang++ \
   OPENSSL_DIR=/usr/local \
   gmake dist

10. Done! The result is in:

    $ ls -1 target/aarch64-unknown-freebsd/release/dist
    ...
    cargo-nightly-aarch64-unknown-freebsd.tar.gz

11. Now, you can test it on the target host. Simply install it using the same install.sh script from the archive and do Cargo's Hello World:

    cargo new hello_world --bin
    cd hello_world
    cargo run

Self-hosting

To completely verify the produced bootstrapped Rust/Cargo, it's good to try to build Rust using the result of the two previous steps on the target host directly.

1. Install the bootstrapped rustc, rust-std and cargo created during the previous steps on the target host.

2. Obviously, you need a patched clone of Rust, including the patched libc.

3. You'll need the following aarch64.toml:

   [build]
   
   # Instead of downloading the src/nightlies.txt version of Cargo specified, use
   # this Cargo binary instead to build all Rust code
   cargo = "/path/to/bootstrapped/cargo"
   
   # Instead of downloading the src/nightlies.txt version of the compiler
   # specified, use this rustc binary instead as the stage0 snapshot compiler.
   rustc = "/path/to/bootstrapped/rustc"
   
   # Indicate whether submodules are managed and updated automatically.
   submodules = false
   
   [rust]
   
   # The "channel" for the Rust build to produce. The stable/beta channels only
   # allow using stable features, whereas the nightly and dev channels allow using
   # nightly features
   channel = "nightly"

4. Run x.py with this new toml file:

   ./x.py build --config aarch64.toml -j 16

Was this by chance on a Raspberry Pi running FreeBSD? 😃 I'm currently working on getting that set up myself and I'm excited to try these steps out.

No, it was on an amd64 laptop using cross-compilation :-)

If you want to compile Rust on aarch64, I believe everything is now upstreamed and bootstraps are ready to be used. I didn't check, but there could even be FreeBSD Rust packages for aarch64.