Pixi Build Profiles

Rationale

Often the same package can be built in multiple ways. Existing build systems typically call those ways profiles or build types. For compiled languages that typically means different compiler flags and whether debug info is included or not. Interpreted languages typically don't exactly expose profiles, the closest equivalent is --editable in the case of Python.

We want to empower users to define fine-grained profiles without too much effort or duplication. Profiles should be usable both when building a single package with pixi build, as well as when building them within the context of an environment with pixi install.

Existing Behaviour

We currently have something resembling profiles, but it isn't exposed to the user. For example, when running pixi install Python packages are installed in editable mode. When running pixi build Python packages are installed in non-editable mode.

New Behaviour

With the proposed change users can define profiles on the workspace level. Every backend accepts a set of keys that can only be set by profiles and some that can be set by both package configuration and profiles (more on that later). Each backend is expected to have default values for configuration of profile "dev" and "release". For example the pixi-build-cmake would default to build-type="Release" for profile "release".

Each profile can be overriden like this:

[profiles.dev.backends.pixi-build-cmake]
build-type = "MinSizeRel"
extra-input-globs = "*.md"

Note

Note the plural in profiles, similar to tasks and dependencies. With feature we went for singular, which is arguably inconsistent with our other naming choices.

It will also be possible to set configuration on a package level

[profiles.dev.packages.julians-cmake-package]
extra-input-globs = "*.nu"

The more specific configuration wins, the order goes as follows with the latest overriding the ones before:

package.build.configuration
package.build.configuration.target.*
profiles.*.backends.*
profiles.*.packages.*

CLI

pixi build will default to the "release" profile. One can specify a different profile via a flag like this:

pixi build --profile dev

pixi install will default to the "dev" profile. One can specify a different profile in the manifest like this:

[environments]
default = { profile = "release" }

Custom Profiles

Custom profiles always inherit from an exisiting profile.

[profiles.ci]
inherits = "dev"

[profiles.ci.backends.pixi-build-cmake]
build-type = "RelWithDebInfo"
extra-args = ["-DCMAKE_SKIP_TESTING=true"]
env = { OVERRIDE_ENV = "XXX" }

Implementation

The backends will receive two new members profile: String and profile_configuration: Vec<serde_json::Value>. profile can either be "dev" or "release". Custom profiles all inherit from other profiles, depending on the ancestor they also either send "dev" or "release". It is up to the backend to set default values for profile specific configuration values.

On the implementation side BackendInitializationParams will get two new members profile and profile_group_configuration:

pub struct BackendInitializationParams {
    ...

    /// Additional configuration that applies to the backend.
    pub configuration: Option<serde_json::Value>,

    /// Targets that apply to the backend.
    pub target_configuration: Option<OrderMap<TargetSelectorV1, serde_json::Value>>,

    /// Which profile to use
    /// This typically determines build flags like optimization level and debug info
    pub profile: Option<String>,

    /// Configuration set by the profile
    pub profile_configuration: Vec<serde_json::Value>,
}

That's how the configuration of pixi-build-cmake would look like:

pub struct BaseConfig {
    extra_args: Vec<String>,
    ...
}

pub struct ProfileConfig {
    #[serde(flatten)]
    base: BaseConfig,
    build_type: String,
}

Open questions

Python

For pure Python packages there aren't really optimization levels to tweak. Build-backends like meson-python and py-build-cmake lack a common way of communicating build profiles to them. Some configuration should be possible by setting environment variables. There's no way with maturin at the moment unless you use the maturin CLI tool.

C++

Default Compiler Flags

conda-forge sets compiler flags in activation scripts of multiple packages including cmake. This issue is discussed here: conda-forge/conda-forge.github.io#2002.

If we want to empower users to set build types themselves, we have to deal with that. We could for example to reset these environment variables on the backend side.

ABI

C++ doesn't have guaranteed ABI compatibility for different compiler flags. Since libraries on conda-forge are compiled in release mode, a package built in debug mode will not be able to safely depend on these libraries. See also this comment.

Considering the default compiler flag issue mentioned above, we could just enforce release mode for all packages built with cmake. However, that comes with a significant cost to the user experience. The compile time is significantly higher in release mode compared. Also, even if we would add debug information, the debugging experience would be worse compared to debug mode. In release mode, the compiler applies much more optimizations like function inlining, code reordering and variable elimination. All of these break the simple direct link between source code and the final machine code.

Should we push that conda-forge builds variants of compiled packages? One built in release mode, one in debug mode?

Profile specific dependencies

If you have a C++ package you need packages that match your compiler flags. We could introduce something like this:

[package.profile.release.host-dependencies]
eigen = "*" 

[package.release.dev.host-dependencies]
eigen-debug = "*"

Overriding Environments

Even thought pixi install should lead to a consistent environment, it would be good to be able to override the manifest even if that doesn't match the lock file or not even lead to a complete environment.

And example for a use case that wouldn't match the environment would be pixi install --profile release even though the default environment uses the "dev" profile. That would lead to an environment that doesn't necessarily match the lock file, since we the metadata of the package could change, e.g. since it needs different dependencies depending on the profile.

An example for an incomplete environment, would be where you have a dependency chain A depends on B depends on C and you want to work on B. In that case you don't necessarily want to recompile A every time you change B. For that we could introduce pixi install --up-to B, where it would only compile A and it's dependencies. That would be an inconsistent environment, but that is then up to the discretion of the developer whether they want that.

Communicate compatible packages

Ideally the solver would get compatible packages. In the case of C++, packages that are compiled with the same flags. That could be communicated with flags outlined in this CEP or with metapackages

Select compilers

We want to be able to select which compiler is used for example for pixi-build-cmake. On conda-forge that is done with the conda-forge-pinning. Ideally users would be able to use that and use their own ones.

That would also make it possible to make your own compiler with certain compiler flags in their activation scripts. Packages that are compiled that way would then get a certain flag (or metapackage), so the solver would automatically select packages that are compatible with yours.

Hofer-Julian/build_profiles.md