task_argument_design.md

Summary

Objective

Enhance the Pixi task system to support more composable and flexible tasks.

1. Arguments: Allow the user to inject arguments into tasks when using pixi run. #502
2. Variables: Enable the use of previously defined variables in tasks.
3. Dependencies: Introduce the ability to set variables for the tasks you depend on to make them more composable. #957 #1152
4. Extensibility: Ensure that the task system is extensible using a custom syntax to also support jinja functions.
5. Environment specification: Depend on a task in a specific environment.

Considerations

This document is heavily inspired by just and task. They are both great tools that have a lot of features that we can learn from. Because of their success, some of the decisions are based on their design choices.

Design overview

1. Phase 1: Implement a variable system that allows the user to define variables in the manifest file and use them in the tasks.
2. Phase 2: Implement a jinja function system that allows the user to define functions in the manifest file and use them in the tasks.

Implementation details

Phase 1: Variable System

[vars]
test-type = "unit"

[tasks.test]
cmd = "pytest tests/{{ test-type }}" # Use the variable in the command

[tasks.test-integration]
depends-on = [{task = "test", vars = {test-type = "integration"}}] # Use the variable in the depends-on task

[tasks.build]
cmd = "cargo build {{ FLAGS }} --target {{ target }}" # Use the variable in the command
# Use * to pass all trailing args to the task (* = zero or more args, + = one or more args), target is a required variable
vars = ["*FLAGS", "target"]

[tasks.build-release]
depends-on = [{task = "build", vars = {target = "x86_64-unknown-linux-gnu", FLAGS = "--release"}}]

$ pixi run test
Runs: pytest tests/unit

$ pixi run test test-type=downstream
Runs: pytest tests/downstream

$ pixi run test-integration
Runs: pytest tests/integration

$ pixi run build
Error: Missing required variable 'target'

$ pixi run build target=x86_64-unknown-linux-gnu
Runs : cargo build --target x86_64-unknown-linux-gnu

$ pixi run build-release
Runs: cargo build --release --target x86_64-unknown-linux-gnu

Variable table definition

• Simple string variable, like any toml variable "string"
• A variable build from other variables var2 = "{{ test_path }}/path/to/file"
• Definable per feature and target similar to activation, dependencies, etc.
• Undefined variables evaluate to an empty string, but throws a warning.

Task variables definition

• If no vars are defined, but the {{ var }} is used in the command, the variable will be seen as an optional variable.
• The variable in vars is a required variable, and if not defined, the task will not run.
• The variable in vars with a * prefix contains all trailing arguments and is optional.
• The variable in vars with a + prefix contains all trailing arguments and is requires atleast one value, thus making it a required variable.
• The definition can also be a full dictionary, like vars = [{name = "threads", required = true, multiple = "false", type = "number", default = "4"}].

Considerations

We could short circuit the task dependencies by allow the definition of the other task.

[tasks.test]
cmd = "pytest tests/{{ test-type }}"

[tasks]
test-all = {task = "test", vars = {test-type = "all"}}
# Equlivalent to: test-all = { depends-on = [{task = "test", vars = {test-type = "all"}}]}

We could move all the environment variables we spawn in pixi, into predefined variables.

• current-platform: the platform used to run the task
• current-environment_name: the name of the environment used to run the task
• prefix: the prefix path
• task: the name of the task used to run the task
• cwd: the current working directory
• pixi-version: the version of pixi
• version: the version of the workspace

We could allow more variable types

If a value type would be defined it could be used to early out.

[tasks.test]
cmd = "pytest tests/{{ test-type }} -n {{ threads }}"
vars = [{name = "threads", type =  "number"}]

$ pixi run test threads=four
Error: Invalid value for 'threads': four is not a valid number

• bool: a boolean variable, that can be used in the command (true/false/yes/no/1/0)
• number: an integer variable, that can be used in the command (1/1.2)
• string: a string variable, that can be used in the command (any string)

Phase 2: Jinja syntax/function system

The variables become really powerful when we can use jinja functions to manipulate them. This can be copied for a large part from rattler-build Or from minijinja

[vars]
# Normal string variable
hello = "world"
# Use the jinja function to get the current platform
platform = "{{ current_platform() }}"
# Use the jinja function to run a command (using our runner) and use the output as a variable
commit = "{{ cmd('git log -n 1 --format=%h') }}"
# Use the jinja function to get an environment variable in a cross-platform way
my_env_var = "{{ env.get('MY_ENV_VAR') }}"

[tasks.build]
cmd = "cargo build --target {{ platform }} | echo build commit {{ commit }}" # Use the global variables in the command

[tasks]
hello = "echo hello {{ hello }}"
# Use the jinja function to check if the platform variable contains linux
hello-you = "echo hello {{ 'tux' if target.contains('linux') else 'you' }}"

Questions

• Should we use {{ }} or ${{ }} for the syntax?

I am not sure about that one. "*" and "+" feel a bit implicit compared to how explicit the other suggestions are.

For me I actually like them :)

@Hofer-Julian to respond to the task dependency environment selection

I see two options, verbose or syntax.

• Verbose, clear and simple:

[feature.c.tasks]
something = { depends_on = [{ task = "test", environment = "a" }] }

• Custom syntax, short and easy on the eyes (IMO)

[feature.c.tasks]
# The dot works as it's not correct toml to name the task `a.test = "echo test"`
something = { depends_on = ["a.test"] } 

@ruben-arts personally, I'd prefer the more verbose one. For the other, one it isn't immediately obvious what it does, and the use case isn't common enough that everyone will just get used to it.