Tags

Tags can be used to specify custom configurations for groups of tasks. Tags can be added to tasks via the tags argument of the @taskor @<pipeline-name>.task decorator:

from sdag import task

@task("submit.sh", tags=["online", "small", "custom"])
def some_task(): ...

The corresponding configurations be either set in the pyproject.toml under [[tools.sdag.tags]] or in the .sdag.toml file under [[tags]]:

pyproject.toml.sdag.toml

[[tool.sdag.tags]]
tag = "online"
cmd = "bash"

[[tool.sdag.tags]]
tag = "small"
cpus-per-task = 1

[[tool.sdag.tags]]
tag = "custom"
script-path = "a/custom/script.sh"
envs = {"ENV_NAME" = "value"}

[[tags]]
tag = "online"
cmd = "bash"

[[tags]]
tag = "small"
cpus-per-task = 1

[[tags]]
tag = "custom"
script-path = "a/custom/script.sh"
envs = {"ENV_NAME" = "value"}

Tags have higher priority than pyproject.toml/.sdag.toml global configurations. Whenever a task is assigned more than one tag, tag configurations are applied in the same order they are provided. Tags can be used to customize the behavior of groups of tasks. For example, some supercomputers have dedicated queues for jobs requiring internet access while others only allow outbound connections on login nodes. Tags can be used to enforce such policies. Moreover, Tags specified in the .sdag.toml file override those in the pyproject.toml. So, different .sdag.toml files can be used to make pipelines run transparently across different machines without changing the source code. Tags currently allow one to customize: - cmd (bash or sbatch) - script-path: Used to change the execution script for selected tasks. - envs: Used to set additional environment variables - All Slurm settings available in pyproject.toml.