Skip to content

Structuring Larger Projects

When working with larger projects made of many pipelines, it is convenient to set the directory sdag searches pipelines into explicitly. Both flat and src layouts are supported. The only important point is to add __init__.py files to all subfolders in the tree to let sdag treat them as regular, importable packages.

This examples takes advantage of the hello.py module created in the first example:

from sdag import pipeline


@pipeline
def hello_world():
    say_hello(name="sdag")


@hello_world.task("submit.sh")
def say_hello(name: str):
    print(f"hello from {name}")

#!/bin/bash
#SBATCH --account=<account-name>
#SBATCH --partition=<partition-name>
#SBATCH --qos=<qos-name>
#SBATCH --nodes=1
#SBATCH --tasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --time=00:00:30

sdag-execute

Flat layout

Create a <my-package>/workflows directory and an empty __init__.py module inside it. Now copy the hello.py module from the previous example in the workflows/ folder and the submit.sh script in the main project one. Finally, copy the following snippet inside the pyproject.toml file:

pyproject.toml
[tool.sdag]
dag-dir = "<my_package>/workflows"

Your project tree should look like this:

flat layout
<my-package>
├── pyproject.toml
├── submit.sh      <- Execution script
└── <my_package>
    ├── __init__.py
    └── workflows
        ├── __init__.py    <- Don't forget to add this
        └── hello.py       <- hello_world pipeline

src layout

Create a src/<my-package>/workflows directory and an empty __init__.py module inside it. Now copy the hello.py module from the previous example in the workflows/ folder and the submit.sh script in the main project one. Finally, copy the following snippet inside the pyproject.toml file:

pyproject.toml
[tool.sdag]
dag-dir = "src/<my_package>/workflows"

Your project tree should look like this:

src layout
<my-package>
├── pyproject.toml
├── submit.sh      <- Execution script
└── src
    └── <my_package>
        ├── __init__.py
        └── workflows
            ├── __init__.py    <- Don't forget to add this
            └── hello.py       <- hello_world pipeline

Importing and running pipelines

Now that everything is set up, you can list the available pipelines:

sdag list

you should see the hello_world pipeline available. You can run the pipeline through:

sdag run hello_world --local

If you don't like to specify --local every time, add the following setting to the pyproject.toml:

pyproject.toml
[tool.sdag]
dag-dir = "<flat-or-src-layout>/workflows"
cmd = "bash"

So that every task will be executed locally with bash.