Handling Multiple Tasks

Create the following two files in the main project directory:

many_tasks.pysubmit.sh

from sdag import pipeline


@pipeline
def many_tasks():
    say_hello()
    say_hello()
    say_hello()


@many_tasks.task("submit.sh")
def say_hello():
    print("hello world!")

sdag-execute

You can view the graph of this pipeline in the terminal via sdag view:

sdag view many_tasks

which should print something like:

                    ╭──────╮
                    │_root_│
                    ╰──────╯
                        │
                        │
     ┌──────────────────└──────────────────┐
     ▼                  ▼                  ▼
┌─────────┐        ┌─────────┐        ┌─────────┐
│say_hello│        │say_hello│        │say_hello│
└─────────┘        └─────────┘        └─────────┘

As these tasks do not depend on each other, they will be scheduled at the same time. You can confirm this by running the pipeline:

sdag run many_tasks --local

You can use positional arguments to signal graph edges. Let's add a second pipeline to the same many_tasks.py module:

import asyncio

@pipeline
def dependent_tasks():
    t1 = first()
    t2 = first()
    second(t1, t2)


@dependent_tasks.task("submit.sh")
def first():
    print("I run first!")


@dependent_tasks.task("submit.sh")
async def second():
    await asyncio.sleep(1)
    print("I run second!")

Notice async tasks are supported: in such cases sdag will start the event loop for you. sdag view dependent_tasks will now print:

       ╭──────╮
       │_root_│
       ╰──────╯
           │
           │
   ┌───────└──────┐
   ▼              ▼
┌─────┐        ┌─────┐
│first│        │first│
└─────┘        └─────┘
   │              │
   │              │
   └───────┌──────┘
           ▼
       ┌──────┐
       │second│
       └──────┘

Which means that second will only run if both first tasks successfully complete. This example can be run via:

sdag run dependent_tasks --local

Info

To run these examples with Slurm, turn submit.sh into a sbatch script as explained in the Hello World section and execute pipelines without the --local option.