SLURM

Note

This is a quick reference. For a full tutorial, click here.

Step 1: Add target environment

Tip

To get a sample pipeline to try this out, see this.

# add a target environment named 'slurm'
soopervisor add slurm --backend slurm

The command above will generate an entry named slurm in the soopervisor.yaml file, and a slurm/template.sh file, you can use the latter to customize how Soopervisor executes the tasks in your pipeline. Under the hood, Soopervisor executes a sbatch job.sh command for each task in your pipeline, where job.sh is generated by using template.sh as a template. For more information and customization options, click here.

Step 2: Submit jobs

To submit the jobs to SLURM:

# submit pipeline to the cluster
soopervisor export slurm --skip-tests --ignore-git

Note that the command above will only export outdated tasks (the ones whose source code has changed since the last execution), to force exporting all tasks:

# force exporting all tasks regardless of status
soopervisor export slurm --skip-tests --ignore-git --mode force

Important

For your pipeline to run successfully, tasks must write their outputs to a common location. You can do this either by creating a shared disk or by adding a storage client. Click here to learn more.