SLURM¶
Note
This is a quick reference. For a full tutorial, click here.
Step 1: Add target environment¶
Tip
To get a sample pipeline to try this out, see this.
# add a target environment named 'slurm'
soopervisor add slurm --backend slurm
The command above will generate an entry named slurm
in the
soopervisor.yaml
file, and a slurm/template.sh
file, you can use
the latter to customize how Soopervisor executes the tasks in your pipeline.
Under the hood, Soopervisor executes a sbatch job.sh
command for each task
in your pipeline, where job.sh
is generated by using template.sh
as a template.
For more information and customization options, click here.
Step 2: Submit jobs¶
To submit the jobs to SLURM:
# submit pipeline to the cluster
soopervisor export slurm --skip-tests --ignore-git
Note that the command above will only export outdated tasks (the ones whose source code has changed since the last execution), to force exporting all tasks:
# force exporting all tasks regardless of status
soopervisor export slurm --skip-tests --ignore-git --mode force
Important
For your pipeline to run successfully, tasks must write their outputs to a common location. You can do this either by creating a shared disk or by adding a storage client. Click here to learn more.