SLURM
=====

.. note::

    This is a quick reference. For a full
    tutorial, :doc:`click here. <../tutorials/slurm>`


Step 1: Add target environment
------------------------------

.. tip::

    To get a sample pipeline to try this out,
    `see this. <https://docs.ploomber.io/en/latest/user-guide/templates.html#downloading-a-template>`_

.. code-block:: sh

    # add a target environment named 'slurm'
    soopervisor add slurm --backend slurm

The command above will generate an entry named ``slurm`` in the
``soopervisor.yaml`` file, and a ``slurm/template.sh`` file, you can use
the latter to customize how Soopervisor executes the tasks in your pipeline.
Under the hood, Soopervisor executes a ``sbatch job.sh`` command for each task
in your pipeline, where ``job.sh`` is generated by using ``template.sh`` as a template.
For more information and customization options, :doc:`click here. <../api/slurm>`


Step 2: Submit jobs
-------------------

To submit the jobs to SLURM:

.. code-block:: sh

    # submit pipeline to the cluster
    soopervisor export slurm --skip-tests --ignore-git

Note that the command above will only export outdated tasks (the ones whose
source code has changed since the last execution), to force exporting all
tasks:

.. code-block:: sh

    # force exporting all tasks regardless of status
    soopervisor export slurm --skip-tests --ignore-git --mode force

.. important::

    For your pipeline to run successfully, tasks must write their outputs to a
    common location. You can do this either by
    creating a shared disk or by adding a storage client.
    :doc:`Click here to learn more. <../user-guide/task-comm>`