Toil / Slurm launcher¶

The toil_slurm runner executes CWL workflows on HPC infrastructure. The launcher starts a Toil leader on the submission host. Toil submits work to Slurm, and compute nodes run workflow tools with Singularity.

flowchart LR
    API[AntFlow Server API] <--> Launcher[AntFlow Launcher]
    Launcher --> Toil[Toil leader on submission host]
    Toil --> Slurm[Slurm scheduler]
    Slurm --> Workers[Compute nodes with Singularity]
    Toil <--> Storage[Shared working files and job store]
    Workers <--> Storage

Requirements¶

A launcher registered with AntFlow Server.
A Slurm submission host with Python 3.12+, scheduler commands, and permission to submit jobs under your site's account, partitions, and QOS policies.
A Python environment containing the launcher, its bundled antflow-cwltool, and Toil with CWL support. The launcher repository installs toil[cwl]~=9.5.
That Python environment accessible at the same absolute path on compute nodes.
Shared writable working storage for the workflow files and Toil job store.
Writable Toil log directories, Singularity cache, and temporary storage.
Network access from the launcher to the Server API and from workflow tools to their inputs, registries, and publication destinations.

Modules must work in a noninteractive shell

The runner loads singularity (or singularity/<version>) and node-js through the site's module system. These modules and the module command must work in a noninteractive bash -c subprocess. An interactive login shell alone is not sufficient.

The standard Kubernetes Helm deployment does not provision Slurm, Singularity, or site modules. Deploy this launcher on the submission host using your HPC environment's service-management tooling.

Install on the submission host¶

Select a compatible launcher version and check out that release from the launcher repository. From that checkout, create a Python environment on shared storage and install the runtime dependencies:

python3 -m venv /shared/antflow/venv
source /shared/antflow/venv/bin/activate
make install

Replace /shared/antflow with your site's shared path. The repository's install target includes the bundled CWL runner and Toil dependency.

Configure the launcher¶

Create a site-specific settings file. This example uses an hpc profile and illustrative paths, partitions, and QOS values:

Use Access the API to identify the base URL of the instance where you registered the launcher. Replace [hpc.api].url with that full literal URL. This is a TOML template; shell variables such as $API_URL are not expanded here.

[default]
activate_execution_log = true
enable_retry = true
retry_ttl = 86400
log_level = "INFO"
loop_delay = 5
timeout = 60
publish_report_key = "results"
keep_working_folder = false
archive = true
default_container = "debian:bookworm-slim"
javascript_eval_timeout = 60
cwltool_extra_args = []

[default.concurrent]
max_jobs_running = 2
max_workers = 10

[default.http_server]
host = "127.0.0.1"
port = 8001
webhook_url = "http://127.0.0.1:8001/webhooks/"

[hpc]
cwl_runner = "toil_slurm"
supported_flavors = ["hpc"]
working_dir = "/shared/antflow/launcher-a/work"

[hpc.api]
url = "https://antflow.example.com/api"

[hpc.client.slurm]
account = "project-account"
time_limit = "02:00:00"
partition = "cpu"
qos = "normal"
gpu_partition = "gpu"
gpu_qos = "gpu"

[hpc.client.singularity]
version = "<site-singularity-version>"
config_dir = "/shared/antflow/launcher-a/singularity-config"
cache_dir = "/shared/antflow/launcher-a/singularity-cache"
tmp_dir = "/shared/antflow/launcher-a/singularity-tmp"
bind = []

[hpc.runner.toil_slurm]
env_dir = "/shared/antflow/venv"
log_dir = "/shared/antflow/launcher-a/toil-logs"
tmp_dir = "/tmp"
retry_count = 0

Replace all site-specific values before starting the launcher. All six fields under client.slurm, including the GPU fields, are read by the current command builder. Use values valid for your cluster.

If your site uses the unversioned singularity module, omit client.singularity.version. Do not assume an Apptainer-only module is a drop-in replacement: the current runner explicitly loads Singularity.

Storage and environment¶

Setting	Purpose
`working_dir`	Shared root for per-job inputs, outputs, and the persistent Toil job store.
`runner.toil_slurm.env_dir`	Python environment exported through batch-job `VIRTUAL_ENV` and `PATH`.
`runner.toil_slurm.log_dir`	Required root for Toil execution and Slurm batch logs.
`runner.toil_slurm.tmp_dir`	Toil `--workDir`; node-local scratch is suitable if available and writable on every execution node.
`client.singularity.config_dir`	Singularity configuration directory.
`client.singularity.cache_dir`	Shared or site-managed image cache, used by CWL and Singularity.
`client.singularity.tmp_dir`	Singularity temporary directory.
`client.singularity.bind`	Additional bind specifications for datasets, such as `/datasets:/datasets:ro`.

Create the configured directories and ensure the submission user and compute jobs can access them. Paths shared between the submission host and compute nodes must have consistent absolute names.

An optional root log_dir controls launcher service log files. It does not replace the required runner.toil_slurm.log_dir.

Start the launcher¶

Supply APP_TOKEN securely in the service environment. Activate the Python environment and select the settings file and profile:

source /shared/antflow/venv/bin/activate
export SETTINGS_FILES_FOR_DYNACONF=/shared/antflow/launcher-a/settings.toml
export ENV_FOR_DYNACONF=hpc
antflow-launcher run

Activating the environment makes toil-cwl-runner available to the launcher. runner.toil_slurm.env_dir configures batch-job environment variables; it does not activate the launcher's own environment.

Keep the launcher running with your site's service supervisor.

Restarting does not reconnect to the Toil leader

The local Toil leader is tied to the launcher process. Restarting the launcher does not automatically reconnect to the previous leader.

Execution, retries, and logs¶

The launcher stages each job and starts toil-cwl-runner with the Slurm batch system and Singularity enabled. Per-job state includes:

toil/jobstore: persistent Toil state for restart.
toil/outputs: workflow outputs.
tmp and tmp-out: CWL temporary directories.
<runner.toil_slurm.log_dir>/<job-id>: Toil execution logs.
<runner.toil_slurm.log_dir>/<job-id>/slurm: Slurm batch logs.

An existing job store causes the runner to request Toil restart. Launcher enable_retry and retry_ttl control retained failed workflow state; runner.toil_slurm.retry_count separately controls Toil's internal job retries.

client.slurm.time_limit limits Slurm batch-job wall time. The root timeout setting controls Server API request timeouts, not workflow duration.

Inspect queue and resource allocation using your site's Slurm tools. Enter a Slurm job ID from the queue or execution logs, not an AntFlow job ID. The prompt uses Bash:

squeue --user "$USER"
read -r -p "Slurm job ID: " SLURM_JOB_ID
sacct --jobs "$SLURM_JOB_ID"

Detailed Toil and Slurm logs remain in the configured log directory. The launcher uploads captured runner stderr, not all of these log files; manage their retention separately.

Check Slurm jobs after cancellation

Dismissing an execution terminates the local runner process. The launcher does not explicitly issue scancel for all submitted jobs. After dismissal or a launcher shutdown, check the queue and cancel remaining jobs according to your site's procedure.

Troubleshooting¶

Module not found: verify Singularity and node-js module availability in the noninteractive shell used by the runner.
toil-cwl-runner not found: activate the environment before starting the launcher and verify its path is accessible on compute nodes.
Submission rejected: check the configured account, partitions, QOS, and time limit.
Files unavailable on compute nodes: check shared paths, ownership, binds, and node-local scratch availability.
Image conversion or cache failures: check Singularity storage capacity, directory permissions, registry access, and certificate trust.

Source configuration¶

This backend is documented from the launcher repository's HPC profile and Toil/Slurm runner.

What's next¶

Run a workflow: submit a job to this launcher and inspect its logs and results.
Execution policy: configure launcher eligibility and the server's job-retry policy.