Skip to content

Publish a workflow

Prepare a CWL application package and make it available for execution through AntFlow. AntFlow discovers process definitions from GitLab; you do not need to register each process manually.

For a complete beginner example, start with Create your first workflow.

Before you start

You need:

  • A CWL workflow and all files it references.
  • A GitLab project you can push to on the instance used by your platform.
  • Runtime images, or a container build environment and a registry where you can publish them.
  • Access to AntFlow and a launcher suitable for the workflow.
  • cwltool for local validation and testing.

Confirm the processing topic and repository-access requirements with your platform administrator. The Helm chart defaults to the topic antflow, but your instance may use a different topic or no topic filter.

1. Assemble and validate the package

Keep the workflow, referenced CWL tools, and supporting files together in a local repository. Follow the application-package requirements. The CWL must include a class: Workflow definition for AntFlow to discover the process.

Use container image references for the workflow steps. Include Dockerfiles and build sources for images you maintain; these are not needed when you use existing published images.

Add a README.md describing the process, its inputs, and its outputs. It helps users understand the project but is not required for workflow discovery.

For richer execution forms, add CWL input-form annotations. You can also add a project preview image to the README.

Validate the workflow from the package directory:

cwltool --validate workflow.cwl

Replace workflow.cwl with your workflow's path. For a packed CWL document, include the workflow identifier, such as workflow.cwl#main, where required. Fix validation errors before proceeding. Check YAML syntax, the workflow definition, and references to other package files.

Validation is not an execution test

This checks the workflow definition, not whether its commands, images, data access, or outputs work. Rerun validation after changing the CWL.

2. Prepare the runtime images

Skip building images if your workflow already references suitable published images. Otherwise, build each step's image with its scripts, binaries, and dependencies inside it. The CWL command must use their installed paths rather than assume that files from your development machine will be mounted at runtime.

For an image you maintain, choose its full registry/repository reference and version tag once. These interactive commands use Bash; keep using the same terminal for the build and push:

read -r -p "Full container image reference, including version tag: " IMAGE
docker build --tag "$IMAGE" .

Run the image's processing command with representative inputs and inspect its outputs. When all runtime images are available locally, test the complete workflow:

cwltool --outdir outputs workflow.cwl inputs.yaml

Prepare inputs.yaml with values for your workflow and use the same workflow path or identifier you validated. This command executes the workflow; it may consume resources and contact services declared by the workflow.

Authenticate to your registry using its supported login procedure. Push images that passed your tests to that registry:

docker push "$IMAGE"

Reference the published image in the corresponding CWL tool:

The following is a YAML template, not a shell command. Replace its example dockerPull value with the exact reference you entered for IMAGE; CWL does not expand shell variables.

requirements:
  DockerRequirement:
    dockerPull: registry.example.com/my-team/my-workflow:1.0.0

Use versioned image references rather than overwriting a tag for an already published workflow. Confirm private-registry credentials with the administrator; do not put them in the CWL or commit them to Git.

3. Check the execution environment

Choose an available launcher with the resources and connectivity your workflow needs. A successful local test does not establish that it can run on another environment.

  • Kubernetes: confirm that the cluster can pull the images, including any private-registry authentication, and that the workflow's resource requirements fit the available nodes.
  • HPC with Singularity: coordinate image conversion and provisioning with the administrator. Ask for the required SIF filenames and image-cache location, and test the workflow on the target system with cwltool --singularity.
  • Local execution: confirm that the host has the required container runtime, storage, and resources.

For infrastructure setup, use the launcher guides. Authors using HPC should also review the Toil / Slurm guide.

Check data and service access from the execution environment separately from the launcher's connection to AntFlow. Workflow steps must reach their input data and any external services they use. For an offline HPC environment, preprovisioning images does not make remote input data available.

4. Push the package to GitLab

Create a GitLab project for the package, or use an existing one. If the local package directory is not already a Git repository, initialize it and add your project's clone URL from GitLab. This changes the local repository; the push below publishes its committed files:

git init -b main
read -r -p "GitLab project clone URL (SSH or HTTPS, without credentials): " GIT_REMOTE
git remote add origin "$GIT_REMOTE"

Review the files to ensure they contain no credentials or generated test data, then commit and push the package:

git add .
git commit -m "Publish application package"
git push -u origin main

Use your repository's branch name if it is not main. Push all referenced package files, not just the top-level CWL document.

5. Make the workflow discoverable

  1. In GitLab, open the project's Settings > General > Topics.
  2. Add the processing topic configured for your AntFlow instance, if it uses a topic filter. Matching is case-sensitive.
  3. Save the settings, sign in to AntFlow, and open Projects.
  4. Select the project and the branch, tag, or commit you pushed.
  5. Confirm that the process description lists the expected inputs and outputs.

If the project is missing, check its topic and the GitLab permissions of the token used by AntFlow. If the project appears but the process does not, check that the selected Git reference contains the validated CWL and its dependencies. See project discovery and filters.

Give other users access

For collaborators using their own GitLab access, share the project with them or their group using the role from the permission table. Your administrator can confirm which group to use.

For access through the server credential, ask an AntFlow administrator to publish the project. Check Published project access and Token selection before choosing that access model.

If the workflow needs project configuration or personal credentials, use Environment variables or CWL secret inputs. Keep secret values out of the package and reusable input files.

Test through AntFlow

Check result publication

To run without automatic result uploads, declare outputs: {} on the workflow; see Workflows without declared outputs. Job-log uploads are controlled separately by the launcher.

S3 output uploads and STAC cataloging are separate operations. Ask your administrator which STAC targets the instance enables before sharing a workflow.

To keep output uploads while opting out of remote STAC API publication, use the workflow's publish:catalog property. Do not disable publish:enabled for this purpose: that controls the entire automatic publication step, including output archiving. See Workflow publication properties for the syntax, defaults, and administrator-enabled static-catalog behavior.

Run a representative job

Follow Run a workflow with representative inputs. Inspect the job's logs and results before sharing the workflow with other users. Test on each execution environment you intend to support.