Technical analysis for radio astronomers and SKA/SRCNet engineers Date: May 2026
Rapthor is the LOFAR direction-dependent effects (DDE) calibration and imaging pipeline, developed at ASTRON and adopted as the basis of the SKAO ICAL (Initial Calibration) pipeline. It automates the iterative self-calibration loop:
[Input MSes] → calibrate (DP3/DDECal) → image (WSClean+IDG) → extract sources → loop
Each pipeline operation (calibrate, image, subtract, mosaic, source-find) is implemented as a CWL (Common Workflow Language) workflow. Toil acts as the CWL execution engine and provides the pluggable batch-system layer. The supported backends are:
--batchSystem=single_machine— local development/testing--batchSystem=slurm— HPC cluster, Toil leader submits Slurm jobs--batchSystem=kubernetes— cloud/containerised clusters
Toil owns the full job DAG: it parses CWL, resolves step dependencies, marshals intermediate data, and issues individual compute tasks to the chosen batch backend. Rapthor's Python orchestration code drives the outer self-calibration loop, calling Toil repeatedly per operation.
Compute profile of Rapthor's job types:
| Operation step | Primary tool | Resource profile |
|---|---|---|
| Direction-dependent calibration | DP3 / DDECal | CPU-heavy (many cores), high RAM (tens of GB per facet), I/O-heavy (MS read) |
| Wide-field imaging | WSClean + IDG | CPU or GPU (IDG-GPU mode); very high RAM (100+ GB for large fields); I/O-heavy |
| Beam application / subtraction | DP3 | CPU-moderate, I/O-heavy |
| Source finding | PyBDSF / similar | CPU-light to moderate, memory-moderate |
| Mosaicking | custom | I/O-dominated, CPU-light |
The mix is therefore heterogeneous: some steps want GPUs (WSClean+IDG), most want many CPU cores and a lot of RAM, and several are gated by I/O bandwidth from measurement sets that can be hundreds of gigabytes per pointing.
PanDA (Production and Distributed Analysis) was created for ATLAS/LHC but is now general-purpose. Its canonical deployment for the SKA ecosystem uses the CERN-hosted server (pandaserver-doma.cern.ch) with per-site Harvester edge agents. For SRCNet, deploying Harvester as an optional per-node service is already documented in the SRCNet Site Operator manual.
Core PanDA components relevant here:
| Component | Role |
|---|---|
| PanDA Server | Central RESTful service; holds the global job queue, all job metadata, status, and dispatch logic |
| JEDI | Job Execution & Definition Interface — the optimisation/brokerage engine that transforms abstract tasks into concrete jobs and assigns them to sites |
| Harvester | Per-site edge agent; submits pilot jobs to local batch systems (Slurm, Kubernetes, HTCondor, ARC CE, etc.); mediates communication when outbound network is restricted |
| Pilot | Transient agent that runs on worker nodes; pulls job payloads from PanDA and executes them; reports heartbeats and final status |
| iDDS | Intelligent Data Delivery Service — high-level workflow DAG engine sitting on top of PanDA; handles task inter-dependencies, conditional branching, active learning loops |
| Rucio | Data management (not strictly PanDA, but tightly integrated); tracks replicas across RSEs (Rucio Storage Elements), triggers transfers, exposes locality information to JEDI's brokerage |
Late binding is the defining architectural property of PanDA's pilot system. Understanding it precisely is essential before mapping Rapthor onto it.
In early binding (classical batch submission), a job is submitted to a specific queue at a specific site. If that site is congested, the job waits. If the site fails, the job is lost. The job is irrevocably coupled to a slot at submission time.
In PanDA's late binding, the work and the resource travel independently and are married only at the last possible moment — when a live pilot on a worker node requests a job.
The canonical PanDA pilot flow is:
1. JEDI/brokerage assigns a set of jobs to a site (based on data locality, load, etc.)
2. Harvester at that site submits N pilot jobs to the local scheduler (Slurm, K8s, etc.)
— these pilots carry NO payload yet; they just request a slot
3. The local scheduler allocates compute resources and the pilot starts executing
4. The pilot contacts the PanDA server: "I am alive at site X with these specs,
give me a job"
5. PanDA server dispatches a queued, site-appropriate job to this pilot
6. The pilot downloads the job definition, runs the payload, reports results
7. The pilot optionally requests another job (worker reuse) or exits
This is the 1-to-1 pull model. The binding between job and compute slot happens at step 4, after the slot has been acquired — hence "late binding."
Harvester supports several binding patterns, each suited to different resource topologies:
1-to-1 pull (classic pilot): Workers start before jobs. Each worker pulls one job on startup. No prefetch. Heartbeats sent directly by pilot to PanDA. Best for grid/cloud resources with fast queue turnaround.
1-to-1 push with late binding (most relevant for HPC): Jobs are prefetched and queued in Harvester's local database while workers are simultaneously submitted to the batch scheduler. When a worker slot becomes available, Harvester gives it exactly one pre-fetched job. This handles HPC sites with long batch queues — Harvester optimistically prefetches jobs during the wait, so there is no additional latency when the nodes finally start. For Slurm-backed SRCNet nodes with multi-hour queue waits, this is the key mode.
1-to-many (MPI jobs): One PanDA job maps to N workers (N MPI ranks). Each worker makes a subdirectory $ACCESS_POINT/$PandaID; Harvester combines all workers' reports into one heartbeat. Used for jobs that require tightly-coupled multi-node communication.
Many-to-1 (job packing): Multiple small PanDA jobs are bundled into one worker. Harvester dynamically determines job count based on available free slots at the site. Useful for embarrassingly parallel steps where individual jobs are short.
Many-to-1 with backfill: Worker maker queries the local batch system in real time for idle slots, then packs enough jobs to fill them. Ideal for opportunistic resource use.
JEDI's brokerage assigns jobs to sites considering, in rough priority order:
- Hard constraints — does the site have the required software / container image? Does it have GPU nodes if the job requires GPU?
- Data locality — Rucio is queried: where are the input replicas? Sites with local replicas are strongly preferred; if no local copy exists, the cost of transferring data before execution is estimated.
- Resource matching — number of cores requested, RAM per core, walltime limits, scratch space
- Load balancing — current queue depth, number of running/pending jobs, global share allocations
- Transfer backlogs — if the network path to a site is congested, its score is penalised
The brokerage has a plugin architecture so organisations can extend or replace it with domain-specific logic. For SRCNet this is important: SKA's data placement and fair-share policies differ substantially from ATLAS's.
In a federated system like SRCNet — where compute nodes are at different institutions across multiple countries with heterogeneous network links, storage backends, and queue policies — late binding provides:
- Resilience: if a site becomes unavailable after job submission, the pilot simply fails and a new one can be submitted to a different site; the job itself is never lost
- Opportunism: idle capacity at any site is automatically exploited without resubmitting jobs
- Data-aware placement: at dispatch time, Rucio can confirm which sites actually have the data right now (replication may still be in progress), avoiding premature binding to a site that doesn't yet have the input
- Resource-type routing: GPU vs. CPU decisions can be deferred until a suitably-equipped pilot checks in, rather than pre-selecting resource types at workflow compile time
| Rapthor / Toil concept | PanDA / iDDS concept |
|---|---|
CWL Workflow |
iDDS DAG / pchain workflow |
CWL CommandLineTool step |
PanDA Task (a class of jobs) |
Individual Toil BatchJob (one scatter element) |
PanDA Job (one unit of execution) |
| Toil batch backend (Slurm/K8s) | Harvester + local scheduler |
| Toil job store (file-based checkpoint) | PanDA Server + JEDI database |
| Toil leader process (DAG orchestration) | iDDS workflow engine |
Toil --batchSystem plugin |
Harvester plugin (WorkerMaker + Messenger) |
CWL ResourceRequirement hints |
PanDA job resource spec (coreCount, ramCount, diskCount, gpuSpec) |
| Toil scatter (parallel over facets) | PanDA task with N input files → N parallel jobs |
Rapthor's outer loop (calibrate → image → extract → re-calibrate) is currently driven by Python code in Rapthor's Pipeline class. In PanDA+iDDS terms, this maps naturally to a conditional workflow — iDDS supports loops and junction tasks that inspect upstream output and decide whether to re-enter the loop. This is an exact match conceptually, but requires porting the convergence logic from Rapthor's Python into an iDDS junction task.
Rapthor processes each facet or calibration direction independently. In CWL this is expressed as a scatter over a list of direction items. In PanDA, this is the native model: one task produces N jobs, one per input item. The brokerage then places each job independently at the best-available site. This is where late binding pays off most — a 100-direction scatter creates 100 PanDA jobs, each of which can run on whichever site happens to have a free pilot and a local data replica.
Architecture:
Rapthor (Python orchestrator)
└── Toil CWL runner (leader process, DAG management)
└── [NEW] PanDABatchSystem (AbstractBatchSystem subclass)
└── PanDA Server REST API
└── JEDI brokerage → Harvester → Slurm/K8s at SRC nodes
How it works:
Toil's AbstractBatchSystem API requires implementing these key methods:
class PanDABatchSystem(AbstractBatchSystem):
def issueBatchJobs(self, job_descs):
# Submit each job_desc to PanDA via pandaclient REST API
# Map Toil ResourceRequirement → PanDA jobSpec (coreCount, ramCount, etc.)
# Return list of PanDA jobIDs
def getRunningBatchJobIDs(self):
# Poll PanDA server for status of submitted jobs
# Return {jobID: elapsed_time} for running jobs
def getUpdatedBatchJob(self, maxWait):
# Block until a job completes; return (jobID, exit_code, walltime)
def killBatchJobs(self, jobIDs):
# Send kill command to PanDA server
def getSchedulingStatusMessage(self):
# Return human-readable queue status for loggingThe Toil leader orchestrates the CWL DAG exactly as it does today. When a step is ready to execute, Toil calls issueBatchJobs(), which translates the Toil job description into a PanDA job specification and posts it to the PanDA server. PanDA then handles site selection, data staging, pilot submission, and execution. Toil polls for completion and proceeds to the next step.
Data staging caveat: Toil normally manages its own job store (a shared filesystem path or S3 bucket) for intermediate files. With PanDA distributing jobs across multiple sites, the job store must be accessible from all sites — either a globally-replicated object store, or Rucio-managed datasets that PanDA pre-stages. This is the primary engineering challenge.
Pros:
- Minimal changes to Rapthor's codebase (only a new
--batchSystem=pandaplugin) - Preserves Toil's battle-tested CWL DAG management and checkpoint/restart
- The outer self-calibration loop stays in Python/Rapthor
- Developers can test incrementally with existing CWL workflows
- Rapthor users get
--batchSystem=pandaalongside existingslurm/kubernetesoptions
Cons:
- The Toil leader process must remain alive for the duration of the entire pipeline run (potentially days)
- Toil's job store and intermediate file paths must be globally accessible across SRC sites — this may require re-engineering the file staging
- PanDA's brokerage sees only individual Toil jobs, not the full Rapthor DAG, limiting its ability to co-schedule related steps near the same data
- No access to PanDA's higher-level features (iDDS loops, multi-site DAG scheduling, advanced monitoring via BigPanDA)
Verdict: This is the pragmatic near-term path. It delivers late-binding benefits for individual job placement while preserving Rapthor's architecture. The TES (GA4GH Task Execution Service) batch system plugin (toil-batch-system-tes) is an existence proof that third-party Toil backends work in production; a similar pattern for PanDA is well-defined.
Architecture:
Rapthor (Python orchestrator, refactored)
└── pchain / pandaclient API
└── PanDA Server + iDDS workflow engine
├── JEDI task scheduling
├── Harvester → Slurm/K8s at SRC nodes
└── Rucio data placement
How it works:
The Rapthor CWL files are submitted directly to PanDA via pchain:
pchain --cwl rapthor_calibrate.cwl --yaml inputs.yaml \
--outDS user.rapthor.run1234 \
--site SRCNET_UK,SRCNET_NL,SRCNET_AUOn the server side, iDDS parses the CWL, builds a DAG of PanDA tasks, and releases jobs as upstream dependencies complete. The self-calibration loop would be expressed using iDDS's junction/loop primitives.
Key iDDS features relevant to Rapthor:
- Data-aware job release: iDDS monitors upstream task outputs via Rucio events and releases downstream jobs only when their inputs are confirmed available
- Conditional branching: Junction tasks inspect convergence metrics and re-enter the calibration loop or terminate
- Incremental data production: iDDS supports releasing child jobs as soon as some (not all) parent outputs are ready — critical for Rapthor's scatter steps where facets complete at different rates
- Multi-site DAG: individual tasks within the same workflow can run at different SRC sites, with data replication managed by Rucio between tasks
Pros:
- Full PanDA ecosystem benefits: BigPanDA monitoring dashboard, retry logic, multi-site data locality, GPU routing
- No persistent leader process required — PanDA server holds all state
- The self-calibration loop is a first-class workflow primitive, not ad hoc Python
- Natural fit for SRCNet's PanDA+Rucio deployment model
- Better fault tolerance: any task can be retried at any site independently
Cons:
- Significant refactoring of Rapthor required; the outer loop logic must be ported to iDDS junction tasks
- PanDA's
pchain/CWL support was developed for ATLAS workflows and has HEP-centric assumptions (dataset naming conventions, ATLAS-specific resource tags); radio astronomy workflows will need adaptation - Rapthor uses many domain-specific container images with complex DP3/WSClean dependencies; ensuring these are staged at every SRC site adds operational overhead
- Loss of Toil-specific features (e.g., Toil's checkpoint restart from mid-workflow, which is currently used when a run is interrupted)
- Higher development cost
Verdict: The right long-term target for Rapthor as a production SRCNet workload. The architectural match is excellent — Rapthor's scatter-over-facets pattern is precisely what PanDA was designed for at scale. But this is a significant engineering project requiring close collaboration between the Rapthor/SKAO pipeline team and the PanDA/DOMA community.
Architecture:
Toil (--batchSystem=tes)
└── GA4GH TES API endpoint
└── PanDA TES gateway (to be developed)
└── PanDA Server → Harvester → ...
The toil-batch-system-tes plugin already exists, implementing Toil's batch API against the GA4GH Task Execution Service standard. If PanDA exposed a TES-compliant API, this would provide an off-the-shelf bridge. The PanDA community has been aware of TES compatibility, though a production-ready gateway does not yet exist. This option is worth monitoring but is not actionable today.
Rapthor's WSClean imaging step can run in three modes: CPU-only, idg (CPU Image Domain Gridder), or idg-gpu (GPU-accelerated). Currently, the user must select the mode at pipeline configuration time and ensure the job runs on an appropriate node type. With PanDA late binding:
- The job is submitted with a resource requirement tag (
requiresGPU=TrueorrequiresGPU=False) - PanDA's brokerage routes the job to whichever pilot has the right hardware
- If GPU nodes are free, GPU-mode WSClean jobs run there; if the GPU queue is congested, CPU fallback can be brokered automatically
- This dynamic routing is impossible with static Slurm job submissions
LOFAR/SKA observations are large (hundreds of GB to TB per pointing). Rucio manages replicas across SRC nodes. PanDA's brokerage consults Rucio before assigning jobs:
- A calibration job for direction d will be placed at the SRC node that already holds the input measurement set
- If two SRC nodes both have the data, the one with shorter queue depth is chosen
- If no local copy exists, Rucio initiates a transfer and PanDA waits for transfer completion before dispatching the job — eliminating the case where a job starts and cannot find its input
For Rapthor, this means the large intermediate MS products (post-subtraction visibilities) only need to live at one site per iteration; PanDA will find them.
SRCNet nodes are expected to run on top of HPC clusters (SLURM-managed). HPC queue wait times can be hours. PanDA's push-with-late-binding mode allows Harvester to prefetch Rapthor jobs during the queue wait, so execution begins immediately when nodes become available. For a 100-direction scatter, 100 PanDA jobs are waiting in Harvester's local queue; as soon as the Slurm allocation fires, they flood onto the nodes with no additional latency.
Rapthor's outer loop runs for multiple cycles (typically 5–15). Each cycle is a complete recalibration + re-image pass. Under current Toil, if the leader process dies mid-loop, restart requires manual intervention. Under PanDA+iDDS, the server holds all state and the loop re-enters automatically. Individual task failures trigger retries at alternative sites without human intervention.
A single Rapthor run over a large survey field may decompose into hundreds of independent facets. With PanDA, these can be distributed across multiple SRC nodes simultaneously — e.g., 50 facets processed at SRCNET-UK and 50 at SRCNET-NL in parallel. The brokerage handles data placement and load balancing transparently. This is a qualitative improvement over Rapthor's current single-cluster model.
SRCNet will host multiple pipelines from multiple science teams (Rapthor/DDE for HBA, pulsar pipelines, transient searches, etc.). PanDA's global-share mechanism can enforce per-team, per-pipeline resource quotas across the entire federation. This is significantly more flexible than per-cluster fair-share policies.
PanDA is not hypothetical for SRCNet — it is already being deployed:
- The SRCNet Site Operator documentation includes a dedicated Harvester section describing how to connect a site's Kubernetes cluster to the DOMA PanDA server (
https://pandaserver-doma.cern.ch/api/v1) - Harvester is listed as an optional service in the SRCNet service stack, alongside mandatory services (Rucio RSE, Gatekeeper, monitoring)
- The SRCNet Workloads GitLab group (
gitlab.com/ska-telescope/src/src-workloads) is developing standardised job specifications for SRCNet-compatible workflows - CHEP 2024 included a paper explicitly on "Integrating the PanDA Workload Management System" in the SRCNet context
This means the infrastructure layer already exists. What is missing is Rapthor-specific job definitions and either a Toil-PanDA batch plugin or a CWL-to-pchain workflow submission path.
Toil uses a job store — a shared filesystem path or object store — to pass intermediate files between steps. When running on a single Slurm cluster, this is a GPFS/Lustre mount. With PanDA distributing jobs across multiple sites, the job store must be globally accessible. Two approaches:
- Rucio-mediated staging: convert Rapthor's intermediate products to Rucio datasets, use Rucio rules to transfer them to the execution site before each step. This aligns with the long-term SRCNet architecture but requires significant Rapthor changes.
- Globally-mounted object store: deploy an S3-compatible store (Ceph) accessible from all SRC sites and use Toil's S3 job store. Simpler but creates a single point of failure and a potential bandwidth bottleneck.
Rapthor requires DP3, WSClean, EveryBeam, IDG, and Python dependencies — a large, complex Singularity/Docker image. PanDA assumes that software is either pre-installed at the site (via CVMFS or equivalent) or can be downloaded. For SRCNet, CVMFS or a per-site pull-through container registry will be needed. This is a general SRCNet problem, not unique to PanDA, but it must be solved before any pipeline can run at multiple sites.
PanDA's CWL support via pchain was developed primarily for ATLAS workflows. Radio astronomy CWL workflows use different scatter patterns, resource requirements, and container specifications. Testing Rapthor's actual CWL files against pchain's parser is an essential early step; there will almost certainly be unsupported CWL features or different namespace conventions that need adaptation.
Rapthor's convergence criterion involves inspecting image quality metrics (noise, dynamic range) and comparing them across iterations. Expressing this as an iDDS junction task requires:
- The junction task to read and parse Rapthor's output metrics
- The junction to write a JSON dict updating the workflow parameter dictionary (which direction groups converged, which need re-processing)
- iDDS to conditionally re-enter the loop for non-converged directions
This is architecturally supported by iDDS but is non-trivial to implement and will require coordination between the Rapthor developers and the iDDS team.
Rapthor's CWL scatter creates many small-to-medium jobs (one per calibration direction or facet). PanDA pilots have a startup overhead (authentication, pulling the job spec, container startup). For very short jobs (< 5 minutes), this overhead is significant. Rapthor's calibration and imaging jobs typically run for 30 minutes to several hours per facet, so this is not a major concern — but the short utility steps (flagging, solution application, quality checks) may benefit from job packing (many-to-1 Harvester mode).
As of mid-2026, there is no production-ready toil-batch-system-panda plugin. The closest analogue is toil-batch-system-tes (GA4GH TES). Writing the PanDA batch system plugin is the first concrete engineering task for the Option A path, and while the Toil API is well-documented, PanDA job submission via pandaclient has its own authentication model (IAM tokens) and job specification conventions that must be mapped correctly.
Rapthor currently has limited run-time monitoring beyond Toil's own logging. PanDA's BigPanDA dashboard provides job-level monitoring across all sites out of the box — wall time, CPU efficiency, memory usage, I/O rates. This is a genuine advantage, but connecting Rapthor-specific metadata (which LOFAR field, which calibration cycle, which direction) to BigPanDA job labels requires deliberate instrumentation.
For an SRCNet deployment timeline, we suggest a staged approach:
Stage 1 (6–12 months): Toil-PanDA batch plugin proof of concept
- Implement
PanDABatchSystemas a Toil plugin, usingpandaclientto submit Toil jobs as PanDA jobs at a single SRCNet node running Harvester+Kubernetes - Use a shared Ceph/S3 object store as the Toil job store, accessible from that node
- Run Rapthor's calibration operation (not the full loop) on test data
- Validate late-binding slot acquisition via Harvester in push-with-late-binding mode
- Measure overhead vs. direct Slurm submission
Stage 2 (12–18 months): Multi-site extension
- Extend the Toil job store to a globally-replicated S3 bucket accessible from 2–3 SRC nodes
- Enable PanDA brokerage to place Rapthor jobs across multiple sites based on Rucio data locality
- Test cross-site data transfers between pipeline steps
- Validate container image availability via CVMFS or per-site registry
Stage 3 (18–30 months): iDDS workflow integration
- Port Rapthor's self-calibration loop to an iDDS conditional workflow with junction tasks
- Submit full Rapthor CWL to pchain (or a custom CWL-to-PanDA translation layer)
- Deprecate the Toil leader process for production SRCNet runs
- Enable full BigPanDA monitoring with Rapthor-specific job metadata
PanDA's late-binding architecture is a natural fit for Rapthor's workload profile: a large scatter of heterogeneous compute tasks (CPU-heavy calibration, GPU-optional imaging, I/O-heavy data operations) running across a federated set of SRCNet nodes that hold distributed replicas of large observation datasets. The brokerage's data-locality awareness, combined with Rucio's replica tracking, directly addresses the dominant scheduling constraint for radio astronomy pipelines — getting the compute to run near the data without the user having to manually manage data placement.
The most practical near-term integration path is a Toil AbstractBatchSystem plugin for PanDA, preserving Rapthor's existing CWL/Toil architecture while gaining PanDA's resource matching and multi-site dispatch. The longer-term target is full iDDS workflow integration, which enables genuinely distributed multi-site pipeline execution with the self-calibration loop expressed as a first-class PanDA workflow primitive.
SRCNet has already committed to Harvester as a per-node service and PanDA as its workload management layer, so the infrastructure investment is not Rapthor-specific — the effort here is in the integration glue and the per-pipeline workflow definitions, not in standing up PanDA from scratch.
- PanDA WMS Documentation
- PanDA Brokerage — PanDAWMS docs
- PanDA Pilot Architecture — PanDAWMS docs
- PanDA Harvester Wiki (GitHub)
- iDDS: Intelligent Distributed Dispatch and Scheduling (arXiv:2510.02930)
- Working with iDDS — PanDAWMS docs
- Workflow description in CWL — PanDA pchain docs
- PanDA system — Rubin/LSST PanDA User Guide
- Utilizing Distributed Heterogeneous Computing with PanDA in ATLAS (CHEP 2024)
- PanDA: Production and Distributed Analysis System (EPJ Research Infrastructures, 2024)
- Rapthor GitHub
- Rapthor documentation
- SRCNet Harvester documentation
- SRCNet Site Operator Documentation
- Toil Batch System API
- toil-batch-system-tes (TES plugin reference)
- SKA Regional Centres — SKAO
- Data Management System Analysis for Distributed Computing Workloads