DNE Jobs — Architecture Primer

Deployment

flowchart TD
    subgraph K8s["Kubernetes — eu-central-1"]
        API["<b>sem-experiment-management-tool</b>\n1 replica · port 8080 · Spring: prod"]
        W["<b>worker-scheduler</b>\n1 replica · headless\nSpring: prod, worker-scheduler, job-supervisor\n10 virtual threads · polls every 1 s"]
        JOBS["<b>One-off Jobs</b>\nrun-job-scheduler\nrun-delete-ended-exp-entity-job\nrun-import/sync-mkp-exp-job\nrun-sync-ad-groups-job"]
    end

    Client(["User / upstream"]) -->|HTTP| API
    API -->|insert InternalJob| DB
    W -->|poll & claim| DB
    JOBS -->|batch| DB

    DB[("Aurora PostgreSQL")]
    SM["AWS Secrets Manager\ngoogle · slack · databricks"]
    S3["S3 warehouse"]
    GAds["Google Ads API (LROs)"]
    SDA["sem-data-api (internal)"]

    W -->|schedule / cancel| GAds
    API --> SDA
    K8s -.->|secrets| SM
    W --> S3

Worker: 10 Java 21 virtual threads, atomic claim via FOR UPDATE SKIP LOCKED, supervisor heartbeat every 1 min. Priority: PENDING > UP_FOR_RETRY → non-GOOGLE before GOOGLE → oldest first.

Job operations

Operation	Trigger	Max attempts	Retry delay
`GOOGLE_SETUP_EXPERIMENT`	`POST /schedule`	144	5 min (~12 h)
`GOOGLE_CLEANUP_EXPERIMENT`	Setup terminal failure	3	default
`GOOGLE_CANCEL_EXPERIMENT`	`POST /cancel`	3	default
`GOOGLE_EXPIRE_EXPERIMENT`	`POST /expire`	3	default
`GOOGLE_SYNC_EXPERIMENT_END_DATE`	`PUT` abEnd while SCHEDULED	3	default

GOOGLE_SETUP_EXPERIMENT — the complex one:

Creates draft experiments in Google Ads per customer/campaign group
Creates A/B arms: control (100−split%) + treatment (split%)
Submits scheduling requests → receives LRO op names; persists them to the job row
Polls LROs — retries if any still in-flight
Accepts partial success if ≥95% of ≥20 experiments scheduled (rest logged as skipped)
Fetches B-side draft campaign IDs on success
LRO stall safeguards: no op name > 30 min → cancel + retry; op stuck > 1 h → cancel + retry
Terminal failure: creates GOOGLE_CLEANUP_EXPERIMENT, resets to DRAFT, sends Slack alert

GOOGLE_CLEANUP_EXPERIMENT — cancels partially-created Google Ads experiments; resets experiment to DRAFT. Catch block ensures local state is cleared even if job itself exhausts retries.

GOOGLE_CANCEL_EXPERIMENT — cancels Google Ads experiments; sets status to CANCELLED (terminal).

GOOGLE_EXPIRE_EXPERIMENT — cancels Google Ads experiments early; status stays SCHEDULED (record preserved).

GOOGLE_SYNC_EXPERIMENT_END_DATE — updates end date on active Google Ads experiments; idempotent via synced-list in job properties.

GOOGLE_SETUP_EXPERIMENT flow

flowchart TD
    START([POST /schedule]) --> TBS[experiment → TO_BE_SCHEDULED\ninsert job PENDING]
    TBS --> CLAIM[Worker claims job\nstatus → RUNNING]

    CLAIM --> P1[Phase 1: createExperiments\ncreate missing draft experiments\nper customer / campaign group]
    P1 --> P2[Phase 2: createTreatmentArms\narm A: control 100−split%\narm B: treatment split%]
    P2 --> P3[Phase 3: scheduleExperiments\nsubmit each experiment → receive opName\npersist opNames to job.properties]

    P3 --> POLL{Poll LROs\nGetOperation per opName}

    POLL -->|no opName yet\n> 30 min waiting| STALL1[cancel + retry\nre-submit next attempt]
    POLL -->|opName exists\nLRO > 1 h in-flight| STALL2[cancel + retry\nre-submit next attempt]
    STALL1 --> RETRY
    STALL2 --> RETRY

    POLL -->|any LRO still in-flight| RETRY[UP_FOR_RETRY\nactionableAt = now + 5 min]
    RETRY -->|next attempt ≤144| CLAIM

    POLL -->|all done, <95% success\nand ≥20 experiments| NRETRY[NonRetryableException\njob → FAILED]
    POLL -->|all done\n≥95% success or <20 total| P4

    P4[Phase 4: setDraftCampaignIds\nGAQL query → B-side campaign IDs\npersist to experiment.properties]
    P4 --> SUCCESS[experiment → SCHEDULED\ncreate B-side ExperimentScope rows\njob → SUCCEEDED\nSlack ✓]

    RETRY -->|attempt 144 exhausted| TERMINAL[terminal failure\ninsert GOOGLE_CLEANUP_EXPERIMENT\nexperiment → DRAFT\nSlack ✗]
    NRETRY --> TERMINAL2[insert GOOGLE_CLEANUP_EXPERIMENT\nexperiment → DRAFT\nSlack ✗]

Experiment status transitions

Per-transition details

DRAFT → TO_BE_SCHEDULED

Guard: status must be DRAFT, type must be DNE
Sets experiment.status = TO_BE_SCHEDULED, lastUpdatedBy = caller
Inserts GOOGLE_SETUP_EXPERIMENT (144 attempts, 5-min retry, 0 initial delay)

TO_BE_SCHEDULED → SCHEDULED (GOOGLE_SETUP_EXPERIMENT success)

Writes draft_campaign_ids, draft_experiment_groups, draft_experiment_resource_names to experiment.properties
Inserts ExperimentScope rows: entityType=CAMPAIGN, side=B for each draft campaign ID
Sends Slack success notification to creator

TO_BE_SCHEDULED → DRAFT (GOOGLE_SETUP_EXPERIMENT terminal failure)

Inserts GOOGLE_CLEANUP_EXPERIMENT (3 attempts) — cancels partial Google Ads experiments, clears draft_experiment_groups
Resets experiment.status = DRAFT
Sends Slack failure notification
Catch-safe: if cleanup job itself exhausts retries, a catch block still clears local state and resets status — but Google Ads experiments may remain orphaned

SCHEDULED → CANCELLED (POST /cancel)

Guard: abEnd must be after today
Sets experiment.status = CANCELLED immediately (before job runs)
Inserts GOOGLE_CANCEL_EXPERIMENT (3 attempts) — calls cancelExperiments() per customer group, tracks cancelled list in job properties for idempotent recovery
Calls refreshConflictsAsync() to update scope conflict detection
On job failure: status stays CANCELLED, no rollback, no Slack alert

SCHEDULED → SCHEDULED (POST /expire)

Guard: status must be SCHEDULED, abEnd must be after today
Status does NOT change — record is preserved
Inserts GOOGLE_EXPIRE_EXPERIMENT (3 attempts) — same cancelExperiments() logic as cancel but skips the status update
On job failure: status stays SCHEDULED, no rollback

SCHEDULED → SCHEDULED (PUT abEnd)

Guard: new abEnd must be in the future; no active GOOGLE_SYNC_EXPERIMENT_END_DATE job already exists
Persists new abEnd immediately; inserts GOOGLE_SYNC_EXPERIMENT_END_DATE (3 attempts)
Sync handler calls updateExperimentsEndDate() per customer group, tracks synced list for idempotent recovery
Calls refreshConflictsAsync()
On job failure: abEnd is already saved in DB but Google Ads end date is out of sync — no rollback

Known rough edges

Issue	Detail
Monolithic setup handler ⚠️	All four setup phases run in a single job attempt with no explicit checkpointing. A failure in phase 4 re-runs phases 1–3 on the next retry, relying entirely on Google Ads call idempotency. With up to 144 retries over 12 hours, repeated re-execution of early phases creates a wide window for divergence, duplicate state, and silent errors that are hard to attribute after the fact.
Orphaned Google Ads state	Cleanup job failure leaves experiments in Google Ads
No central state machine	Valid transitions are implied by code paths spread across controller, service, and job handler — no single enforcement point
Cancel and cleanup share logic but are distinct jobs	`GOOGLE_CLEANUP_EXPERIMENT` and `GOOGLE_CANCEL_EXPERIMENT` call the same canceller — the only difference is whether they update experiment status afterward
No way to abort an in-flight setup	Calling `/cancel` while `GOOGLE_SETUP_EXPERIMENT` is mid-retry queues the cancel job but does not stop the setup retries — both jobs can be active simultaneously

hecvd/dne-jobs-primer.md

Select an option

No results found