graph LR
S1["Story 1: Source attribution"]
S2["Story 2: Bucketing"]
S3["Story 3: Data-driven popup"]
S4["Story 4: Exposure tracking"]
S5["Story 5: A/A validation"]
S6["Story 6: A/B experiment"]
S1 --> S5
S2 --> S5
S3 --> S5
S4 --> S5
S5 -->|must pass| S6
style S1 fill:#dcfce7,stroke:#16a34a
style S2 fill:#dcfce7,stroke:#16a34a
style S3 fill:#dcfce7,stroke:#16a34a
style S4 fill:#dcfce7,stroke:#16a34a
style S5 fill:#fef3c7,stroke:#d97706
style S6 fill:#fee2e2,stroke:#dc2626
All build stories (1-4) must complete before Story 5 (A/A) starts. With a single engineer, Stories 1-4 are sequential (~9 days). Story 5 must pass validation before Story 6 (A/B) starts. The engineer is free during the A/A and A/B run periods.
Analysts can identify whether a Welcome Offer purchase originated from the popup or the coin store.
Implementation note: the popup doesn't complete the purchase directly. It closes itself, opens the cashier modal, and pre-selects the package. Both the popup and coin store paths converge at the cashier. The source tag must be set before this convergence point.
| Task | Scope | Description |
|---|---|---|
| 1a | pok-store | Accept optional source field on the claim endpoint. Values: welcome_offer_popup, coin_store. Include in PACKAGE_CLAIMED event payload. Unit/integration tests. |
| 1b | gp-game-client | Pass source=welcome_offer_popup when a Welcome Offer purchase is initiated from the popup (before the cashier modal opens). Pass source=coin_store when a Welcome Offer purchase is initiated from the coin store. Verify source reaches the claim request. |
| 1c | pok-snowflake | Update PACKAGE_CLAIMS_TASK to extract source from the PACKAGE_CLAIMED payload. Add column to output table (ALTER TABLE ADD COLUMN IF NOT EXISTS). Same pattern as the geo-location field extraction (POK-26733). |
New users are assigned one of two Welcome Offer variants at registration, consistently and auditably.
The original production package (1603957654412) is reused as the control. Only one new package is needed per experiment phase.
| Task | Scope | Description |
|---|---|---|
| 2a | pok-store (signup-bonus service) | Hash-based assignment with experiment-specific seed. Two Welcome Offer package IDs configurable via environment variables (FTO_CONTROL_PACKAGE_ID, FTO_VARIANT_PACKAGE_ID). When env vars are set, the PM skips these package IDs in the normal isAssignedToNewUsers assignment loop and instead assigns one based on the user's bucket. When env vars are empty/unset, the PM falls back to normal assignment (all isAssignedToNewUsers packages assigned to everyone). Integration tests. |
The Welcome Offer popup displays the correct offer based on the package assigned to the user.
| Task | Scope | Description |
|---|---|---|
| 3a | gp-game-client | Render price, GC amount, SC amount from package data instead of hardcoded i18n. Static text (title, labels, dismiss) stays in i18n. Unit tests. |
| 3b | gp-game-client | Include package ID in FTO loaded/clicked/declined log events for variant identification. Note: these events go to Loki (via stdout), not Snowflake. Useful for debugging and monitoring, but not for experiment analysis. |
The analysis team can identify which users were actually shown the Welcome Offer, accounting for restricted market filtering that happens client-side.
Context: the backend assigns Welcome Offer packages to all new users (it has no market awareness). The client removes the Welcome Offer feature for users in restricted markets based on a combination of residential state and geolocation at login time. This filtering is fluid (market config changes with deploys) and can't be reconstructed from Snowflake data after the fact. The client is the only place that knows definitively whether the popup was shown.
| Task | Scope | Description |
|---|---|---|
| 4a | pok-store | Add WELCOME_OFFER_DISPLAYED to EventType.java. New endpoint (e.g., POST /packages/{packageId}/displayed) that writes this event to the customer-package event store with user ID, package ID, and timestamp. Unit/integration tests. |
| 4b | gp-game-client | When the popup is shown, fire a POST to the new endpoint alongside the existing LogManager.metric('FTO loaded') call. Fire-and-forget (don't block the popup on the response). Catch and log errors to Loki so silent failures are visible during monitoring. |
| 4c | pok-snowflake | Add a CURATED layer SQL task/view for WELCOME_OFFER_DISPLAYED. Follow the same pattern as the geo-verification CLEANSED layer work (POK-26739). Extract user ID, package ID, timestamp from the event payload. Without this, analysts can still query RAW/CLEANSED by filtering EVENT_TYPE = 'WELCOME_OFFER_DISPLAYED', but the payload is raw JSON. |
Pipeline note: the event flows to Snowflake automatically. pok-store's Snowflake process manager reads all events from event_store.events_v2 by position with no type filtering. The path is: PostgreSQL -> Kinesis Firehose -> S3 ({env}-transaction-dlz/v2_store/) -> Snowpipe -> Snowflake RAW -> CLEANSED (deduplication). The packages-rmp read model projector silently ignores unknown event types, so no breakage.
The experiment infrastructure is verified end-to-end before the real variant is introduced.
The original production package (1603957654412) serves as Group A. A single new package (identical contents) serves as Group B.
| Task | Scope | Description |
|---|---|---|
| 5a | pok-store | Create one new Welcome Offer package ($20/SC40) in QC and prod. Config must match current prod package: claim limit 1, relative expiry 7 days (604800000ms), unassign-on-next-purchase false, play type Promotional, platform globalpoker.com. Set FTO_CONTROL_PACKAGE_ID to original (1603957654412), FTO_VARIANT_PACKAGE_ID to new package. |
| 5b | QA | Verify in QC: both variants render correctly in popup and coin store, bucketing is balanced, package ID and source logged correctly, PACKAGE_ASSIGNED_TO_CUSTOMER and WELCOME_OFFER_DISPLAYED events visible in Snowflake RAW, rollback works (clear env vars, original assignment resumes). |
| 5c | Ops | Deploy A/A to prod. Run ~2 weeks. Monitor for errors. Hand off to Behavioural Science for statistical validation. |
Gate: A/A must pass before proceeding to Story 6.
The treatment variant is live and measurable for new registrations.
| Task | Scope | Description |
|---|---|---|
| 6a | pok-store | Create treatment package ($10, 100k GC, SC20) in QC and prod. All config identical to control except price and contents. |
| 6b | QA | Verify in QC: treatment renders correctly in popup and coin store, correct package assigned to correct bucket, source attribution and exposure tracking still work. |
| 6c | Ops | Update FTO_VARIANT_PACKAGE_ID from A/A package to treatment package. FTO_CONTROL_PACKAGE_ID stays the same (original). Run ~6 weeks + 8 day measurement window. Monitor for errors. Hand off to Behavioural Science for analysis. |
Rollback: Clear the env vars (FTO_CONTROL_PACKAGE_ID, FTO_VARIANT_PACKAGE_ID). The PM falls back to normal isAssignedToNewUsers assignment, which assigns the original package to all new users. No package config changes needed.
gantt
title Welcome Offer A/B Test (single engineer)
dateFormat YYYY-MM-DD
axisFormat %b %d
section Build
Story 1 - Source attribution :s1, 2026-04-14, 2d
Story 2 - Bucketing :s2, after s1, 2d
Story 3 - Data-driven popup :s3, after s2, 3d
Story 4 - Exposure tracking :s4, after s3, 2d
section A/A Validation
Create package + QA :s5ab, after s4, 3d
A/A run in prod :s5c, after s5ab, 14d
A/A analysis :s5d, after s5c, 3d
section A/B Experiment
Create variant + QA :s6ab, after s5d, 2d
A/B run in prod :s6c, after s6ab, 42d
Measurement window :s6d, after s6c, 8d
Final analysis :s6e, after s6d, 5d
Note: Build start date is placeholder (TBD based on resourcing discussion). Stories 1-4 are sequential (single engineer). The experiment phases (A/A run, A/B run, measurement window) don't require active engineering - the engineer is free for other work during those periods.
| Story | Team | Estimate |
|---|---|---|
| Story 1: Source attribution | pok-store + gp-game-client + pok-snowflake | ~2 days |
| Story 2: Bucketing | pok-store (signup-bonus service) | ~1-2 days |
| Story 3: Data-driven popup | gp-game-client | ~2-3 days |
| Story 4: Exposure tracking | pok-store + gp-game-client + pok-snowflake | ~2 days |
| Story 5: A/A validation | QA + Ops | ~2.5 weeks (setup + run) |
| Story 6: A/B experiment | QA + Ops | ~7 weeks (setup + run + measurement) |
Build phase (Stories 1-4): ~2 weeks sequential (single engineer). All work across pok-store, gp-game-client, and pok-snowflake is owned by POK Engineering. Total end-to-end: ~12.5 weeks from build start to final results. Active engineering time is ~2.5 weeks; the remaining ~10 weeks are experiment runtime and analysis (no engineering required).
From meeting 2026-03-27:
- 10% relative MDE, alpha = 0.10 (two-sided), 80% power
- ~10,000 total samples (across both arms), ~6 weeks
- Registration volume: ~1,700/week (average since 2026-01-01)
- Sample size contingent on 13% baseline conversion rate (to be confirmed)
- Primary: First purchase conversion within 24h of registration
- Secondary: Second purchase within 7 days (conditional + unconditional), ARPU
- Exploratory: Conversion by source (welcome_offer_popup vs coin_store)
- Secondary metrics are exploratory only; primary metric is the sole go/no-go basis.
Experiment analysis uses Snowflake. The pok-store event pipeline is generic: all events written to event_store.events_v2 automatically flow to Snowflake RAW/CLEANSED with no type filtering or configuration changes.
pok-store (event_store.events_v2)
-> Snowflake PM polls by position (all events, no filtering)
-> Kinesis Firehose (batches by size/time)
-> S3 ({env}-transaction-dlz/v2_store/)
-> Snowpipe auto-ingest
-> Snowflake RAW
-> Deduplication task
-> Snowflake CLEANSED
-> [Manual] CURATED views/tasks (tasks 1c and 4c)
| Data | Event | Destination | Purpose |
|---|---|---|---|
| Variant assignment | PACKAGE_ASSIGNED_TO_CUSTOMER |
Snowflake (auto) | Which user got which variant, when |
| Exposure | WELCOME_OFFER_DISPLAYED (new) |
Snowflake (auto) | Which users were actually shown the popup. Accounts for restricted market filtering that happens client-side. |
| Purchases | PACKAGE_CLAIMED + source field |
Snowflake (auto) | Who purchased, what package, amount, source (welcome_offer_popup or coin_store) |
| Registration | v_account_registration |
Snowflake | Registration timestamp, state/country |
| Client logs | FTO loaded/clicked/declined | Loki only | Debugging and monitoring. Not in Snowflake. |
The full experiment funnel in Snowflake: assigned -> displayed -> claimed. The difference between assigned and displayed is the restricted market population. Restricted market filtering is fluid (depends on residential state + geolocation at login, config changes with deploys), so the WELCOME_OFFER_DISPLAYED event is the definitive exposure signal.
| File | Repo | Relevance |
|---|---|---|
src/webapp/js/base/ui/fto/fto-dialog-sign-up-special.tsx |
gp-game-client | Popup component (render + purchase) |
src/webapp/js/base/ui/fto/fto-container.tsx |
gp-game-client | Popup container |
src/webapp/js/base/ui/views/App.tsx |
gp-game-client | Popup trigger logic (line 361) |
src/webapp/js/base/config/ApplicationConfig.ts |
gp-game-client | Package IDs per env (prod: 1603957654412) |
src/webapp/js/base/redux/cashier/PurchaseActions.ts |
gp-game-client | Purchase flow + package fetching |
src/webapp/i18n/en.json |
gp-game-client | Hardcoded popup text (fto.* keys) |
server/signup-bonus-pm/ |
pok-store | Package assignment at registration |
server/packages/domain/.../EventType.java |
pok-store | Event type definitions |
server/packages/http/.../CustomerPackageEndpoints.kt |
pok-store | Claim endpoint (add source + displayed endpoint here) |
server/packages/domain/.../PackageAggregate.kt |
pok-store | Event generation |
server/packages/domain/.../CustomerPackageAggregate.java |
pok-store | Claim limit logic (per-customer-per-package) |
infra/snowflake-pipeline/ |
pok-store | Pipeline infra (no changes needed) |
sdf/sql/packages/tasks/PACKAGE_CLAIMS.sql |
pok-snowflake | Package claims CURATED task (add source field here) |
sdf/package_claims.tf |
pok-snowflake | Package claims Terraform setup |
flyway/sql/sdf/repeatable/curated/ |
pok-snowflake | CURATED layer (add WELCOME_OFFER_DISPLAYED view here) |
See investigation gist for the full list. Key blockers before build starts:
Current prod package config- Resolved. Claim limit: 1, relative expiry: 7 days, unassign-on-next-purchase: false.- 13% baseline confirmation - sample size depends on this
- Build start date - pending resourcing discussion (Phillip + Rami)