GeoComply Rollout Observability Roadmap

Context

Geo analytics events serve two systems for different purposes:

Loki (operational): "Is the geo flow healthy right now?" Aggregate rates, real-time alerts, engineer audience.
Snowflake (analytical): "What is geo doing to our players over time?" Per-player, historical, joins with other domains, PM/analyst/compliance audience.

See first-time-repeat-player-analysis.md for the full investigation and data pipeline architecture.

Current state

Loki path (in draft PRs, awaiting review):

9 geo analytics events instrumented in game client (POK-26711/12/13 branches)
Grafana dashboard: 29 panels, 6 rows (canary, funnel, outcomes, A/B, geo comparison, monitoring)
6 alert rules in canonical YAML (unified source for TF + local dev)
P50/P95 verification latency panel + alert
Monte Carlo simulator with web GUI for local testing
Events flow: Browser -> LogManager -> /log -> stdout -> CloudWatch -> Firehose -> Loki

Snowflake path (gap identified, not yet implemented):

IDENTITY_LOGIN table has 5 geo columns, all hardcoded NULL (schema-forward placeholder)
CloudFront IP-based geo is captured and JWT-embedded but filtered out before reaching the user event store
No GeoComply data reaches Snowflake
Login, registration, and player activity data already flows via user-eventstore-snowflake-pm -> S3 DLZ -> Snowpipe

Epic: GeoComply Rollout Observability

Phase 0: Loki Instrumentation (pre-canary, must-have)

Game client events, Grafana dashboard, and alert rules. This is the minimum needed for operational visibility when the canary turns on.

Ticket	Type	Repo	Description	Status
POK-26711: Add geo analytics event abstraction	Story	gp-game-client	Sink pattern, LogManager + FullStory sinks, `setGeoContext`, platform detection	Draft PR, awaiting review
POK-26712: Instrument login verification flow	Story	gp-game-client	rollout.evaluated, notice.shown/dismissed/skipped, verification.started/completed, lobby.loaded	Draft PR, awaiting review
POK-26713: Instrument post-login monitoring flow	Story	gp-game-client	monitor.location_checked, monitor.signed_out	Draft PR, awaiting review
TBD: Geo location dashboard + alerts	Story	pok-infra	29-panel dashboard, 6 alert rules (canonical YAML), unified TF + local dev, Monte Carlo simulator, durationMs latency tracking	Draft PR, awaiting review

Phase 1: Snowflake Pipeline (pre-canary, must-have)

All backend and Snowflake changes must be deployed before the canary turns on so we capture data from day one.

1a. CloudFront geo baseline (IP-based, all users)

Ticket	Type	Repo	Description	Owner	Deploy order
POK-26733: Extract geo fields into IDENTITY_LOGIN	Task	pok-snowflake	4 Flyway migrations: CLEANSED columns, payload extraction, CURATED NULLs replaced, DMFs. PR #2227.	Our team	Deploy first
POK-26734: Add geo fields to LOGGED_IN event	Task	pok-user	4 optional fields on LoggedInEvent, AuditLoginCommand, AuditUserLoginRequest. PR #1207.	Our team	Deploy second
POK-26734: Send CloudFront geo in login audit	Task	pok-auth0	Extract viewer_country from user_metadata.cookies, send as top-level fields. PR #454.	Our team	Deploy third

1b. GeoComply verification path (SDK-based, canary users)

Ticket	Type	Repo	Description	Owner	Depends on
TBD: GEO_VERIFICATION_COMPLETED event type + endpoint	Task	pok-user	New event type in user event store. Fields: outcome, subdivision, country, providerName, providerVersion, sourceType, durationMs, restrictionType. New `POST /audit/geo-verification/:id` endpoint.	Our team	-
TBD: Game client calls pok-user after verification	Task	gp-game-client	POST to pok-user alongside existing `emitGeoEvent` in geo-gate-init.ts. Both Loki and Snowflake receive the event.	Our team	pok-user endpoint
TBD: Snowflake ingestion for geo verification events	Task	pok-snowflake	New CLEANSED table for verification events. CURATED view joining login + verification by user/session.	Our team	pok-user endpoint
TBD: Establish retention baselines	Task	Snowflake SQL	Query `ACCOUNT_ACTIVITY_SUMMARY` for pre-geo retention numbers before canary.	Our team	Phase 1a data flowing

Phase 2: First-Time Geo Segmentation (at canary 5%)

Distinguish first-time geo users from repeat users in Loki dashboards. Requires a server-side flag so the signal is reliable across devices.

Ticket	Type	Repo	Description	Owner	Depends on
TBD: Validate dashboards with real traffic	Task	pok-infra	Confirm panels populate with real data. Tune time ranges and alert thresholds.	Our team	Phase 0 merged + canary live
TBD: Add isFirstGeoVerification server-side flag	Story	pok-user + gp-game-client	Add `first_geo_verification_at` to Auth0 `user_metadata` or pok-user. Read in `runGeoGate()`, set via `setGeoContext()`. Write back on first successful verification.	Our team	Phase 0 merged
TBD: Add segmented dashboard panels	Task	pok-infra	First-time vs repeat: failure rate, latency, funnel drop-off. Separate alert thresholds per segment.	Our team	isFirstGeoVerification

Phase 3: Snowflake Integration (before scaling beyond 5%)

Moved to Phase 0 (pre-canary). The backend and Snowflake must be ready before the canary turns on so we capture verification data from day one. See Phase 0 items marked "GeoComply verification path."

Phase 4: 100% Rollout Preparation

When the canary scales from 5% to 100%, 95% of active players hit geo for the first time. These changes separate transition noise from real new-player signal.

Ticket	Type	Repo	Description	Owner	Depends on
TBD: Add geoRolloutPhase dimension	Task	gp-game-client	Distinguish "canary_5", "canary_25", "ga" in events.	Our team	-
TBD: Pre-populate first-geo flags for canary users	Task	pok-user	Mark all canary-period verifiers so they're not counted as "first time" when the switch flips.	Our team	isFirstGeoVerification
TBD: First-time user funnel dashboard	Task	pok-infra	Dedicated panels and alert thresholds for first-time geo users (expected to have higher failure rates).	Our team	isFirstGeoVerification + geoRolloutPhase

Critical path

NOW ──────────────────────────────────────────────────────────────
  Phase 0:  Merge game client instrumentation PRs (POK-26711/12/13)
            Merge dashboard + alerts PR (pok-infra)
  Phase 1a: Deploy pok-snowflake #2227 FIRST (CF geo columns)
            Deploy pok-user #1207 (accept CF geo fields)
            Deploy pok-auth0 #454 (send CF geo)
  Phase 1b: Build + deploy GEO_VERIFICATION_COMPLETED endpoint (pok-user)
            Build + deploy game client POST to pok-user (gp-game-client)
            Build + deploy Snowflake verification ingestion (pok-snowflake)
            Establish retention baselines (Snowflake SQL)

CANARY 5% ────────────────────────────────────────────────────────
  Phase 2:  Validate dashboards with real traffic
            Add isFirstGeoVerification server-side flag
            Segmented dashboard panels

BEFORE 25%+ ──────────────────────────────────────────────────────
  Phase 3:  GEO_VERIFICATION_COMPLETED event + endpoint (pok-user)
            Game client calls pok-user (gp-game-client)
            Snowflake ingestion (pok-snowflake, Data Eng)
            Retention baseline queries

BEFORE 100% ──────────────────────────────────────────────────────
  Phase 4:  geoRolloutPhase dimension
            Pre-populate first-geo flags
            First-time user funnel dashboard

Ownership

Component	Owner	Phases
gp-game-client	Our team	0, 2, 3, 4
pok-infra	Our team	0, 2, 4
pok-auth0	Our team	1
pok-user	Our team	1, 2, 3, 4
pok-snowflake	Our team (reviewed by Data Engineering)	1, 3

rami-ruhayel-vgw/geo-analytics-roadmap.md

Select an option

No results found