Author: Afeez Lawal
Scope: Multi-channel notification system (push, SMS, email) for 1M+ users
Design a notification system that delivers push, SMS, and email notifications reliably at scale, with no duplicate sends, no missed sends, and graceful degradation when third-party providers fail.
- Send notifications across three channels: push ( Firebase Cloud Messaging/Apple Push Notification service), SMS (Termii, Twilio), and email (SendGrid, Mailgun)
- Support triggered notifications (user action) and scheduled/batch notifications (campaigns, digests)
- Per-user channel preferences: users can opt in or out of any channel
- Notification history and delivery status visible to users and ops
- 1M+ active users
- Guaranteed atleast, once delivery with idempotency to prevent duplicates
- P99 delivery latency under 5 seconds for triggered notifications
- System remains functional if one or more providers go down
- Audit trail on every notification event
[Trigger Source] → [Notification API] → [Event Queue] → [Dispatcher]
↓
┌────────────────────────────┐
│ Channel Workers (Celery) │
├──────────┬────────┬────────┤
│ Push │ SMS │ Email │
└──────────┴────────┴────────┘
↓
[Provider Adapters]
FCM / APNs / Termii
Twilio / SendGrid
Mailgun
↓
[Delivery Status DB]
↓
[Webhook Handler]
Trigger sources include the application backend, scheduled Celery beats, and admin initiated campaigns. Everything flows through a single API surface and onto a message queue before any provider is touched.
Stores reusable templates per channel. Variables are interpolated at dispatch time.
| Field | Type | Notes |
|---|---|---|
| id | UUID | Primary key |
| name | VARCHAR | Unique identifier e.g. withdrawal_success |
| channel | ENUM | push, sms, email |
| subject | VARCHAR | Email/push only |
| body_template | TEXT | Supports {{variable}} interpolation |
| created_at | TIMESTAMP |
Created the moment a notification is triggered. This is the source of truth before dispatch.
| Field | Type | Notes |
|---|---|---|
| id | UUID | Primary key |
| idempotency_key | UUID | Client-generated, unique constraint |
| user_id | FK | Recipient |
| template_id | FK | |
| channel | ENUM | push, sms, email |
| context | JSONB | Template variables |
| scheduled_at | TIMESTAMP | Null = send immediately |
| status | ENUM | queued, dispatched, delivered, failed, cancelled |
| created_at | TIMESTAMP |
One record per send attempt. Allows retries without ambiguity.
| Field | Type | Notes |
|---|---|---|
| id | UUID | Primary key |
| request_id | FK → NotificationRequest | |
| provider | VARCHAR | e.g. sendgrid, termii |
| provider_message_id | VARCHAR | Provider's own reference |
| attempt_number | INT | Starts at 1 |
| status | ENUM | pending, sent, delivered, bounced, failed |
| error | TEXT | Raw error from provider |
| sent_at | TIMESTAMP | |
| delivered_at | TIMESTAMP | Set via webhook |
| Field | Type | Notes |
|---|---|---|
| user_id | FK | |
| channel | ENUM | |
| enabled | BOOLEAN | |
| updated_at | TIMESTAMP |
Every notification request carries a client generated idempotency_key. The API enforces a unique constraint on this field.
On receipt:
- Check if a
NotificationRequestalready exists with this key - If yes — return the existing record, do not enqueue again
- If no — create the record, enqueue, return
202 Accepted
At the worker level, before calling any provider:
- Lock the
NotificationRequestrow withSELECT FOR UPDATE - Check current status — if already
dispatchedordelivered, exit without sending - Only proceed if status is
queued
This two-layer check prevents duplicates even if the queue delivers the message more than once (which is guaranteed atleast once queues do under failure).
The provider's own message_id is stored on NotificationDelivery immediately after the API call returns, before any status update. If a duplicate delivery attempt reaches the worker, the existing provider_message_id is detected and the send is skipped.
Validate payload, resolve user preferences, write NotificationRequest as queued, publish event to queue. Return 202 Accepted immediately — no provider is called in the request lifecycle.
A lightweight dispatcher worker reads from the queue and routes to the correct channel worker queue (push, sms, email). Scheduling logic runs here — if scheduled_at is in the future, the event is re-enqueued with a delay rather than processed immediately.
Each channel has its own Celery worker pool. The worker:
- Acquires a row lock on the
NotificationRequest - Checks idempotency state
- Interpolates the template with context variables
- Calls the primary provider
- Writes a
NotificationDeliveryrecord immediately with the provider's reference - Updates
NotificationRequeststatus todispatched
Providers send delivery receipts asynchronously. A webhook handler receives these, matches on provider_message_id, and updates NotificationDelivery.status to delivered, bounced, or failed. This is the only place final delivery status is set.
Each channel supports multiple providers ranked by priority. If the primary provider fails, the worker falls over to the next one automatically — without re-queuing or user intervention.
Push: FCM (primary) → APNs direct (iOS fallback)
SMS: Termii (primary) → Twilio (fallback)
Email: SendGrid (primary) → Mailgun (fallback)
Failover logic lives in a ProviderAdapter base class. Each adapter exposes a send() method. The channel worker iterates the provider list until one succeeds or all are exhausted.
A provider is marked temporarily unavailable in Redis after N consecutive failures within a rolling window. While the circuit is open, the worker skips that provider entirely and goes straight to the fallback. The circuit resets after a configurable cooldown period.
Retries are distinct from failover. Failover is same attempt, different provider. Retry is a new attempt after a delay.
- Max retries: 5
- Backoff: exponential, 1m, 2m, 4m, 8m, 16m
- Retry only on transient errors (timeout, 429, 5xx). Do not retry on permanent errors (invalid phone number, unsubscribed email, invalid device token).
- Each retry creates a new
NotificationDeliveryrecord with an incrementedattempt_number - After max retries exhausted,
NotificationRequestis markedfailedand an alert fires
| Failure Scenario | Behavior |
|---|---|
| Primary provider down | Automatic failover to secondary provider, no delay |
| All providers for a channel down | Mark failed, alert ops, retry when circuit resets |
| Queue broker down | API returns 503, requests are not lost, client retries with same idempotency key |
| Webhook handler down | Delivery status stays sent until handler recovers and replays missed events |
| Database unavailable | Queue absorbs backpressure; workers pause and resume on recovery |
The system is designed so that losing any single component creates degraded delivery, not zero delivery.
For bulk sends (e.g. a 500K-user campaign), a fan-out approach is used:
- Admin creates a campaign with a target segment and scheduled time
- At schedule time, a Celery beat job runs a paginated query against the user segment
- Each page (e.g. 1,000 users) is published to the queue as a batch event
- Workers fan out individual
NotificationRequestrecords from each batch - Rate limiting per provider is enforced at the worker level using Redis token buckets, stays within provider API limits without dropping messages
This prevents a single large campaign from starving real-time triggered notifications. Campaign workers run on a separate low-priority queue.
Metrics to track:
- Notifications enqueued / dispatched / delivered / failed per channel per minute
- Provider error rate and latency per provider
- Circuit breaker state per provider
- Queue depth and consumer lag
- End-to-end delivery time (enqueue → delivered webhook)
Logs: Every state transition on NotificationRequest and NotificationDelivery is logged with a correlation ID that traces back to the original trigger. Logs are structured JSON shipped to a log aggregator.
| Component | Choice | Reason |
|---|---|---|
| API | FastAPI / Django DRF | |
| Task queue | Celery + Redis | Proven, supports priorities and ETAs |
| Message broker | Redis (or RabbitMQ for durability) | Low latency, pub/sub support |
| Database | PostgreSQL | ACID guarantees for idempotency checks |
| Circuit breaker state | Redis | Fast reads, TTL-based reset |
| Rate limiting | Redis token bucket | Atomic increments, per-provider limits |
- No duplicates: Idempotency key enforced at API layer and re-checked at worker layer before every send
- No missed sends: Atleast once queue delivery + retry with backoff
- Graceful degradation: Circuit breakers, provider fallover, and queue-based decoupling mean no single failure takes the system to zero
- Auditability: Every state change is recorded, who triggered it, which provider handled it, when it was delivered