WSO2 API Manager — Control Plane / Data Plane on Kubernetes (OSS)

A complete, open‑source only deployment guide for running WSO2 API Manager (APIM) as the Control Plane (CP) and Choreo Connect or APK (WSO2 Kubernetes Gateway) as the Data Plane (DP) on Kubernetes. Includes architecture, domain mappings, load balancing, and ready‑to‑adapt K8s manifests.

TL;DR

CP runs APIM (Publisher, Dev Portal, Admin, Key Manager) with Postgres.
DP runs a K8s‑native gateway (Choreo Connect or APK). You publish from CP to one or more DP environments.
Scale DP independently with HPAs (Horizontal Pod Autoscalers), put Ingress/LB in front, and optionally do multi‑cluster/global LB.
All components used here are Apache 2.0 or equivalent OSS.

Architecture Overview

flowchart TB
  subgraph CP[Control Plane, Namespace: apim-cp]
    APIM[WSO2 API Manager\nPublisher / Dev Portal / Admin / Key Manager]
    DB[(PostgreSQL\nAM_DB / SHARED_DB)]
    APIM --> DB
  end

  subgraph DP1[Data Plane, Namespace: apim-dp]
    Adapter[Adapter]
    Router[Router, Envoy]
    Enforcer[Enforcer]
    Router -->|JWT validate via JWKS| KM[(Key Manager\nJWKS @ CP)]
    Adapter -->|API configs| Router
    Adapter --> Enforcer
  end

  Clients((API Clients)) -->|HTTPS| Router
  APIM -- publish/deploy --> Adapter
  KM -.-> APIM

Note: Replace Choreo Connect (Adapter/Router/Enforcer) with APK components if you choose APK as DP. The control flow remains the same.

Domains (DNS) & Endpoints

Use dedicated hostnames. Example mapping (adapt to your DNS zone):

Purpose	Hostname	Notes
Publisher Portal (CP)	`publisher.example.bank`	Human operators create/version APIs
Dev Portal (CP)	`devportal.example.bank`	Application registration, keys
Admin Portal (CP)	`admin.example.bank`	Global admin tasks
Key Manager / JWKS (CP)	`keymgr.example.bank`	OIDC issuer, token, JWKS used by DP
DP Gateway – Prod Cluster A	`gw.prod-a.example.bank`	External traffic entry to DP A
DP Gateway – Prod Cluster B	`gw.prod-b.example.bank`	External traffic entry to DP B
Single global DNS (optional)	`api.example.bank`	CNAME/GLB to one or many DPs

On‑prem: use MetalLB or an external L4/L7 LB (F5/HAProxy/NGINX) to back the gateway Ingress.

Load Balancing in Data Plane

Load balancing in the DP happens at multiple layers:

flowchart LR
  Client((Client))
  LB[External LB / Ingress Controller]
  Svc[ClusterIP Service]
  subgraph RouterPods[Router Pods]
    R1[Router Pod 1]
    R2[Router Pod 2]
    R3[Router Pod N]
  end

  Client -->|HTTPS| LB --> Svc --> RouterPods

External LB / Ingress Controller: Distributes traffic to the DP gateway service.
ClusterIP Service: Kubernetes service balances traffic across multiple Router pods.
Router Pods: Stateless Envoy instances that process requests.
Enforcer Pods: Scaled separately, working in parallel for token validation and policy enforcement.

Configuration Steps

Ingress or LoadBalancer Service: Expose gw.example.bank externally.
ClusterIP Service: Points to choreo-connect-router deployment.
HorizontalPodAutoscaler: Scale Router and Enforcer independently.
PodDisruptionBudgets: Ensure minimum availability during upgrades.

Prerequisites

Kubernetes cluster(s) with storage class for PVCs
Namespaces: apim-cp, apim-dp, and platform (for shared addons)
NGINX Ingress Controller (platform)
cert-manager for TLS (platform) — or your internal CA
PostgreSQL for APIM databases (stateful)
Working DNS for the hostnames above

Step 1 — Control Plane (APIM) Setup

1.1 Databases

Create two databases on Postgres (same instance is fine to start):

AM_DB — API-M config/runtime
SHARED_DB — users, tokens, etc.

1.2 Deploy APIM (CP)

Deploy APIM in apim-cp namespace.
Configure DB URLs/creds via K8s Secrets.
Expose only: /publisher, /devportal, /admin, and Key Manager/JWKS endpoints.
Do not expose the built‑in gateway from CP.

Ingress (example):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: apim-cp
  namespace: apim-cp
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
    - hosts: [publisher.example.bank, devportal.example.bank, admin.example.bank, keymgr.example.bank]
      secretName: apim-cp-tls
  rules:
    - host: publisher.example.bank
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: apim
                port:
                  number: 9443
    - host: devportal.example.bank
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: apim
                port:
                  number: 9443
    - host: admin.example.bank
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: apim
                port:
                  number: 9443
    - host: keymgr.example.bank
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: apim
                port:
                  number: 9443

HPA (suggested start):

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: apim-cp
  namespace: apim-cp
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: apim
  minReplicas: 2
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

Step 2 — Data Plane (DP) Setup (Choreo Connect default)

Components:

Adapter — receives API deployments from APIM
Router (Envoy) — handles incoming traffic
Enforcer — auth, rate limiting enforcement

2.1 Configure DP ↔ CP trust

Point Enforcer/Router to https://keymgr.example.bank for JWT/JWKS.
Import CP CA into DP truststore if you use an internal CA.
Configure Adapter to connect to APIM’s control endpoints.

2.2 Expose the Gateway

Expose a single external hostname per DP via Ingress or Service type: LoadBalancer.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dp-gateway
  namespace: apim-dp
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
    - hosts: [gw.prod-a.example.bank]
      secretName: dp-gw-tls
  rules:
    - host: gw.prod-a.example.bank
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: choreo-connect-router
                port:
                  number: 9095  # adjust to your router service port

2.3 Scale & Load Balance inside the cluster

Use HPAs and a ClusterIP Service in front of each component. Requests reach the Router Service via Ingress.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: cc-router
  namespace: apim-dp
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: choreo-connect-router
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: cc-enforcer
  namespace: apim-dp
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: choreo-connect-enforcer
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

PodDisruptionBudget (recommended):

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: cc-router-pdb
  namespace: apim-dp
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: choreo-connect-router

Optional — Data Plane with APK (WSO2 Kubernetes Gateway)

Deploy APK controller + data plane in apim-dp.
Register APK environment with APIM CP; expose a gateway hostname per cluster similar to Choreo Connect.
HPAs/PDBs and Ingress patterns remain the same.

Step 3 — Register DP Environments in APIM

In Publisher, create Environments (e.g., dp-prod-a, dp-prod-b) that map to your DP(s).
When publishing an API, select one or more environments. APIM pushes the config to DP Adapter(s).

Step 4 — Publish & Invoke Flow

sequenceDiagram
  autonumber
  actor Dev as API Publisher (Human)
  participant Pub as APIM Publisher (CP)
  participant Ad as Adapter (DP)
  participant Enf as Enforcer (DP)
  participant Rt as Router (DP)
  actor App as Client App

  Dev->>Pub: Create/Update API, select environments (dp-prod-a)
  Pub-->>Ad: Push API deployment (events/config)
  Ad-->>Rt: Configure routes
  Ad-->>Enf: Configure auth/policies

  App->>Rt: HTTPS request with JWT
  Rt->>Enf: AuthZ / Throttle check
  Enf->>Pub: (JWKS/Introspection @ Key Manager)
  Enf-->>Rt: Allow/Block
  Rt-->>App: Response from upstream

Load Balancing Patterns for the Data Plane

A) Inside a single cluster (intra‑cluster)

Ingress → Service (ClusterIP) → multiple Router pods (stateless).
HPAs scale Router and Enforcer separately.
No sticky sessions required; JWT validation is stateless.

B) At the cluster edge (north‑south)

Use Service type=LoadBalancer (cloud) or Ingress + external LB (on‑prem via NGINX/MetalLB/F5/HAProxy).
Terminate TLS at Ingress/LB, or use TLS passthrough if you need end‑to‑end TLS.

C) Across multiple clusters/regions

Publish the same API to multiple environments (dp-prod-a, dp-prod-b).
Front them with global DNS/LB (weighted/geographic/latency) and health checks.

flowchart TB
  subgraph Global
    DNS[api.example.bank\n Global DNS/LB]
  end
  DNS --> A[gaw.prod-a.example.bank\n Cluster A Ingress/LB]
  DNS --> B[gaw.prod-b.example.bank\n Cluster B Ingress/LB]
  A --> R1[Router Pods]
  B --> R2[Router Pods]

Keep each DP’s hostname addressable directly as a fallback (e.g., gw.prod-a.example.bank).

Security Hardening

TLS everywhere (Ingress certificates from cert‑manager or internal CA).
JWKS trust: DP must trust CP’s JWKS/Issuer; rotate signing keys on a schedule.
Secret management: use K8s Secrets and limit RBAC access.
NetworkPolicies: restrict DP egress to CP and your upstreams only.
mTLS (optional) between DP and upstream backends.

Observability & Ops

Metrics: scrape APIM and DP with Prometheus; Grafana dashboards.
Logs: ship access logs to Loki/ELK; include correlation IDs.
Readiness/Liveness: tune probes for APIM, Router, Enforcer, Adapter.
Backups: Postgres full + WAL; test restores.
Canary: publish to a canary DP environment, then roll out to prod DPs.

Sizing & Tuning (starting points — load test and adjust)

APIM (CP): 2–5 replicas; requests ~500m CPU / 2Gi RAM per pod.
Choreo Connect (DP):
- Router: start ~500m CPU / 1Gi RAM per pod, HPA 2→10 @ 60% CPU
- Enforcer: similar to Router; scale with token/policy complexity
- Adapter: 1–3 replicas (low RPS)
Postgres: fast disk, tune connections/WAL; separate PVCs.

Example: Minimal NetworkPolicies (restrict DP egress)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: dp-egress
  namespace: apim-dp
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              name: apim-cp
      ports:
        - protocol: TCP
          port: 9443
    - to:
        - ipBlock:
            cidr: 10.0.0.0/8  # your upstreams/VIPs
      ports:
        - protocol: TCP
          port: 443

End‑to‑End Checklist

Appendix — Notes for APK as DP

APK components run as K8s controllers/gateways; register with APIM as environments.
Ingress/HPAs/NetworkPolicies patterns are the same as Choreo Connect.
Choose APK if you want deeper K8s Gateway API alignment and multi‑cluster growth.

Official Configuration References (CP & DP)

Use these docs to confirm hostnames ↔ service ports, Ingress examples, and per‑component settings.

Control Plane (APIM)

APIM on Kubernetes (Helm resources) — installing APIM and exposing portals via Ingress: https://apim.docs.wso2.com/en/4.5.0/install-and-setup/install/deploying-api-manager-with-kubernetes-resources/
Publisher/Dev Portal URLs & publishing flow — default endpoints (/publisher, /devportal, /admin): https://apim.docs.wso2.com/en/4.5.0/manage-apis/deploy-and-publish/publish-on-dev-portal/publish-an-api/
K8s Deployment Pattern 2 (All‑in‑one GW) — reference for CP/Ingress basics and service exposure: https://apim.docs.wso2.com/en/latest/install-and-setup/setup/kubernetes-deployment/kubernetes/am-pattern-2-all-in-one-gw/

Data Plane — Choreo Connect (if you choose CC)

Version note: For APIM 4.5, WSO2’s current docs keep Choreo Connect guides under the 4.1/4.2 doc lines. The deployment model with APIM as Control Plane is unchanged and works with APIM 4.5. If you want the most up‑to‑date K8s‑native gateway guidance, prefer APK below.

Deploy CC on K8s with APIM as Control Plane (doc hosted under 4.2): https://apim.docs.wso2.com/en/4.2.0/deploy-and-publish/deploy-on-gateway/choreo-connect/getting-started/deploy/cc-on-kubernetes-with-apim-as-control-plane/
CC Router configuration (listeners/ports): https://apim.docs.wso2.com/en/4.0.0/deploy-and-publish/deploy-on-gateway/choreo-connect/configurations/router-configurations/
CC Enforcer configuration (auth/JWKS): https://apim.docs.wso2.com/en/4.0.0/deploy-and-publish/deploy-on-gateway/choreo-connect/configurations/enforcer-configurations/
Multiple Gateways with CC (distributed adapter): https://apim.docs.wso2.com/en/4.1.0/deploy-and-publish/deploy-on-gateway/choreo-connect/configure-multiple-gateways-with-choreo-connect/

Data Plane — APK (WSO2 Kubernetes Gateway) (recommended for APIM 4.5)

APIM 4.5 compatibility matrix (shows APK 1.3.0 pairing): https://apim.docs.wso2.com/en/4.5.0/includes/compatibility-matrix/
Set up APIM→APK (APIM APK Agent) (4.5 doc): https://apim.docs.wso2.com/en/4.5.0/install-and-setup/setup/distributed-deployment/configuring-apim-as-a-gateway/
APK Quick Start with APIM CP (1.3.0): https://apk.docs.wso2.com/en/1.3.0/control-plane/apk-as-gateway-in-apim/apk-as-gateway-in-apim-qsg/
APK deployment patterns on VMs/K8s: https://apk.docs.wso2.com/en/1.3.0/setup/deployment/apk-as-gateway-in-apim-deployment-patterns-vm/
APK Ingress exposure (service ↔ host/port mapping): https://apk.docs.wso2.com/en/latest/setup/ingress.html

(WSO2 Kubernetes Gateway)

Install & expose via Ingress — mapping gateway service to host/port: https://apk.docs.wso2.com/en/latest/setup/ingress.html
APIM Control Plane + APK in K8s — deployment patterns and values: https://apk.docs.wso2.com/en/1.3.0/setup/deployment/apk-as-gateway-in-apim-deployment-patterns-k8s/
APK Quick Start with APIM CP — step‑by‑step, includes sample host mappings: https://apk.docs.wso2.com/en/1.3.0/control-plane/apk-as-gateway-in-apim/apk-as-gateway-in-apim-qsg/

Ingress & LB (general K8s/on‑prem)

NGINX Ingress Controller — Ingress class, TLS, backend service/targetPort mapping: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/

MetalLB — L2 mode VIPs for on‑prem clusters: https://metallb.universe.tf/configuration/
Sample L2 config

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: public-pool
  namespace: metallb-system
spec:
  addresses: ["192.168.10.100-192.168.10.120"]
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2adv
  namespace: metallb-system
spec: {}

HAProxy (outside cluster) — fronting your gateway VIP with TCP passthrough (optional):

frontend api_front
  bind *:443
  mode tcp
  default_backend k8s_gateway
backend k8s_gateway
  mode tcp
  balance roundrobin
  server gwvip 192.168.10.101:443 check

aungkyawminn/WSO2 API Manager - Scalable K8 Deployment Guide.md