This guide describes a reusable security model for people, workloads, automation, and API tools using Keycloak, SPIFFE/SPIRE, and Open Policy Agent (OPA).
The three systems have distinct responsibilities:
- Keycloak authenticates people and OAuth clients and issues browser sessions or access tokens.
- SPIFFE/SPIRE authenticates running workloads with short-lived, automatically rotated identities.
- OPA evaluates authorization policy using identity and resource facts that an application has already verified.
Authentication establishes who or what is acting. Authorization decides whether that identity may perform a specific action on a specific resource. Neither should be treated as a substitute for the other.
Two further concerns build on that foundation, and are covered in later sections. A relationship engine answers object-specific questions a policy engine structurally cannot, such as inherited access and reverse queries. Per-request attribution records who consumed what, keyed on the same verified identities, so usage and cost are non-repudiable.
Every protected request follows the same control sequence:
Person or OAuth client ──Keycloak──┐
├── Application ──OPA decision── Enforced operation
Workload or agent ──SPIFFE/SPIRE───┘
Keycloak or SPIFFE establishes the principal. The application validates that identity, resolves authoritative resource facts, and constructs the policy input. OPA returns an authorization decision. The application enforces that decision, applies domain invariants, performs the operation, and records the durable outcome.
OPA belongs on both paths whenever the operation is policy-controlled:
- for a person or account-based agent, OPA evaluates verified Keycloak claims;
- for an OAuth client, OPA evaluates verified client claims and grants;
- for a SPIFFE workload, OPA evaluates the verified SPIFFE ID;
- for a composed workflow, OPA may evaluate the human or client principal and the workload identity together.
A traditional API key is usually a long-lived bearer secret:
API key → API → permissions associated with whoever possesses the key
This standard replaces that model with a verified identity and a separate policy decision:
Keycloak or SPIFFE → short-lived identity → OPA decision → enforced operation
The replacement depends on the actor:
| Actor | API-key replacement |
|---|---|
| Person | Keycloak browser login and a short-lived user token |
| Person or agent using a public CLI | Keycloak account login and a short-lived bearer |
| In-cluster agent or service | SPIFFE X.509-SVID for mTLS |
| Workload requesting an OAuth token | SPIFFE JWT-SVID client assertion |
| External integration outside SPIRE | Dedicated confidential OAuth client and short-lived access tokens |
Keycloak tokens carry a verified subject or client, issuer, audience, roles, and expiration. SPIFFE identities are tied to workload selectors and rotate automatically. OPA then answers a narrower question than “does this caller know the key?”:
May this verified principal perform this action on this resource now?
This provides:
- short-lived credentials instead of indefinitely valid shared keys;
- one identity per person, agent, client, or workload;
- automatic workload credential rotation;
- audience, scope, and role restrictions;
- resource-level authorization through OPA;
- attribution to the exact authenticated principal;
- revocation by disabling the account, client, workload registration, or policy grant.
OPA does not replace authentication. It replaces coarse key-based permissions with policy evaluated from verified identity and authoritative resource facts.
A confidential OAuth client secret is still a static bootstrap credential, so it is only a partial replacement for an API key. Use it for external integrations that cannot participate in SPIRE. For SPIRE-managed workloads, prefer direct SPIFFE mTLS or a JWT-SVID client assertion so no static client secret is stored.
Replace API keys through a controlled transition:
- Inventory every key, owner, consumer, scope, and target API.
- Classify each consumer as a person, agent account, OAuth client, or workload.
- Assign the matching Keycloak or SPIFFE identity.
- Define its least-privilege roles, scopes, audiences, and OPA policy.
- Validate the new authentication and authorization path independently.
- Observe both paths during an approved migration window without broadening permissions.
- Revoke the old API key and verify that it can no longer access the API.
- Remove key-processing code, stored key material, and obsolete operational procedures.
Do not convert one shared API key into one shared OAuth account or client secret. That preserves the same attribution and blast-radius problem under a different credential format.
| Actor | Authentication | Identity material | Authorization |
|---|---|---|---|
| Person using a web application | Keycloak browser login | Browser session or user access token | Roles and OPA policy |
| Browser automation representing a person | Real Keycloak browser flow | Dedicated test account | The same policy path as the represented user |
| Interactive CLI representing a person or dedicated agent account | Public Keycloak client | Short-lived account bearer | API roles and OPA policy |
| In-cluster agent or service | SPIFFE/SPIRE mutual TLS | Rotating X.509-SVID | Exact SPIFFE identity and OPA policy when required |
| Workload requesting a Keycloak token | Federated workload client | JWT-SVID client assertion | OAuth grants, roles, and OPA policy |
| Privileged administration agent | Dedicated confidential Keycloak client | Confidential client credential | Narrow administration grants and OPA policy |
| External unattended integration | Confidential Keycloak client | Client ID and client secret | Least-privilege grants, roles, and OPA policy |
Do not use a human password as a workload credential. Do not assign a secret to a public client. Do not expose one workload's identity material to another process. An account reserved for an agent is an agent principal, not a human principal, even when it authenticates through the same public CLI client.
A person should begin at the application:
- The application redirects the browser to Keycloak.
- The person signs in using the organization's approved login methods.
- Keycloak returns the browser to an approved redirect URI.
- The application validates the session or access token.
- The application resolves current resource facts.
- OPA evaluates policy for actions that require a policy decision.
- The application enforces the decision and its own domain invariants.
Browser applications own their Keycloak client ID, redirect URIs, and session configuration. Browser automation should exercise this real redirect flow rather than calling the token endpoint directly.
Use dedicated, non-production test identities for automated browser testing. Store their credentials in an approved secret manager, load them only into the test process, and never include them in source control, screenshots, command arguments, or test reports.
A workload identity should describe where the workload runs and which service account owns it. A common Kubernetes identity format is:
spiffe://example.com/ns/<namespace>/sa/<service-account>
The standard workload flow is:
- The pod mounts the SPIFFE Workload API.
- SPIRE verifies the workload's selectors.
- The workload obtains a short-lived X.509-SVID and trust bundle.
- The identity material is renewed automatically.
- The workload establishes mutual TLS.
- Each peer validates the trust bundle and expected SPIFFE URI.
- The receiving application authorizes the exact verified identity.
When applications require certificate files, a helper may materialize them in a memory-backed volume:
/run/spire/svid/tls.crt
/run/spire/svid/tls.key
/run/spire/svid/ca.pem
Prefer direct Workload API integration where supported. Materialized SVID files must remain readable only by the owning workload and must not be copied into a secret store, image, artifact, log, or general-purpose pod.
Mutual TLS is complete only when both peers validate identity:
- validate the trust bundle;
- require a successfully authorized TLS peer;
- extract SPIFFE identities only from URI subject alternative names;
- require exactly one expected SPIFFE URI when the protocol calls for one identity;
- compare the full SPIFFE ID, not a prefix or partial path;
- reject untrusted, missing, ambiguous, or unexpected identities.
A request field or environment variable containing text that begins with
spiffe:// is not proof of workload identity.
When supported by the deployed Keycloak release, a SPIRE-managed workload can use a JWT-SVID as an OAuth client assertion. This removes the need to store a static client secret in the workload.
The high-level flow is:
- Register a dedicated Keycloak client for the workload.
- Bind the client to the exact expected SPIFFE identity and audience.
- Give the client only the grants and roles it requires.
- Obtain a short-lived JWT-SVID from the SPIFFE Workload API.
- Send the JWT-SVID as the OAuth client assertion.
- Validate the resulting token on the target API.
Feature names, protocol details, and stability may vary by Keycloak release. Treat feature enablement, client registration, audience selection, and upgrades as reviewed security changes. Enabling a server feature does not register or authorize a workload client.
Tools should use the client type that matches how they operate.
An interactive CLI uses a public Keycloak client. It may authenticate a person or a dedicated agent account and exchange approved account credentials or an interactive authorization result for a short-lived bearer, depending on the organization's supported flow.
The client ID identifies the CLI application. The authenticated account identifies the principal. Policy must distinguish a person from a dedicated agent account even when both use the same public client.
The CLI must:
- have no embedded client secret;
- read credentials from protected environment or browser interaction;
- keep tokens in memory;
- send request bodies through standard input when command arguments could expose sensitive values;
- verify the authenticated API context before performing protected work;
- clear credentials and tokens when finished.
An agent using this flow has an account-based Keycloak identity. It has not proved a SPIFFE workload identity.
An unattended integration or privileged administration agent outside the SPIRE trust domain uses a dedicated confidential client:
- provision one client per independently governed integration;
- store its secret in an approved secret manager;
- restrict grants, scopes, roles, audiences, and allowed origins;
- rotate the secret;
- never reuse a human account or public CLI client;
- migrate to federated workload identity if the integration later joins the SPIRE trust domain.
The client ID identifies the integration. The client secret proves possession of its credential. OPA should receive verified client claims, never the secret or raw bearer token.
OPA is a policy decision point. The application remains the policy enforcement point. OPA does not authenticate a person, validate an OAuth bearer, perform a SPIFFE handshake, or execute the protected business operation.
The standard policy flow is:
- Authenticate the actor on the owning API surface.
- Resolve current resource state from the system of record.
- Construct policy input on the server.
- Request one named decision from the expected OPA package.
- Allow only an explicit positive decision.
- Deny on missing configuration, outage, timeout, malformed output, or any non-allow result.
- Enforce the decision locally.
- Record the authenticated identity, action, resource, decision, and policy version in the audit trail.
Use a stable envelope that distinguishes people, OAuth clients, and workloads:
{
"input": {
"action": "document:publish",
"principal": {
"type": "human",
"subject": "verified-subject",
"username": "verified-username",
"client_id": null,
"roles": ["publisher"],
"auth_source": "keycloak"
},
"workload": {
"spiffe_id": "spiffe://example.com/ns/content/sa/publisher-api"
},
"resource": {
"type": "document",
"id": "document-123",
"owner_id": "user-456",
"version": "17"
},
"context": {
"request_id": "request-789"
}
}
}Policy input must contain only server-verified or system-of-record facts:
- Keycloak subject, client ID, scopes, and roles come from a validated token or session.
- A SPIFFE ID comes from the verified mutual-TLS peer or a trusted, explicitly pinned workload configuration.
- Ownership, version, and resource status come from the authoritative domain service.
- Action names are server-defined constants, not arbitrary client strings.
Use principal types such as human, agent_account, oauth_client, and
workload. Omit facts that do not apply rather than inventing values or
representing an agent, client, or service as a person.
Authorization succeeds only when OPA returns the exact approved decision shape. The enforcement point must deny when:
- the OPA endpoint or policy path is not configured;
- OPA cannot be reached;
- the request times out;
- the response is not successful;
- the result is missing or malformed;
- the policy explicitly denies the action.
An OPA outage must not silently fall back to role-only authorization.
Rego source should be version controlled, reviewed, and deployed through the managed OPA release workflow. Production policy must not be patched manually.
Each rollout should identify:
- policy package and decision path;
- source revision and policy bundle version;
- owning application enforcement point;
- input schema version;
- expected allow and deny cases;
- fail-closed behavior;
- rollback procedure.
High-impact operations often cross both identity boundaries:
- Keycloak authenticates the person.
- OPA decides whether that person may request the operation.
- The API persists an immutable action plan.
- A dedicated SPIFFE workload authenticates to an isolated executor gateway.
- The gateway authorizes the exact workload identity.
- A short-lived, single-use capability binds the approved plan to execution.
- The system records the human, workload, policy decision, and durable outcome.
The executor must not receive the person's browser token or provider credentials unless the owning business workflow explicitly requires and protects them.
For advisory or model-backed work:
- Keycloak authenticates the person.
- OPA evaluates the person's roles, resource ownership, action, and relevant workload identity.
- The API and isolated worker authenticate each other with SPIFFE mTLS.
- The worker returns a constrained result.
- The person explicitly applies or commits the result through the owning application workflow.
The worker should have only the network, model, and data access necessary for its single responsibility.
Never print, persist, commit, or place the following in command arguments, logs, screenshots, validation artifacts, support tickets, or issue comments:
- passwords;
- bearer or refresh tokens;
- OAuth client secrets;
- private keys;
- execution capabilities;
- SVID key material.
Use short lifetimes, automatic rotation, narrow file permissions, memory-backed storage, exact audiences, and least-privilege grants.
Policy engines and relationship engines answer different questions, and a system that conflates them accumulates authorization logic in both places.
A policy engine answers: is this request permissible in this context? A relationship engine answers: does this subject have this relation to this specific object?
Both must allow. Neither subsumes the other.
Ask of any authorization question:
Can I answer this without storing who-shared-what-with-whom?
- Yes — a policy engine such as OPA. The facts arrive in the token, the TLS handshake, or the system of record.
- No — a relationship engine implementing the Zanzibar model, such as OpenFGA. The answer requires persisted relationship tuples.
Attribute-shaped questions include role membership, tenant equality, data classification, clock comparisons, and verified workload identity. All of those are answerable from facts already present at decision time.
Relationship-shaped questions include per-object sharing, folder or project inheritance, nested group membership, and delegated administration. Those require stored relationships between objects.
One capability separates the two decisively. A policy engine evaluates a supplied input document and holds no relationship state, so it cannot answer:
List every object this subject may read.
That is a reverse query, and it requires traversing stored relationships. It is the access-control problem behind document-level retrieval permissions: when a corpus carries per-user access, retrieval must pre-filter to the set a subject may see rather than filter after the fact.
Adopt one when any of the following becomes a product requirement, and not before:
- per-object sharing between individual subjects;
- hierarchy or inheritance, where container permissions flow to contents;
- reverse queries over a large object set;
- document-level permissions for retrieval-augmented workloads;
- nested groups or delegated administration.
A tuple store that must stay consistent with the system of record is a standing liability. Adopting it before one of the above exists means carrying that liability with no offsetting benefit. When the requirement does arrive, the answer is usually both engines, not a replacement: the policy engine gates the request context, the relationship engine gates the specific object.
Three relationship patterns cannot be expressed as attribute comparisons, and each deserves an explicit test:
| Pattern | What it proves |
|---|---|
| Ownership implies access | a computed relation, not a stored grant |
| Container inheritance | access arrives with no direct grant on the object |
| Group through container | membership resolves transitively |
Container inheritance is the load-bearing case. Removing a subject from a container revokes every object inside it in a single write. Include a subject with no relationships at all, and assert it reaches nothing, so default-deny is demonstrated rather than assumed.
A relationship engine that cannot be reached must deny. Unreachable service, server error, and malformed response are all denials. An authorization component whose failure mode is "allow" is not an authorization component.
Authorization answers whether an operation may proceed. It does not answer who performed it, what it consumed, or who is accountable for the cost. Multi-tenant API platforms need both, and attribution is only trustworthy when the identity it attaches to cannot be forged.
The same verified identities used for authorization make a usage record defensible:
| Attributed fact | Source | Caller can assert it? |
|---|---|---|
| Subject | verified token subject | No — signed by the issuer |
| Workload identity | SVID subject alternative name from the TLS handshake | No — private key never leaves the workload |
| Tenant | derived from the verified token issuer | No — issuer is validated |
| Request identifier | generated server-side | Only if it parses as a UUID |
Without that chain, a usage record is a self-reported number. With it, a cost cannot be attributed to the wrong party or disowned afterwards.
Attribution must key on the stable subject identifier from the token, not the username or email. Those change, and a billing record whose key mutates cannot be reconciled historically. Carry the human-readable name alongside for reporting only.
A single request identifier, generated as a UUID, links the authorization decision, the usage record, and application logs:
request id ──► authorization audit which decision permitted this?
──► usage ledger what did it consume, and who pays?
──► application logs what happened?
Two rules follow. A timestamp is unsuitable, because it collides under concurrency and cannot safely key a financial record. And a caller-supplied request identifier should be honoured only when it parses as a UUID, otherwise a client can forge or collide the key that spend is attributed to.
Use integer arithmetic. Floating point accumulates representation error under repeated summation, and a chargeback total that disagrees with itself is worse than no total.
Choosing the unit matters more than it appears. Providers publish token prices per million tokens, so a single token costs a very small fraction of a cent:
a rate of $0.10 per million tokens = 1e-7 USD per token
expressed in micro-dollars: 0.1 → truncates to zero
A unit that truncates to zero bills that traffic as free. Choose a unit fine enough that every published rate divides exactly — ten times finer than a nanodollar leaves ample headroom — and store rates in the vendor's own denominator so a price can be transcribed from a pricing page without intermediate arithmetic.
Also model each priced dimension separately. Cached and uncached input, cache writes at differing lifetimes, and cache reads are distinct rates, not multipliers on a single input price. Collapsing them misprices cache-heavy traffic in both directions.
Resolve rates through an interface satisfied by a single authoritative pricing implementation. Two rate tables that drift produce two defensible-looking invoices that disagree, and reconciling them after the fact is far more expensive than the indirection.
Apply the same discipline to unpriced models: a model with no rate must be treated as expensive, never free, or a pricing gap becomes a route to unmetered consumption.
Write a usage record for every outcome, including denials. Increment the attempt and denial counters; leave the cost counter untouched. Billing then reflects only authorized consumption, while abuse and misconfiguration remain visible in the same ledger. Dropping denials hides precisely the behavior worth investigating.
Usage is append-only and write-heavy — one record per operation. That suits a key-value store with a tenant-scoped partition key:
usage record partition = tenant#{tenant}#subject#{subject}
sort = timestamp#{request id}
monthly rollup partition = tenant#{tenant}#rollup
sort = month#{YYYY-MM}
secondary index for request-identifier lookup
Three properties follow from that layout:
- Tenant isolation is structural. The tenant is the first key segment, so a query cannot cross tenants even when application code omits a filter.
- Counters update atomically. Server-side atomic addition means concurrent writers cannot lose increments, which a read-modify-write of a single row risks under contention.
- Reporting cost is flat. Monthly accounting reads one rollup per tenant, so the cost of reporting does not grow with operation volume.
Check quotas against recorded spend before admitting an operation, so the ledger governs behavior rather than merely describing it afterwards. Gate order matters: evaluate tenant model restrictions, then request-size ceilings, then the policy decision, then budget. Record every refusal with a distinct reason.
Budget enforcement of this shape is eventually consistent — concurrent operations can each observe the same pre-call total and marginally overshoot. A hard ceiling requires a conditional write against the counter. State that limit explicitly rather than implying a guarantee the system does not provide.
If a figure is derived rather than metered, mark it. Persist a boolean and a source string on the record, not merely a log line, so a modelled number can never be read as a metered one. A report that cannot separate the two is a reporting defect.
Validate each boundary independently:
- Human browser: the real Keycloak redirect completes and the protected application renders.
- OAuth bearer: the owning API surface reports the expected authenticated subject or client.
- SPIFFE mTLS: the exact protected listener returns a normal response after trust-bundle and peer-identity validation.
- OPA authorization: the intended package receives server-derived facts, allows the approved case, and denies the unauthorized case.
- Failure behavior: authentication and authorization fail closed during invalid credentials, unexpected identities, OPA outage, timeout, and malformed policy output.
- Domain outcome: the authoritative system records the exact durable result separately from identity and policy evidence.
A successful authentication on one API surface does not prove another surface accepts the same credential. A successful Keycloak login does not prove SPIFFE mTLS. A successful SPIFFE handshake does not prove OPA authorization. An OPA allow decision does not prove the business operation completed.
Extend the per-boundary validation discipline to both areas:
- Relationship decision: the engine grants access through an inherited or computed relation with no direct grant, and denies a subject with no relationship path.
- Reverse query: the returned object set matches exactly, including the empty set for an unrelated subject.
- Relationship engine failure: unreachable, error, and malformed responses all deny.
- Cost arithmetic: published rates divide exactly, sub-cent rates do not truncate to zero, and repeated summation shows no drift.
- Tenant isolation: a query scoped to one tenant cannot return another tenant's records, verified from the key rather than the filter.
- Refused attempts: denials appear in attempt counters and are absent from cost counters.
- Quota enforcement: an exhausted budget refuses the next operation, and the refusal does not itself change billed spend.
An allow decision from a policy engine does not prove the relationship engine agrees. A relationship grant does not prove the operation was billed. A recorded cost does not prove the underlying operation completed.
- Identify whether the actor is a person, agent account, OAuth client, or workload.
- Select the matching Keycloak or SPIFFE authentication flow.
- Define the exact principal, action, resource, and context sent to OPA.
- Resolve policy facts from verified identity and authoritative data.
- Deny when authentication or policy evaluation is unavailable.
- Use exact SPIFFE identity matching and least-privilege OAuth grants.
- Keep credentials and SVID material out of logs and durable artifacts.
- Deploy Rego through a managed, versioned policy workflow.
- Validate authentication, authorization, and business outcome separately.
- Preserve durable audit evidence without storing sensitive credentials.
- Decide whether each authorization question is attribute-shaped or relationship-shaped, and route it to the matching engine.
- Introduce a relationship engine only once sharing, inheritance, or reverse queries are an actual requirement.
- Deny when a relationship engine is unreachable, errors, or returns a malformed response.
- Key usage attribution on the stable subject identifier, never the username.
- Generate a UUID request identifier server-side and use it to join the authorization decision, the usage record, and logs.
- Represent money as integers in a unit fine enough that no published rate truncates to zero.
- Resolve rates through one authoritative pricing implementation, and treat an unpriced model as expensive rather than free.
- Record refused attempts without billing them.
- Scope usage storage by tenant in the partition key, not by a filter.
- Check quotas against recorded spend before admitting an operation, and state plainly whether the ceiling is hard or eventually consistent.
- Persist a flag distinguishing measured figures from modelled ones.