HTTPS on xKS Gateway — Manual Workaround for RHOAI 3.4 TP (pre-PR #90)

HTTPS on xKS Gateway — Manual Workaround (RHOAI 3.4 TP)

The rhai-on-xks helm chart creates the inference gateway with HTTP only (port 80). These steps add HTTPS manually after the chart is installed.

Verified working on AKS (May 28–29, 2026).

Which case should I use?

Case	Use case	Trusted by browsers?	`curl -k` needed?	Extra setup?
Case 1	Dev/test clusters, quick HTTPS enablement, internal APIs where clients can skip cert verification	No	Yes	No
Case 2	Custom domain for internal teams, branded URLs, internal services OK with self-signed CA	No	Yes	DNS record
Case 3	Enterprise / cloud production — existing corporate CA, cloud-managed certs (ACM, Google-managed SSL, Azure Key Vault), or wildcard certs	Yes	No	Existing cert files only
Case 4	Public-facing demo/PoC endpoints, no existing cert infrastructure, need a free trusted cert quickly (Let's Encrypt)	Yes	No	DNS + cert-manager patching (see below)

Most enterprise and cloud customers will use Case 3 — on AKS, EKS, GKE, or CoreWeave you typically bring your own cert from the cloud provider's certificate service or your corporate PKI. No cert-manager ACME flow needed, no extra flags, simplest path to production-trusted HTTPS.

Case 4 (Let's Encrypt / ACME) is for situations where you need a trusted cert but have no existing cert infrastructure. It requires additional cert-manager configuration that is not officially supported by the RHAI operator today. See the bold prerequisites in Case 4.

Prerequisites

rhai-on-xks helm chart deployed
cert-manager running with rhai-ca-issuer ClusterIssuer available
kubectl configured for the target cluster

Verify:

kubectl get clusterissuer rhai-ca-issuer
kubectl get gateway inference-gateway -n redhat-ods-applications

Background: RHAI certificate chain

The rhai-on-xks helm chart (via the cloud manager CR) auto-provisions an internal PKI chain:

rhai-ca (Certificate, cert-manager namespace) — self-signed root CA (isCA: true), created by the opendatahub-selfsigned-issuer ClusterIssuer. Owned by the AzureKubernetesEngine CR.
rhai-ca-issuer (ClusterIssuer) — uses the rhai-ca secret as its CA signing key.
The issuer automatically signs these downstream certificates:
- llmisvc-webhook-server (redhat-ods-applications) — TLS for the LLMInferenceService admission webhook
- rhai-operator-webhook-cert (redhat-ods-operator) — TLS for the RHAI operator webhook

The gateway TLS certificate (inference-gateway-cert) is not auto-provisioned — it must be created manually (Cases 1, 2, 4) or supplied as an existing secret (Case 3). All cases that use rhai-ca-issuer produce self-signed certificates, so external clients must skip verification (curl -k) or trust the CA.

Case 1: Internal HTTPS (self-signed, no custom domain)

Use case: Dev/test environments, internal cluster access, quick HTTPS enablement. Clients access the gateway by IP address and accept self-signed certificates (curl -k or verify=False in code).

HTTPS with internal CA. External clients must use curl -k (skip verification).

Step 1: Create certificate

kubectl apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: inference-gateway-cert
  namespace: redhat-ods-applications
spec:
  secretName: inference-gateway-cert-secret
  issuerRef:
    name: rhai-ca-issuer
    kind: ClusterIssuer
    group: cert-manager.io
  dnsNames:
    - "*.redhat-ods-applications.svc.cluster.local"
    - "*.redhat-ods-applications.svc"
EOF

Step 2: Wait for secret

kubectl get certificate inference-gateway-cert -n redhat-ods-applications
# Wait for READY = True

Step 3: Add HTTPS listener to gateway

kubectl patch gateway inference-gateway -n redhat-ods-applications --type=json -p='[
  {"op": "add", "path": "/spec/listeners/-", "value": {
    "name": "https",
    "port": 443,
    "protocol": "HTTPS",
    "allowedRoutes": {"namespaces": {"from": "All"}},
    "tls": {
      "mode": "Terminate",
      "certificateRefs": [{"group": "", "kind": "Secret", "name": "inference-gateway-cert-secret"}]
    }
  }}
]'

Step 4: Azure LB health probe (AKS only)

On AKS, the Azure Load Balancer needs a health probe annotation for port 443, otherwise external traffic on port 443 is silently dropped:

kubectl annotate svc inference-gateway-istio -n redhat-ods-applications \
  "service.beta.kubernetes.io/port_443_health-probe_protocol=tcp"

Without this annotation, HTTPS works from inside the cluster but times out externally.

Step 5: Test

GATEWAY_IP=$(kubectl get gateway inference-gateway -n redhat-ods-applications -o jsonpath='{.status.addresses[0].value}')

# HTTPS (skip verification — self-signed)
curl -k https://${GATEWAY_IP}/redhat-ods-applications/<model-name>/v1/models

# Chat completion over HTTPS
curl -k https://${GATEWAY_IP}/redhat-ods-applications/<model-name>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "<model-name>", "messages": [{"role": "user", "content": "What is Kubernetes?"}], "max_tokens": 50}'

# HTTP still works
curl http://${GATEWAY_IP}/redhat-ods-applications/<model-name>/v1/models

Case 2: Custom domain with internal CA

Use case: Internal teams that want a branded URL (e.g., inference.mycompany.com) instead of a raw IP address. Still uses the internal rhai-ca-issuer, so the certificate is self-signed. Useful for internal DNS, nip.io testing, or environments where clients can trust the internal CA or accept -k.

HTTPS with a custom DNS name, still using the internal rhai-ca-issuer. External clients must trust the internal CA or use curl -k.

Step 1: Set up DNS

Point your domain to the gateway LoadBalancer IP:

GATEWAY_IP=$(kubectl get gateway inference-gateway -n redhat-ods-applications -o jsonpath='{.status.addresses[0].value}')
echo "Create DNS A record: inference.mycompany.com -> ${GATEWAY_IP}"

For quick testing without DNS setup, use nip.io:

# Resolves automatically to your gateway IP — no DNS record needed
curl -k https://inference.${GATEWAY_IP}.nip.io/redhat-ods-applications/<model-name>/v1/models

Step 2: Create certificate with custom hostname

kubectl apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: inference-gateway-cert
  namespace: redhat-ods-applications
spec:
  secretName: inference-gateway-cert-secret
  issuerRef:
    name: rhai-ca-issuer
    kind: ClusterIssuer
    group: cert-manager.io
  dnsNames:
    - "inference.mycompany.com"
    - "*.redhat-ods-applications.svc.cluster.local"
    - "*.redhat-ods-applications.svc"
EOF

Step 3: Add HTTPS listener

Same as Case 1, Step 3.

Step 4: Azure LB health probe (AKS only)

Same as Case 1, Step 4.

Step 5: Test

curl -k https://inference.mycompany.com/redhat-ods-applications/<model-name>/v1/models

Case 3: Bring your own certificate

Use case: Enterprise environments with an existing corporate CA, wildcard certificates, cloud-managed certs (AWS ACM, Google-managed SSL, Azure Key Vault), or certs purchased from a commercial CA (DigiCert, Sectigo, etc.). This is the most common case for production enterprise deployments. No cert-manager Certificate resource is needed — you supply the TLS secret directly.

Verified on AKS (May 29, 2026).

Step 1: Prepare your certificate

Choose Option A if you already have cert files, or Option B to generate a test cert.

Option A: You have a certificate from your corporate CA or cloud provider

You should have two files:

tls.crt — the certificate (PEM format), must cover the hostname clients will use
tls.key — the private key (PEM format)

Create the TLS secret:

kubectl create secret tls inference-gateway-cert-secret \
  --cert=tls.crt \
  --key=tls.key \
  -n redhat-ods-applications

Option B: Generate a test certificate using the cluster's own CA

If you don't have a corporate cert yet, generate one signed by the cluster's rhai-ca for testing. Clients will need --cacert or -k since rhai-ca is self-signed.

# 1. Export the cluster CA cert and key
kubectl get secret rhai-ca -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/rhai-ca.crt
kubectl get secret rhai-ca -n cert-manager -o jsonpath='{.data.tls\.key}' | base64 -d > /tmp/rhai-ca.key

# 2. Get the gateway IP for the certificate hostname
GATEWAY_IP=$(kubectl get gateway inference-gateway -n redhat-ods-applications -o jsonpath='{.status.addresses[0].value}')

# 3. Generate a private key and certificate signing request (CSR)
openssl req -new -newkey rsa:2048 -nodes \
  -keyout /tmp/byo-gateway.key \
  -subj "/CN=inference.${GATEWAY_IP}.nip.io" \
  -out /tmp/byo-gateway.csr

# 4. Sign the certificate with the cluster CA (valid for 365 days)
openssl x509 -req -in /tmp/byo-gateway.csr \
  -CA /tmp/rhai-ca.crt -CAkey /tmp/rhai-ca.key -CAcreateserial \
  -out /tmp/byo-gateway.crt -days 365 \
  -extfile <(echo "subjectAltName=DNS:inference.${GATEWAY_IP}.nip.io,DNS:*.redhat-ods-applications.svc.cluster.local")

# 5. Create the TLS secret
kubectl create secret tls inference-gateway-cert-secret \
  --cert=/tmp/byo-gateway.crt \
  --key=/tmp/byo-gateway.key \
  -n redhat-ods-applications

Step 2: Add HTTPS listener

Same as Case 1, Step 3.

Step 3: Azure LB health probe (AKS only)

Same as Case 1, Step 4.

Step 4: Test

# Option A: corporate/public CA cert — no -k needed
curl https://inference.mycompany.com/redhat-ods-applications/<model-name>/v1/models

# Option B: self-signed test cert — use --cacert to trust the cluster CA, or -k to skip
GATEWAY_IP=$(kubectl get gateway inference-gateway -n redhat-ods-applications -o jsonpath='{.status.addresses[0].value}')
curl --cacert /tmp/rhai-ca.crt https://inference.${GATEWAY_IP}.nip.io/redhat-ods-applications/<model-name>/v1/models

Case 4: Let's Encrypt / ACME (public HTTPS, no existing cert infrastructure)

Use case: Public-facing demo/PoC endpoints, startups, or situations where you need a browser-trusted certificate but have no existing cert infrastructure or corporate PKI. Uses Let's Encrypt (or another ACME CA) to issue a free, publicly trusted certificate. No curl -k needed.

⚠️ This case requires additional cert-manager configuration that is not officially supported by the RHAI operator today. The --enable-gateway-api flag needed for the ACME HTTP-01 solver is not exposed via the odh-gitops Helm chart or the CertManager CR's supported overrideArgs. The workaround below uses unsupportedConfigOverrides, which Red Hat marks as unsupported. Most enterprise customers should use Case 3 (bring your own cert) instead.

Requires a real domain you own, pointing to the gateway LoadBalancer IP. The ACME HTTP-01 challenge must be reachable on port 80 from the public internet.

Verified with Let's Encrypt staging on AKS (May 29, 2026).

Prerequisites for Case 4

The xKS stack uses Gateway API (not legacy Ingress), so the ACME HTTP-01 solver must create a temporary HTTPRoute to serve the challenge. This requires two extra setup steps that are not needed for Cases 1, 2, or 3.

1. Enable Gateway API in cert-manager (unsupported workaround):

The RHAI cert-manager operator does not enable Gateway API by default. The --enable-gateway-api flag is not listed among the officially supported overrideArgs for the OpenShift cert-manager operator. Direct deployment patches are reverted by the operator, so use unsupportedConfigOverrides on the CertManager CR:

kubectl patch certmanager cluster --type=merge -p='{
  "spec": {
    "unsupportedConfigOverrides": {
      "controller": {
        "args": ["--enable-gateway-api"]
      }
    }
  }
}'

Verify the flag was added (wait ~30s for pod restart):

kubectl get deployment cert-manager -n cert-manager \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | grep enable-gateway-api

Why is this needed? Without --enable-gateway-api, cert-manager cannot create HTTPRoute resources for the ACME challenge. The xKS stack has no IngressClass — it is Gateway API only — so the legacy ingress.class solver does not work. See cert-manager Gateway API discussion #7360.

2. Add pull secret to default ServiceAccount:

The ACME solver pod runs in redhat-ods-applications using the default SA but needs to pull from registry.redhat.io:

kubectl patch sa default -n redhat-ods-applications \
  -p '{"imagePullSecrets":[{"name":"rhai-pull-secret"}]}'

Step 1: Set up DNS

Same as Case 2, Step 1.

Step 2: Create Let's Encrypt ClusterIssuer

The solver uses gatewayHTTPRoute (not legacy ingress) because the xKS stack uses Gateway API with Istio. cert-manager creates a temporary HTTPRoute on the inference-gateway to serve the ACME challenge token on port 80, then cleans it up.

kubectl apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@mycompany.com
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: inference-gateway
                namespace: redhat-ods-applications
EOF

Step 3: Create certificate

kubectl apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: inference-gateway-cert
  namespace: redhat-ods-applications
spec:
  secretName: inference-gateway-cert-secret
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
    group: cert-manager.io
  dnsNames:
    - "inference.mycompany.com"
EOF

Step 4: Wait for certificate to be issued

Let's Encrypt validation takes 1-2 minutes:

kubectl get certificate inference-gateway-cert -n redhat-ods-applications -w
# Wait for READY = True

# If stuck, check the challenge and order status:
kubectl get challenges -n redhat-ods-applications
kubectl get orders -n redhat-ods-applications

ACME flow: Certificate → Order → Challenge → solver pod + temporary HTTPRoute → Let's Encrypt validates via HTTP on port 80 → Challenge valid → Certificate issued → cleanup.

Step 5: Add HTTPS listener

Same as Case 1, Step 3.

Step 6: Azure LB health probe (AKS only)

Same as Case 1, Step 4.

Step 7: Test

# Fully trusted HTTPS — no -k needed
curl https://inference.mycompany.com/redhat-ods-applications/<model-name>/v1/models

Known Issue: HuggingFace xet download hang

The RHOAI 3.4 storage-initializer image hardcodes HF_XET_HIGH_PERFORMANCE=1 which enables the HuggingFace xet transfer protocol. This protocol has known bugs causing downloads to hang at 99% (file fully downloaded but stuck as .incomplete).

Workaround — patch the deployment to disable xet after the LLMInferenceService is created:

kubectl patch deployment <name>-kserve -n <namespace> --type=json -p='[
  {"op": "add", "path": "/spec/template/spec/initContainers/0/env/-", "value": {"name": "HF_HUB_DISABLE_XET", "value": "1"}}
]'

This forces the standard HTTPS download path. Model downloads complete in seconds instead of hanging.

Cleanup

To remove HTTPS and revert to HTTP only:

# Remove HTTPS listener
kubectl patch gateway inference-gateway -n redhat-ods-applications --type=json -p='[
  {"op": "remove", "path": "/spec/listeners/1"}
]'

# Remove Azure LB annotation
kubectl annotate svc inference-gateway-istio -n redhat-ods-applications \
  "service.beta.kubernetes.io/port_443_health-probe_protocol-"

# Delete certificate and secret
kubectl delete certificate inference-gateway-cert -n redhat-ods-applications
kubectl delete secret inference-gateway-cert-secret -n redhat-ods-applications

Notes

The gateway keeps both HTTP (80) and HTTPS (443) listeners. To disable HTTP, remove the first listener.
If the RHAI operator reconciles the gateway, manual changes may be overwritten. Monitor after operator upgrades.
For allowedRoutes security, see INFERENG-7126 and PR #108.
On AKS, the Azure LB health probe annotation is critical — without it port 443 works internally but not externally.

aneeshkp/xks-https-workaround.md

Select an option

No results found

Select an option

No results found

HTTPS on xKS Gateway — Manual Workaround (RHOAI 3.4 TP)

Which case should I use?

Prerequisites

Background: RHAI certificate chain

Case 1: Internal HTTPS (self-signed, no custom domain)

Step 1: Create certificate

Step 2: Wait for secret

Step 3: Add HTTPS listener to gateway

Step 4: Azure LB health probe (AKS only)

Step 5: Test

Case 2: Custom domain with internal CA

Step 1: Set up DNS

Step 2: Create certificate with custom hostname

Step 3: Add HTTPS listener

Step 4: Azure LB health probe (AKS only)

Step 5: Test

Case 3: Bring your own certificate

Step 1: Prepare your certificate

Step 2: Add HTTPS listener

Step 3: Azure LB health probe (AKS only)

Step 4: Test

Case 4: Let's Encrypt / ACME (public HTTPS, no existing cert infrastructure)

Prerequisites for Case 4

Step 1: Set up DNS

Step 2: Create Let's Encrypt ClusterIssuer

Step 3: Create certificate

Step 4: Wait for certificate to be issued

Step 5: Add HTTPS listener

Step 6: Azure LB health probe (AKS only)

Step 7: Test

Known Issue: HuggingFace xet download hang

Cleanup

Notes