portkey-networking-issues

overview.md

In the ancient times ...

We had two kinds of traffic being handled as:

  graph TB
  subgraph traffic-from-outside
  GCP-LB-->|"restrict to specific IP addresses"|MetalLB
  end
  subgraph on-prem-boundary
  MetalLB-->|"bind to service via NodePort"|LLMGateway
  NYU-Users-->|OpenShift-Route| LLMGateway
  end

Loading

Note that the end-user URL is https://ai-gateway.apps.cloud.rt.nyu.edu.

The present connundrum:

By using the gateway api we modernised ingress, but also "gated" out traffic from nyu users!

  graph TB 
  subgraph traffic-from-outside
  GCP-LB-->|"restrict to specific IP addresses"|MetalLB
  end
  subgraph on-prem-boundary
  MetalLB-->|"restrict to specific IP address"|Gateway
  Gateway-->|"via Gateway HTTP route, strip prefix"|LLMGateway-prod
  Gateway-->|"via Gateway HTTP route, strip prefix"|LLMGateway-dev
  NYU-Users-.->|"Route blocked by NetworkPolicy"| LLMGateway-prod
  end

Loading

HTTPRoute is set to https://ai-gateway.cloud.rt.nyu.edu/{prod,dev} with a 301 redirect if the prefix is missing. The gateway strips the prefix before redirecting a request to prod/dev. Note that the new URL is .cloud.rt.nyu.edu rather than apps.cloud.rt.nyu.edu.

If a user were to attempt to hit https://ai-gateway.cloud.rt.nyu.edu/prod/v1, Envoy throws a RBAC: access denied error. Our users however will attempt to route their requests to https://ai-gateway.apps.cloud.rt.nyu.edu.