Skip to content

Instantly share code, notes, and snippets.

@nerdalert
Last active July 24, 2026 15:45
Show Gist options
  • Select an option

  • Save nerdalert/586fb0099fd9d4edeee40cfe7ce1718e to your computer and use it in GitHub Desktop.

Select an option

Save nerdalert/586fb0099fd9d4edeee40cfe7ce1718e to your computer and use it in GitHub Desktop.
## Done vs Not Done

Done:

- Praxis has the core APIs needed for trusted pre-read routing mutations.
- The llm-d `ext_proc` compatibility filter has been moved into Praxis AI.
- Praxis AI can register the llm-d `ext_proc` filter behind a feature.
- The filter can open and manage EPP-style gRPC processor streams.
- Request-header and full-duplex request-body routing paths are implemented.
- EPP-returned endpoint mutations can drive `endpoint_selector`.
- Client-spoofed destination headers are ignored.
- Trusted routing headers are stripped before reaching the backend.
- EPP/processor failure paths return configured error status.
- Simulator-backed tests cover the basic Praxis AI to mock EPP to backend path.

Not done:

- Real llm-d scheduler behavior is not proven yet.
- Cache-aware routing is not proven yet.
- `InferencePool` / `InferenceModel` traffic splitting is not proven yet.
- Disaggregated prefill/decode routing is not proven yet.
- GPU-backed vLLM behavior is not proven yet.
- A canonical upstream mock EPP image still needs to be owned/published by Praxis AI.
- Grid still needs a smoke test that consumes the AI-owned mock EPP support.
- Environment-backed Tier 2 tests still need real llm-d infrastructure and observable scheduler evidence.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment