Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Select an option

  • Save bartoszmajsak/b949c31c7488f31b1013e9f8e8af94b4 to your computer and use it in GitHub Desktop.

Select an option

Save bartoszmajsak/b949c31c7488f31b1013e9f8e8af94b4 to your computer and use it in GitHub Desktop.
LLMInferenceService: Pipeline-parallel well-known configs gap
# Pipeline-Parallel Well-Known Configs Gap
## Summary
The LLMInferenceService API accepts pipeline-parallel configurations (`spec.parallelism.pipeline > 0`),
the controller has code paths to handle them, but no well-known config YAML files are shipped.
This means a user can create a valid-looking spec that passes webhook validation but fails
at reconciliation time with `ConfigNotFound`.
## Evidence
**API validation accepts it:**
- `pkg/apis/serving/v1alpha2/llm_inference_service_types_func.go:57` - `IsPipelineParallel()` returns true when `spec.parallelism.pipeline > 0`
- `pkg/apis/serving/v1alpha2/llm_inference_service_validation.go:262` - validation allows `worker` when either `IsDataParallel()` or `IsPipelineParallel()` is true
**Controller has code paths:**
- `pkg/controller/v1alpha2/llmisvc/config_merge.go:56-58` - config name constants defined for pipeline-parallel variants
- `pkg/controller/v1alpha2/llmisvc/config_merge.go:85-87` - names registered in the Presets list
- Decision tree in `combineBaseRefsConfig()` handles the `IsPipelineParallel()` branch
**But no configs are shipped:**
- `config/llmisvcconfig/` has no pipeline-parallel YAML files
- The names are in the Presets list but NOT in `WellKnownDefaultConfigs` (lines 92-101)
## Impact
A user who sets `spec.parallelism.pipeline: 2` with `spec.worker: {}` will:
1. Pass webhook validation
2. Hit `ConfigNotFound` during reconciliation
3. See `PresetsCombined: False` with no clear indication that pipeline-parallel configs don't exist
## Options
1. **Ship the configs** - create `config-llm-worker-pipeline-parallel.yaml` (and prefill/decode variants)
2. **Reject at validation** - add a webhook check that rejects `IsPipelineParallel()` until configs are shipped
3. **Document the limitation** - if this is intentionally deferred, document that users must provide their own config via `baseRefs`
Option 2 is the safest short-term fix - fail early with a clear message instead of failing late with a generic `ConfigNotFound`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment