Set and check (and document) a global label_value_length_limit (and potentially label_name_length_limit) based on Prometheus issue #16525. Currently, Prometheus defaults to no limit (0) for these settings. However, due to the encoding of the Labels data structure, a label name or value longer than 16MB will cause Prometheus to crash. The goal is to set a safe default limit globally and check this limit before constructing the Labels data structure in the scrape loop, as well as checking it in other relevant areas like the remote-write receiver, label_replace, and label_join.
Warning
Changing the default limits will be a breaking change for any users currently relying on extremely large labels (which is an anti-pattern but still possible). We need to decide on the default global limit. The issue discussion suggests 1MB or 200 bytes. I am proposing 2048 bytes as a safe, practical default.
Important
Please review the following design decisions and select your preferred option for each:
| Option | Limit Value | Trade-offs |
|---|---|---|
| A (Recommended) | 1MB (1,048,576 bytes) | Fixes the 16MB crash bug while safely accommodating edge cases (e.g., users storing bitmaps in labels). |
| B | 2KB (2,048 bytes) | Very safe and practical for 99% of normal workloads, minimizes memory risk, but could break existing edge-case configurations. |
| C | 0 (No limit) | Maintains 100% backwards compatibility by default but leaves the 16MB crash vulnerability exposed until users manually set a limit. |
| Option | Scope | Trade-offs |
|---|---|---|
| A (Recommended) | Apply to both label_name_length_limit and label_value_length_limit |
Enforces safety uniformly, as the Labels constructor can crash if either the name or value exceeds 16MB. |
| B | Apply only to label_value_length_limit |
Strictly adheres to the issue title, but leaves the system vulnerable to crashes from oversized label names. |
| Option | Behavior | Trade-offs |
|---|---|---|
| A (Recommended) | Drop sample, increment a metric, and return HTTP 400 (Bad Request) | Explicitly informs the sender that their sample was rejected due to an oversized label, but this could block the sender's remote-write queue if they retry indefinitely. |
| B | Drop sample, increment a metric, and return HTTP 200 (OK) | Allows the sender's queue to continue moving seamlessly, but the sender will silently lose data without receiving an explicit error code. |
Summary of what will change in the configuration defaults.
- Change
DefaultGlobalConfigto include a defaultLabelNameLengthLimit(e.g., 2048) andLabelValueLengthLimit(e.g., 2048). - Update the YAML marshaling/unmarshaling to apply these defaults properly.
- Document the new default limits in the Prometheus configuration documentation.
Summary of how limits will be checked before constructing the Labels data structure.
- Currently,
verifyLabelLimitstakes alabels.Labelsobject, meaning the data structure is already constructed (potentially causing a crash if it's over 16MB). - We need to modify the parsing step or
labels.ScratchBuilderto enforce limits during parsing or right beforeBuilder.Labels()is called. - A potential approach is to pass limits to the
textparse.Parseror check lengths inside theScratchBuilderbefore callingb.Labels().
Summary of how limits will be checked in the remote-write receiver.
- Check
LabelNameLengthLimitandLabelValueLengthLimiton incoming samples in the remote-write receive path. - Reject the request or drop the sample if it exceeds the limit.
Summary of how limits will be checked in PromQL functions.
- Update
label_replaceandlabel_jointo validate the resulting label value length against the global limits (if applicable) or a hardcoded maximum safe limit (since global config is not directly accessible in PromQL evaluation, we may need to use a package-level variable or thread the limit down to the engine).
make testto run all unit tests.- Add specific unit tests for
scrape,remote-write,label_replace, andlabel_jointo verify they correctly reject or drop labels that exceed the limits without crashing.
- Run a local Prometheus instance with the new default limit.
- Attempt to scrape a target exposing a metric with a 20MB label value to verify it is gracefully dropped/logged instead of crashing.