Custom HTTP Headers for Cloud Foundry HTTPS drains

RFC_Custom_Drain_Headers_CF.md

RFC: Support Custom HTTP Headers for Syslog HTTPS Drains

Status

Proposed

Proposal Date

2025-06-21

Authors

• ZPascal

Summary

This proposal introduces support for specifying custom HTTP headers in HTTPS syslog drains for Cloud Foundry's Loggregator system. It enables use cases such as integration with third-party logging systems that require authentication via bearer tokens or custom headers, most notably Splunk's Log Observer Connect / HEC (HTTP Event Collector) and similar modern log routing solutions.

Motivation

Modern logging and Observability platforms increasingly rely on HTTP-based ingestion with authentication mechanisms such as bearer tokens or custom headers. While Loggregator already supports HTTPS drains, there is currently no way to add custom headers, making integration with services like:

• Splunk HEC
• Datadog HTTPS ingestion
• New Relic Logs
• Elastic ingest endpoints

It is difficult or impossible without proxies or sidecar-based workarounds.

Supporting custom headers allows:

• Native integration with services requiring API tokens or custom auth schemes
• Use of standardized mechanisms like Authorization: Bearer <token>
• Elimination of custom proxies or sidecar containers
• A flexible and extensible pattern for future integrations

Approach

We propose the following changes:

1. Extend the syslog drain binding specification to allow a headers field containing key-value pairs.
2. Modify the Metron Agent / Syslog Adapter components that handle HTTPS drains to inject the custom headers into outbound HTTPS requests.
3. Add validation logic to sanitize and prevent forbidden headers (e.g. Host, Content-Length, etc.).
4. Include the headers field in drain metadata exposed to platform operators and log system integrators.

Example:

{
  "drain_url": "https://http-input.splunkcloud.com:443/services/collector/event",
  "headers": {
    "Authorization": "Bearer abc123xyz",
    "X-Splunk-Request-Source": "cloudfoundry"
  }
}

RFC_Custom_HTTP_Headers_For_Syslog_HTTPs_Drains.md

Alternatives

• Use an intermediary proxy that adds headers on behalf of Cloud Foundry: adds complexity and operational overhead.
• Rely on Basic Auth in the URL: insecure and not compatible with modern services.

Impact

Enhances the flexibility and usefulness of HTTPS syslog drains.

Simplifies integration with external logging systems — especially Splunk, which requires an Authorization: Bearer header.

No breaking changes to existing drain configurations.

Minimal performance impact due to headers being set once per HTTPS client.

Prior Art

• Splunk’s HEC HTTP Event Collector requires the use of a bearer token in headers.
• Fluent Bit and Vector support header injection in HTTP outputs.

Unresolved Questions

• How will sensitive values (e.g. bearer tokens) be managed securely? Should integration with CredHub be considered?
• What audit/logging capabilities should be added to monitor or restrict the use of custom headers?

Future Work

Allow secure secret references for header values via CredHub or environment variable substitution.

Support templating headers with per-app metadata (e.g., app name, org, space).

Build integrations with known external logging services (Splunk, Datadog, etc.) as example recipes or docs.