Send development kibana telemetry directly to development ES over OTLP

kibana es otlp setup.md

These instructions can get your local Kibana development environment set up to send logs, metrics, and traces directly to a development Elasticsearch cluster using the Elasticsearch OTLP/HTTP endpoint.

You might want to do this in development so you can have basic otel data without having to run a collector.

These instructions are for running locally in Serverless mode but most things apply for non-serverless also.

This is not a supported configuration and some parts of the Elastic Observability UI will not work as expected.

Elasticsearch

This runs ES in Docker on https://localhost:9200:

yarn es serverless --esProjectType observability --productTier complete --no-uiam --kill --esArgs='logger.level=WARN

Create an API key:

curl -sk --cacert src/platform/packages/shared/kbn-dev-utils/certs/ca.crt \
  -u elastic_serverless:changeme \
  -X POST https://127.0.0.1:9200/_security/api_key \
  -H 'Content-Type: application/json' \
  -d '{"name":"kibana-otlp","role_descriptors":{"otlp":{"indices":[{"names":["metrics*","logs*","traces*"],"privileges":["create_doc","auto_configure"]}]}}}' \
  | jq -r '"Authorization=ApiKey " + .encoded'

Save that output to use in the OTEL_EXPORTER_OTLP_HEADERS environment variable when you start Kibana.

Kibana

Add these to your kibana.dev.yml:

logging:
  appenders:
    otel:
      type: otel
      protocol: proto
      url: https://localhost:9200/_otlp/v1/logs
  root:
    appenders: [default, otel]

telemetry.metrics:
  enabled: true
  exporters:
    - proto:
        url: https://localhost:9200/_otlp/v1/metrics

telemetry.tracing:
  enabled: true
  exporters:
    - proto:
        url: https://localhost:9200/_otlp/v1/traces

Start Kibana:

env OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey <YOUR API KEY>" \
    NODE_EXTRA_CA_CERTS=src/platform/packages/shared/kbn-dev-utils/certs/ca.crt \
yarn serverless-oblt --http2 --no-uiam

Once it's up, these queries should return results in Discover:

FROM traces-generic.otel-default | WHERE service.name == "kibana"
FROM logs-generic.otel-default | WHERE service.name == "kibana"
TS metrics-generic.otel-default | WHERE service.name == "kibana"