## Overview
This article describes how OpenNMS is configured to collect and store metrics from an external Kubernetes cluster using Prometheus as the data source. It covers what is collected, how the collection pipeline is structured, and how data is persisted in RRD storage for threshold evaluation.
## Collection Architecture
The Kubernetes cluster exposes Prometheus metrics on port 9090. OpenNMS Collectd connects to that endpoint and issues PromQL queries on a 30-second polling interval. Results are written as JRB files into the OpenNMS RRD store. Threshold definitions then evaluate those files continuously and generate events when values cross defined limits.
Collectd -> Prometheus (:9090) -> PromQL -> JRB (RRD) -> Threshold Engine -> Events
## What Is Collected
### Deployment and StatefulSet Metrics
- Current vs. desired replica counts per deployment and StatefulSet
- Detects rollout failures and pod unavailability
### Node Resource Utilization
- CPU usage per worker node
- Memory consumption per worker node
- Disk utilization per node --- important for catching eviction conditions before they occur
### Network Interface Metrics --- Worker and Master Nodes
- RX and TX throughput per node
- Packet drop counters per interface
### Cluster Inventory Metrics
- Node count (worker and master separately)
- Overall cluster capacity utilization
## RRD Storage Layout
Each Kubernetes node gets its own directory within the OpenNMS RRD file hierarchy. Metrics are separated into individual JRB files by type, for example:
$RRD_BASE/prometheus/
└── node-01/
├── cpu_usage.jrb
├── mem_usage.jrb
├── net_rx.jrb
└── net_tx.jrb
This structure keeps per-node data isolated and makes threshold configuration and graph definitions straightforward to manage.
## Key Configuration Components
| Component | Purpose |
|---|---|
| `collectd-configuration.xml` | Defines the Prometheus collection package and polling interval |
| `prometheus-datacollection-config.xml` | Maps PromQL expressions to metric names and RRD attributes |
| `thresholds.xml` | Defines high/low thresholds per metric and generates events on breach |
| Collection package | Scopes which nodes/endpoints are polled and at what frequency |
* Above is table.
## Configuration Notes
- The Prometheus endpoint must be reachable from the OpenNMS poller (Use curl -kv). If the cluster is on a separate network segment, ensure routing or a Minion is in place.
- PromQL expressions should be scoped per node using label selectors (e.g. `node=~"worker-.*"`) to avoid aggregating across the cluster where per-node granularity is needed.
- JRB heartbeat and step values should match the 30-second collection interval to avoid gaps in RRD graphs.
- Threshold definitions should reference the exact datasource names declared in the datacollection config.
## Related Topics
- Collectd configuration for Prometheus endpoints
- Writing PromQL expressions for Kubernetes node metrics
- JRB vs. RRD storage in OpenNMS
- Threshold configuration and event definitions
- OpenNMS Minion deployment for remote polling