60 lines
5.5 KiB
Markdown
60 lines
5.5 KiB
Markdown
# Metric schema and dashboard semantics
|
||
|
||
Every Kafka message is one Telegraf JSON measurement. It is **not** an OTLP envelope and it is not a JSON array/batch. Telegraf's Kafka output serializes each measurement separately. The Kafka message key is the `host` tag; the topic is `host-metrics-v1`.
|
||
|
||
Example (values illustrative):
|
||
|
||
```json
|
||
{
|
||
"name": "cpu",
|
||
"tags": {
|
||
"host": "debian-monitor-01",
|
||
"environment": "homelab",
|
||
"role": "monitoring",
|
||
"cpu": "cpu-total",
|
||
"series": "debian-monitor-01 / cpu-total"
|
||
},
|
||
"fields": {
|
||
"usage_active": 18.5,
|
||
"usage_user": 12.1,
|
||
"usage_system": 4.2,
|
||
"usage_idle": 81.5
|
||
},
|
||
"timestamp": 1789689600000
|
||
}
|
||
```
|
||
|
||
Data Prepper decodes the `message` field, converts the **millisecond epoch** into `@timestamp`, records `ingested_at`, and removes the redundant encoded message. The date processor also sets the origination metadata used for time-based sink index naming. Normal records land in `host-metrics-v1-YYYY.MM.dd`. Times and daily boundaries are UTC.
|
||
|
||
`name` and `tags.*` are keywords. `@timestamp` and `ingested_at` are dates. Numeric `fields.*` use doubles; this prevents a shared field such as `used_percent` becoming an integer mapping because the first observed value was zero. Exact very large integer counters above 2^53 are not preserved by double mappings; the dashboards primarily use percentages, current capacities, and derived rates. The original JSON remains in `_source`.
|
||
|
||
| Measurement | Interpretation |
|
||
|---|---|
|
||
| cpu | Percentages on a 0–100 scale. Fleet cards use `cpu-total`; per-core charts exclude it. `usage_active` is Telegraf's non-idle measure. I/O wait and steal are shown separately for diagnosis. |
|
||
| mem | Telegraf/Linux memory semantics. `available` includes reclaimable memory; cached memory is not necessarily pressure. `used_percent` is not a kernel OOM prediction. |
|
||
| system | 1/5/15-minute load, logical CPU count, uptime seconds. `load1_per_cpu` is load divided by logical CPUs. Load includes uninterruptible tasks, not just CPU demand. |
|
||
| disk | Capacity and inode usage per mounted real filesystem. Docker overlay and pseudo filesystems are excluded to avoid duplicates. Separate bind mounts may still describe the same underlying storage. |
|
||
| diskio | Kernel block-device counters. Derived read/write bytes/s, operations/s, and `io_busy_percent`. Layered devices (LVM/device-mapper/physical disks) may represent the same I/O; do not add them blindly. Busy percentage is not a reliable saturation limit for highly parallel NVMe/RAID. |
|
||
| net | Interface byte/packet/error/drop counters and derived rates. Loopback, veth and conventional Docker bridges are excluded. Adjust the tag filter if you want those interfaces. Rates are bytes/s, not bits/s. |
|
||
| swap | Capacity and derived swap-in/out bytes/s. On hosts with no swap, capacity metrics can be zero and activity charts may be empty. |
|
||
| processes | Host process counts by state; no per-process command-line collection. Permission restrictions such as `hidepid` may limit visibility. |
|
||
| kernel | Context switches, interrupts and fork counts; derived per-second rates. |
|
||
|
||
## Rate handling
|
||
|
||
The Starlark processor keeps previous counter values per **measurement + complete tag set**. A rate is `(current - previous) / actual_elapsed_seconds`, not divided by an assumed fixed polling interval. Counters on different hosts/devices never share a baseline. On the first sample, counter reset, repeated/out-of-order timestamp, or gap over 120 seconds, the affected rate is omitted. The next valid interval recovers automatically. Negative rates are never fabricated or silently presented as zero.
|
||
|
||
The state is local to the agent and resets after restart. Raw counters are kept alongside rates. Kafka replay preserves precomputed rates. For disk busy time, a millisecond/second rate is divided by 10 and capped at 100%. Processor state grows with distinct tag combinations; the fixed host-input set has bounded practical cardinality, but endlessly changing interfaces/mount names could grow it during a long-running process.
|
||
|
||
## Reading the dashboards
|
||
|
||
- Every time chart averages values within its automatically chosen bucket, separately for each selected series. Zoom in for detail; wider ranges smooth short spikes.
|
||
- Summary cards explicitly say **peak**, **minimum**, or **hosts seen in selected range**. They are time-range statistics. They do not promise a latest-value fleet total.
|
||
- `tags.series` combines host and mount/interface/device/CPU to keep two hosts' `eth0` or `/` separate.
|
||
- Charts and tables select the top 30 terms by document count. Use a host filter for large fleets or increase the limit deliberately.
|
||
- Latest-sample tables only search the selected time window. A host that stopped reporting before that window will be absent. Inventory-backed absent-host alerting is a separate production concern.
|
||
- No-data is not zero. A blank rate chart during the first two collection cycles is normal; prolonged blanks need investigation.
|
||
- The single-node pipeline is at least once after Kafka: restarts/retries can create duplicate indexed documents. There is no end-to-end exactly-once guarantee. Min/max are insensitive to identical duplicates; averages and counts can be biased. Do not treat document counts as exact sample counts.
|
||
|
||
The optional Prometheus exporter preserves metric sample timestamps. It presents a live snapshot per series, so its history can differ from OpenSearch when Kafka is replayed or when several records arrive between scrapes. It does not reproduce every Kafka sample in Prometheus.
|