host-metrics/docs/METRICS.md

Metric schema and dashboard semantics

Every Kafka message is one Telegraf JSON measurement. It is not an OTLP envelope and it is not a JSON array/batch. Telegraf's Kafka output serializes each measurement separately. The Kafka message key is the host tag; the topic is host-metrics-v1.

Example (values illustrative):

{
  "name": "cpu",
  "tags": {
    "host": "debian-monitor-01",
    "environment": "homelab",
    "role": "monitoring",
    "cpu": "cpu-total",
    "series": "debian-monitor-01 / cpu-total"
  },
  "fields": {
    "usage_active": 18.5,
    "usage_user": 12.1,
    "usage_system": 4.2,
    "usage_idle": 81.5
  },
  "timestamp": 1789689600000
}

Data Prepper decodes the message field, converts the millisecond epoch into @timestamp, records ingested_at, and removes the redundant encoded message. The date processor also sets the origination metadata used for time-based sink index naming. Normal records land in host-metrics-v1-YYYY.MM.dd. Times and daily boundaries are UTC.

name and tags.* are keywords. @timestamp and ingested_at are dates. Numeric fields.* use doubles; this prevents a shared field such as used_percent becoming an integer mapping because the first observed value was zero. Exact very large integer counters above 2^53 are not preserved by double mappings; the dashboards primarily use percentages, current capacities, and derived rates. The original JSON remains in _source.

Measurement	Interpretation
cpu	Percentages on a 0–100 scale. Fleet cards use cpu-total; per-core charts exclude it. usage_active is Telegraf's non-idle measure. I/O wait and steal are shown separately for diagnosis.
mem	Telegraf/Linux memory semantics. available includes reclaimable memory; cached memory is not necessarily pressure. used_percent is not a kernel OOM prediction.
system	1/5/15-minute load, logical CPU count, uptime seconds. load1_per_cpu is load divided by logical CPUs. Load includes uninterruptible tasks, not just CPU demand.
disk	Capacity and inode usage per mounted real filesystem. Docker overlay and pseudo filesystems are excluded to avoid duplicates. Separate bind mounts may still describe the same underlying storage.
diskio	Kernel block-device counters. Derived read/write bytes/s, operations/s, and io_busy_percent. Layered devices (LVM/device-mapper/physical disks) may represent the same I/O; do not add them blindly. Busy percentage is not a reliable saturation limit for highly parallel NVMe/RAID.
net	Interface byte/packet/error/drop counters and derived rates. Loopback, veth and conventional Docker bridges are excluded. Adjust the tag filter if you want those interfaces. Rates are bytes/s, not bits/s.
swap	Capacity and derived swap-in/out bytes/s. On hosts with no swap, capacity metrics can be zero and activity charts may be empty.
processes	Host process counts by state; no per-process command-line collection. Permission restrictions such as hidepid may limit visibility.
kernel	Context switches, interrupts and fork counts; derived per-second rates.

Rate handling

The Starlark processor keeps previous counter values per measurement + complete tag set. A rate is (current - previous) / actual_elapsed_seconds, not divided by an assumed fixed polling interval. Counters on different hosts/devices never share a baseline. On the first sample, counter reset, repeated/out-of-order timestamp, or gap over 120 seconds, the affected rate is omitted. The next valid interval recovers automatically. Negative rates are never fabricated or silently presented as zero.

The state is local to the agent and resets after restart. Raw counters are kept alongside rates. Kafka replay preserves precomputed rates. For disk busy time, a millisecond/second rate is divided by 10 and capped at 100%. Processor state grows with distinct tag combinations; the fixed host-input set has bounded practical cardinality, but endlessly changing interfaces/mount names could grow it during a long-running process.

Reading the dashboards

• Every time chart averages values within its automatically chosen bucket, separately for each selected series. Zoom in for detail; wider ranges smooth short spikes.
• Summary cards explicitly say peak, minimum, or hosts seen in selected range. They are time-range statistics. They do not promise a latest-value fleet total.
• tags.series combines host and mount/interface/device/CPU to keep two hosts' eth0 or / separate.
• Charts and tables select the top 30 terms by document count. Use a host filter for large fleets or increase the limit deliberately.
• Latest-sample tables only search the selected time window. A host that stopped reporting before that window will be absent. Inventory-backed absent-host alerting is a separate production concern.
• No-data is not zero. A blank rate chart during the first two collection cycles is normal; prolonged blanks need investigation.
• The single-node pipeline is at least once after Kafka: restarts/retries can create duplicate indexed documents. There is no end-to-end exactly-once guarantee. Min/max are insensitive to identical duplicates; averages and counts can be biased. Do not treat document counts as exact sample counts.

The optional Prometheus exporter preserves metric sample timestamps. It presents a live snapshot per series, so its history can differ from OpenSearch when Kafka is replayed or when several records arrive between scrapes. It does not reproduce every Kafka sample in Prometheus.