11 KiB
Host metrics → Kafka → Data Prepper → OpenSearch
A complete single-host proof of concept for a Debian VM. Telegraf runs as a native systemd service; Kafka, Data Prepper, OpenSearch and OpenSearch Dashboards run in Docker. Four importable dashboards contain 43 visualizations (including navigation panels) plus an index pattern. No Grafana, Elasticsearch, ZooKeeper or custom ingestion application is required.
flowchart LR
A[Native Telegraf agent\nDebian systemd] -->|JSON / host key| K[Kafka\nhost-metrics-v1\n6 partitions]
K -->|consumer group / acknowledgments| D[Data Prepper\nparse + timestamp]
D --> O[OpenSearch\ndaily indices / 14 days]
O --> U[OpenSearch Dashboards\n4 detailed dashboards]
K -. optional independent group .-> B[Telegraf protocol bridge\nDocker / no host inputs]
B -. scrape .-> P[Prometheus\n7 days]
P -. SQL plugin connector .-> O
Why Telegraf?
Telegraf is an open-source collector with native Debian packaging, host inputs and a Kafka output. Its JSON records contain one measurement with numeric fields and identity tags, which Data Prepper's Kafka source can parse directly. CPU percentages and native memory/filesystem measurements avoid unnecessary OTLP translation. The agent adds reset-aware disk and network rates before Kafka.
OpenTelemetry Collector remains a reasonable alternative, but OTLP metric envelopes need signal-aware decoding and conversion; feeding arbitrary OTLP JSON into parse_json does not create usable metric documents. This implementation uses your allowed alternative collector while keeping the required pipeline intact.
Versions
Stable releases checked on 17 September 2026, pinned for reproducibility:
| Component | Version |
|---|---|
| Apache Kafka, official JVM image | 4.3.1 |
| OpenSearch / OpenSearch Dashboards | 3.8.0 / 3.8.0 |
| OpenSearch Data Prepper | 2.16.0 |
| Native Telegraf / optional bridge | 1.40.0 |
| Optional Prometheus | 3.14.0 |
These image tags were checked against Docker Hub; the OpenSearch, Dashboards, Data Prepper, Telegraf and Prometheus images support Linux amd64 and arm64. The installer pins the signed Debian package to 1.40.0-1. References are in docs/SOURCES.md. New upstream releases require a compatibility check, not an automatic latest upgrade.
Prerequisites
- A Debian 12 or 13 VM with systemd, Internet access, Docker Engine and the Docker Compose v2+ plugin. Use Docker's official Debian installation instructions if needed.
- Suggested starting size: 4 vCPUs, 12 GiB RAM, 60+ GiB free SSD. This is a planning estimate, not a measured capacity guarantee. The OpenSearch heap is 2 GiB; Data Prepper and Kafka each have a maximum 1 GiB heap, with additional native memory and filesystem cache required. Optional Prometheus adds memory and disk usage.
- Run Docker commands as a user who can access Docker, or use a root shell consistently. The native installation commands explicitly use
sudo. - Free host ports: 9092, 9200, 5601; optionally 9090. Accurate host time is required.
Access model: published ports bind to 127.0.0.1. This POC disables OpenSearch authentication/TLS and uses plaintext Kafka inside the local Docker network. Access Dashboards over an SSH tunnel. It is intended for a trusted VM and trusted local containers. Enabling Internet/LAN exposure requires the security changes in docs/SCALING.md.
Install and start
Copy this directory to the Debian VM, for example to ~/host-metrics, and run from that directory:
sudo ./scripts/prepare-host.sh
./scripts/start.sh
sudo ./scripts/install-agent.sh
./scripts/smoke-test.sh
prepare-host.sh installs small host prerequisites and raises vm.max_map_count to 262144 if needed. It does not install Docker or lower an existing kernel setting.
start.sh creates .env, generates a persistent random SQL data-source encryption key, pulls the pinned images, starts the stack, and waits for Dashboards. Preserve .env with the data volumes. Kafka's initializer creates the six-partition topic; the OpenSearch initializer installs numeric mappings and a 14-day retention policy before Data Prepper starts.
install-agent.sh installs Telegraf from the signed InfluxData APT repository, validates collection as the telegraf user and starts telegraf-host-metrics.service. It uses a dedicated configuration directory and unit. On a fresh package installation it disables the package's default unit. An existing Telegraf unit/configuration is preserved. Reinstalling backs up the POC's configuration before replacing it; a running pre-existing collector might collect overlapping metrics independently.
Optionally set host identity at installation:
sudo env METRICS_HOST=debian-monitor-01 \
METRICS_ENVIRONMENT=homelab METRICS_ROLE=monitoring \
./scripts/install-agent.sh
The default host identity is the FQDN, environment is lab, and role is docker-host. Choose a unique stable host name for each server. The Kafka bootstrap endpoint defaults to 127.0.0.1:9092.
Allow 30–90 seconds after agent installation for CPU/rate warm-up and ingestion. The smoke test waits up to approximately three minutes for recent CPU, memory, filesystem, disk-rate and network-rate data and validates field types. It prints Kafka consumer lag for inspection.
Open and import dashboards
On your workstation, keep this SSH tunnel running:
ssh -N -L 5601:127.0.0.1:5601 your-user@your-debian-vm
Open http://localhost:5601. No application login is configured in this local POC.
In OpenSearch Dashboards, go to Management → Dashboards Management → Saved objects → Import (the navigation may appear under Stack Management). Import:
dashboards/host-metrics.ndjson
Select overwrite conflicts when updating an earlier import. This is OpenSearch's saved-object import format: each line is a JSON object. The equivalent pretty-printed host-metrics.json is supplied for inspection and automation; upload the NDJSON file to the UI.
Or import from the Debian VM:
./scripts/import-dashboards.sh
Open Dashboards → Host Metrics / Fleet overview. The import contains:
| Dashboard | Coverage |
|---|---|
| Fleet overview | Hosts seen, CPU/RAM/filesystem peaks, normalized load, latest sample per host, inventory |
| CPU & memory | Total/per-core CPU, user/system/iowait/steal, 1/5/15-minute load, RAM headroom, swap I/O, process states, context switches, interrupts |
| Storage | Per-mount used/free capacity, inode use, capacity table, disk throughput, IOPS, device busy time, requests in progress |
| Network | Per-interface RX/TX, packets, errors, drops and interface inventory |
Dark mode, a consistent grid, prominent summary cards, legends, readable units and navigation links are configured. Use Add filter → tags.host to select a server; tags.environment and tags.role select groups. Links navigate between dashboards but do not promise to retain unsaved filters; check the filter bar after navigation.
Dashboards start at Last 1 hour, refreshing every 15 seconds. Cards explicitly report peaks/minima or hosts seen over that range; they are not instantaneous health indicators. The freshness table shows the newest sample inside the selected range. A missing host is not proof that it is healthy. See metric semantics.
Optional: Observability Metric Analytics
The primary dashboards above query OpenSearch's metric documents. They work without Prometheus.
The dedicated Observability → Metrics / Metric Analytics interface supports Prometheus and a specific OpenTelemetry metrics schema. This project's Telegraf index is a custom schema, so enable the supplied Prometheus adapter to use that interface:
./scripts/enable-metric-analytics.sh
# Wait around 60 seconds after enabling it, then:
python3 ./scripts/check-metric-analytics.py
This starts two optional containers and registers the host_prometheus SQL data source. The bridge consumes the same Kafka topic in its own consumer group, reconstructs Telegraf metrics and exposes them for Prometheus. It does not collect the container/host itself and it does not bypass Kafka. The main Data Prepper → OpenSearch ingestion path continues independently.
Open Observability → Metrics, select host_prometheus, and select metrics such as cpu_usage_active, mem_used_percent, disk_used_percent and net_bytes_recv_per_sec. Save selected visualizations to an operational dashboard if desired. The four supplied NDJSON dashboards are regular OpenSearch dashboards; they are not imports for the separate operational-panel format.
Example PPL query in the observability query interface:
source = host_prometheus.cpu_usage_active
| where cpu = 'cpu-total'
| stats avg(@value) by span(@timestamp, 1m), host
For native Prometheus exploration, also forward port 9090 through SSH. Example PromQL:
cpu_usage_active{cpu="cpu-total"}
mem_used_percent
disk_used_percent{path="/"}
net_bytes_recv_per_sec
The *_per_sec metrics are already rates: do not apply rate() to them. The bridge starts at the newest offset for a new group. It exposes the most recent value per series between scrapes; it is for live metric analytics, not lossless historical replay into Prometheus. Original sample timestamps are exported, so an old Kafka sample does not masquerade as a newly collected sample. OpenSearch remains the historical store for the requested pipeline.
Files and operations
compose.yaml All core services + optional metrics-analytics profile
agent/ Native configuration, enrichment/rates and systemd unit
configs/kafka/ KRaft broker and topic initialization
configs/data-prepper/ Kafka → JSON → dated OpenSearch indices
configs/opensearch/ Node settings, mappings, ISM retention, initialization
configs/dashboards/ Dashboards settings
configs/metrics-bridge/ Optional Kafka-to-Prometheus protocol adapter
configs/prometheus/ Optional scrape configuration
dashboards/ Importable NDJSON + equivalent JSON
scripts/ Installation, startup, import and end-to-end checks
tests/ Offline contracts and rate calculation checks
docs/ Metric semantics, operations, scaling and references
See operations and troubleshooting, scaling to many servers, and validation status.
Validation status
The JSON/TOML contracts, dashboard reference graph and layout, rate mathematics, YAML syntax and shell syntax were checked locally. Image tags and configuration references were checked against upstream sources. The full stack has not been executed here: this development workspace has no Docker runtime and is not your Debian VM. The native agent and dashboard rendering still require the included Debian smoke tests and a browser check. Treat this as a complete POC implementation with explicit runtime validation steps, not a claim of a completed deployment or production certification.