# Host metrics → Kafka → Data Prepper → OpenSearch

A complete single-host proof of concept for a Debian VM. Telegraf runs as a native systemd service; Kafka, Data Prepper, OpenSearch and OpenSearch Dashboards run in Docker. Four importable dashboards contain 43 visualizations (including navigation panels) plus an index pattern. No Grafana, Elasticsearch, ZooKeeper or custom ingestion application is required.

```mermaid
flowchart LR
  A[Native Telegraf agent\nDebian systemd] -->|JSON / host key| K[Kafka\nhost-metrics-v1\n6 partitions]
  K -->|consumer group / acknowledgments| D[Data Prepper\nparse + timestamp]
  D --> O[OpenSearch\ndaily indices / 14 days]
  O --> U[OpenSearch Dashboards\n4 detailed dashboards]
  K -. optional independent group .-> B[Telegraf protocol bridge\nDocker / no host inputs]
  B -. scrape .-> P[Prometheus\n7 days]
  P -. SQL plugin connector .-> O
```

## Why Telegraf?

Telegraf is an open-source collector with native Debian packaging, host inputs and a Kafka output. Its JSON records contain one measurement with numeric fields and identity tags, which Data Prepper's Kafka source can parse directly. CPU percentages and native memory/filesystem measurements avoid unnecessary OTLP translation. The agent adds reset-aware disk and network rates before Kafka.

OpenTelemetry Collector remains a reasonable alternative, but OTLP metric envelopes need signal-aware decoding and conversion; feeding arbitrary OTLP JSON into `parse_json` does not create usable metric documents. This implementation uses your allowed alternative collector while keeping the required pipeline intact.

## Versions

Stable releases checked on **17 September 2026**, pinned for reproducibility:

| Component | Version |
|---|---|
| Apache Kafka, official JVM image | 4.3.1 |
| OpenSearch / OpenSearch Dashboards | 3.8.0 / 3.8.0 |
| OpenSearch Data Prepper | 2.16.0 |
| Native Telegraf / optional bridge | 1.40.0 |
| Optional Prometheus | 3.14.0 |

These image tags were checked against Docker Hub; the OpenSearch, Dashboards, Data Prepper, Telegraf and Prometheus images support Linux amd64 and arm64. The installer pins the signed Debian package to `1.40.0-1`. References are in [docs/SOURCES.md](docs/SOURCES.md). New upstream releases require a compatibility check, not an automatic `latest` upgrade.

## Prerequisites

- A Debian 12 or 13 VM with systemd, Internet access, Docker Engine and the Docker Compose v2+ plugin. Use Docker's [official Debian installation instructions](https://docs.docker.com/engine/install/debian/) if needed.
- Suggested starting size: **4 vCPUs, 12 GiB RAM, 60+ GiB free SSD**. This is a planning estimate, not a measured capacity guarantee. The OpenSearch heap is 2 GiB; Data Prepper and Kafka each have a maximum 1 GiB heap, with additional native memory and filesystem cache required. Optional Prometheus adds memory and disk usage.
- Run Docker commands as a user who can access Docker, or use a root shell consistently. The native installation commands explicitly use `sudo`.
- Free host ports: 9092, 9200, 5601; optionally 9090. Accurate host time is required.

**Access model:** published ports bind to `127.0.0.1`. This POC disables OpenSearch authentication/TLS and uses plaintext Kafka inside the local Docker network. Access Dashboards over an SSH tunnel. It is intended for a trusted VM and trusted local containers. Enabling Internet/LAN exposure requires the security changes in [docs/SCALING.md](docs/SCALING.md).

## Install and start

Copy this directory to the Debian VM, for example to `~/host-metrics`, and run from that directory:

```bash
sudo ./scripts/prepare-host.sh
./scripts/start.sh
sudo ./scripts/install-agent.sh
./scripts/smoke-test.sh
```

`prepare-host.sh` installs small host prerequisites and raises `vm.max_map_count` to 262144 if needed. It does not install Docker or lower an existing kernel setting.

`start.sh` creates `.env`, generates a persistent random SQL data-source encryption key, pulls the pinned images, starts the stack, and waits for Dashboards. Preserve `.env` with the data volumes. Kafka's initializer creates the six-partition topic; the OpenSearch initializer installs numeric mappings and a 14-day retention policy before Data Prepper starts.

`install-agent.sh` installs Telegraf from the signed InfluxData APT repository, validates collection as the `telegraf` user and starts `telegraf-host-metrics.service`. It uses a dedicated configuration directory and unit. On a fresh package installation it disables the package's default unit. An existing Telegraf unit/configuration is preserved. Reinstalling backs up the POC's configuration before replacing it; a running pre-existing collector might collect overlapping metrics independently.

Optionally set host identity at installation:

```bash
sudo env METRICS_HOST=debian-monitor-01 \
  METRICS_ENVIRONMENT=homelab METRICS_ROLE=monitoring \
  ./scripts/install-agent.sh
```

The default host identity is the FQDN, environment is `lab`, and role is `docker-host`. Choose a unique stable host name for each server. The Kafka bootstrap endpoint defaults to `127.0.0.1:9092`.

Allow **30–90 seconds** after agent installation for CPU/rate warm-up and ingestion. The smoke test waits up to approximately three minutes for recent CPU, memory, filesystem, disk-rate and network-rate data and validates field types. It prints Kafka consumer lag for inspection.

## Open and import dashboards

On your workstation, keep this SSH tunnel running:

```bash
ssh -N -L 5601:127.0.0.1:5601 your-user@your-debian-vm
```

Open [http://localhost:5601](http://localhost:5601). No application login is configured in this local POC.

In OpenSearch Dashboards, go to **Management → Dashboards Management → Saved objects → Import** (the navigation may appear under **Stack Management**). Import:

**[`dashboards/host-metrics.ndjson`](dashboards/host-metrics.ndjson)**

Select **overwrite conflicts** when updating an earlier import. This is OpenSearch's saved-object import format: each line is a JSON object. The equivalent pretty-printed [`host-metrics.json`](dashboards/host-metrics.json) is supplied for inspection and automation; upload the **NDJSON** file to the UI.

Or import from the Debian VM:

```bash
./scripts/import-dashboards.sh
```

Open **Dashboards → Host Metrics / Fleet overview**. The import contains:

| Dashboard | Coverage |
|---|---|
| Fleet overview | Hosts seen, CPU/RAM/filesystem peaks, normalized load, latest sample per host, inventory |
| CPU & memory | Total/per-core CPU, user/system/iowait/steal, 1/5/15-minute load, RAM headroom, swap I/O, process states, context switches, interrupts |
| Storage | Per-mount used/free capacity, inode use, capacity table, disk throughput, IOPS, device busy time, requests in progress |
| Network | Per-interface RX/TX, packets, errors, drops and interface inventory |

Dark mode, a consistent grid, prominent summary cards, legends, readable units and navigation links are configured. Use **Add filter → tags.host** to select a server; `tags.environment` and `tags.role` select groups. Links navigate between dashboards but do not promise to retain unsaved filters; check the filter bar after navigation.

Dashboards start at **Last 1 hour**, refreshing every 15 seconds. Cards explicitly report peaks/minima or hosts seen over that range; they are not instantaneous health indicators. The freshness table shows the newest sample **inside the selected range**. A missing host is not proof that it is healthy. See [metric semantics](docs/METRICS.md).

## Optional: Observability Metric Analytics

The primary dashboards above query OpenSearch's metric documents. They work without Prometheus.

The dedicated **Observability → Metrics / Metric Analytics** interface supports Prometheus and a specific OpenTelemetry metrics schema. This project's Telegraf index is a custom schema, so enable the supplied Prometheus adapter to use that interface:

```bash
./scripts/enable-metric-analytics.sh
# Wait around 60 seconds after enabling it, then:
python3 ./scripts/check-metric-analytics.py
```

This starts two optional containers and registers the `host_prometheus` SQL data source. The bridge consumes **the same Kafka topic in its own consumer group**, reconstructs Telegraf metrics and exposes them for Prometheus. It does not collect the container/host itself and it does not bypass Kafka. The main Data Prepper → OpenSearch ingestion path continues independently.

Open **Observability → Metrics**, select **host_prometheus**, and select metrics such as `cpu_usage_active`, `mem_used_percent`, `disk_used_percent` and `net_bytes_recv_per_sec`. Save selected visualizations to an operational dashboard if desired. The four supplied NDJSON dashboards are regular OpenSearch dashboards; they are not imports for the separate operational-panel format.

Example PPL query in the observability query interface:

```text
source = host_prometheus.cpu_usage_active
| where cpu = 'cpu-total'
| stats avg(@value) by span(@timestamp, 1m), host
```

For native Prometheus exploration, also forward port 9090 through SSH. Example PromQL:

```promql
cpu_usage_active{cpu="cpu-total"}
mem_used_percent
disk_used_percent{path="/"}
net_bytes_recv_per_sec
```

The `*_per_sec` metrics are already rates: **do not apply `rate()` to them**. The bridge starts at the newest offset for a new group. It exposes the most recent value per series between scrapes; it is for live metric analytics, not lossless historical replay into Prometheus. Original sample timestamps are exported, so an old Kafka sample does not masquerade as a newly collected sample. OpenSearch remains the historical store for the requested pipeline.

## Files and operations

```text
compose.yaml                      All core services + optional metrics-analytics profile
agent/                            Native configuration, enrichment/rates and systemd unit
configs/kafka/                    KRaft broker and topic initialization
configs/data-prepper/             Kafka → JSON → dated OpenSearch indices
configs/opensearch/               Node settings, mappings, ISM retention, initialization
configs/dashboards/               Dashboards settings
configs/metrics-bridge/           Optional Kafka-to-Prometheus protocol adapter
configs/prometheus/               Optional scrape configuration
dashboards/                       Importable NDJSON + equivalent JSON
scripts/                          Installation, startup, import and end-to-end checks
tests/                            Offline contracts and rate calculation checks
docs/                             Metric semantics, operations, scaling and references
```

See [operations and troubleshooting](docs/OPERATIONS.md), [scaling to many servers](docs/SCALING.md), and [validation status](docs/VALIDATION.md).

## Validation status

The JSON/TOML contracts, dashboard reference graph and layout, rate mathematics, YAML syntax and shell syntax were checked locally. Image tags and configuration references were checked against upstream sources. **The full stack has not been executed here: this development workspace has no Docker runtime and is not your Debian VM.** The native agent and dashboard rendering still require the included Debian smoke tests and a browser check. Treat this as a complete POC implementation with explicit runtime validation steps, not a claim of a completed deployment or production certification.