flavien/host-metrics
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
Flavien Haas 2026-09-17 22:31:35 +02:00
commit 0ae0d515ad
44 changed files with 3314 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
.env.example Normal file
View File

@ -0,0 +1,11 @@
# Stable upstream releases verified 2026-09-17; never use floating latest tags.
KAFKA_VERSION=4.3.1
OPENSEARCH_VERSION=3.8.0
DATA_PREPPER_VERSION=2.16.0
TELEGRAF_VERSION=1.40.0
PROMETHEUS_VERSION=v3.14.0
OPENSEARCH_JAVA_OPTS=-Xms2g -Xmx2g
# Direct workstation access. Configure using scripts/configure-access.sh VM_HOST WORKSTATION_IP.
BIND_ADDRESS=
WORKSTATION_IP=
KAFKA_ADVERTISED_HOST=

7
.gitignore vendored Normal file
View File

@ -0,0 +1,7 @@
.env
__pycache__/
*.tar.gz
.validation/
diagnostics/
.DS_Store
*.sha256

165
README.md Normal file
View File

@ -0,0 +1,165 @@
# Host metrics → Kafka → Data Prepper → OpenSearch
A complete single-host proof of concept for a Debian VM. Telegraf runs as a native systemd service; Kafka, Data Prepper, OpenSearch and OpenSearch Dashboards run in Docker. Four importable dashboards contain 43 visualizations (including navigation panels) plus an index pattern. No Grafana, Elasticsearch, ZooKeeper or custom ingestion application is required.
```mermaid
flowchart LR
A[Native Telegraf agent\nDebian systemd] -->|JSON / host key| K[Kafka\nhost-metrics-v1\n6 partitions]
K -->|consumer group / acknowledgments| D[Data Prepper\nparse + timestamp]
D --> O[OpenSearch\ndaily indices / 14 days]
O --> U[OpenSearch Dashboards\n4 detailed dashboards]
K -. optional independent group .-> B[Telegraf protocol bridge\nDocker / no host inputs]
B -. scrape .-> P[Prometheus\n7 days]
P -. SQL plugin connector .-> O
```
## Why Telegraf?
Telegraf is an open-source collector with native Debian packaging, host inputs and a Kafka output. Its JSON records contain one measurement with numeric fields and identity tags, which Data Prepper's Kafka source can parse directly. CPU percentages and native memory/filesystem measurements avoid unnecessary OTLP translation. The agent adds reset-aware disk and network rates before Kafka.
OpenTelemetry Collector remains a reasonable alternative, but OTLP metric envelopes need signal-aware decoding and conversion; feeding arbitrary OTLP JSON into `parse_json` does not create usable metric documents. This implementation uses your allowed alternative collector while keeping the required pipeline intact.
## Versions
Stable releases checked on **17 September 2026**, pinned for reproducibility:
| Component | Version |
|---|---|
| Apache Kafka, official JVM image | 4.3.1 |
| OpenSearch / OpenSearch Dashboards | 3.8.0 / 3.8.0 |
| OpenSearch Data Prepper | 2.16.0 |
| Native Telegraf / optional bridge | 1.40.0 |
| Optional Prometheus | 3.14.0 |
These image tags were checked against Docker Hub; the OpenSearch, Dashboards, Data Prepper, Telegraf and Prometheus images support Linux amd64 and arm64. The installer pins the signed Debian package to `1.40.0-1`. References are in [docs/SOURCES.md](docs/SOURCES.md). New upstream releases require a compatibility check, not an automatic `latest` upgrade.
## Prerequisites
- A Debian 12 or 13 VM with systemd, Internet access, Docker Engine and the Docker Compose v2+ plugin. Use Docker's [official Debian installation instructions](https://docs.docker.com/engine/install/debian/) if needed.
- Suggested starting size: **4 vCPUs, 12 GiB RAM, 60+ GiB free SSD**. This is a planning estimate, not a measured capacity guarantee. The OpenSearch heap is 2 GiB; Data Prepper and Kafka each have a maximum 1 GiB heap, with additional native memory and filesystem cache required. Optional Prometheus adds memory and disk usage.
- Run Docker commands as a user who can access Docker, or use a root shell consistently. The native installation commands explicitly use `sudo`.
- Free host ports: 9092, 9200, 5601; optionally 9090. Accurate host time is required.
**Access model:** published ports bind to `127.0.0.1`. This POC disables OpenSearch authentication/TLS and uses plaintext Kafka inside the local Docker network. Access Dashboards over an SSH tunnel. It is intended for a trusted VM and trusted local containers. Enabling Internet/LAN exposure requires the security changes in [docs/SCALING.md](docs/SCALING.md).
## Install and start
Copy this directory to the Debian VM, for example to `~/host-metrics`, and run from that directory:
```bash
sudo ./scripts/prepare-host.sh
./scripts/start.sh
sudo ./scripts/install-agent.sh
./scripts/smoke-test.sh
```
`prepare-host.sh` installs small host prerequisites and raises `vm.max_map_count` to 262144 if needed. It does not install Docker or lower an existing kernel setting.
`start.sh` creates `.env`, generates a persistent random SQL data-source encryption key, pulls the pinned images, starts the stack, and waits for Dashboards. Preserve `.env` with the data volumes. Kafka's initializer creates the six-partition topic; the OpenSearch initializer installs numeric mappings and a 14-day retention policy before Data Prepper starts.
`install-agent.sh` installs Telegraf from the signed InfluxData APT repository, validates collection as the `telegraf` user and starts `telegraf-host-metrics.service`. It uses a dedicated configuration directory and unit. On a fresh package installation it disables the package's default unit. An existing Telegraf unit/configuration is preserved. Reinstalling backs up the POC's configuration before replacing it; a running pre-existing collector might collect overlapping metrics independently.
Optionally set host identity at installation:
```bash
sudo env METRICS_HOST=debian-monitor-01 \
METRICS_ENVIRONMENT=homelab METRICS_ROLE=monitoring \
./scripts/install-agent.sh
```
The default host identity is the FQDN, environment is `lab`, and role is `docker-host`. Choose a unique stable host name for each server. The Kafka bootstrap endpoint defaults to `127.0.0.1:9092`.
Allow **30–90 seconds** after agent installation for CPU/rate warm-up and ingestion. The smoke test waits up to approximately three minutes for recent CPU, memory, filesystem, disk-rate and network-rate data and validates field types. It prints Kafka consumer lag for inspection.
## Open and import dashboards
On your workstation, keep this SSH tunnel running:
```bash
ssh -N -L 5601:127.0.0.1:5601 your-user@your-debian-vm
```
Open [http://localhost:5601](http://localhost:5601). No application login is configured in this local POC.
In OpenSearch Dashboards, go to **Management → Dashboards Management → Saved objects → Import** (the navigation may appear under **Stack Management**). Import:
**[`dashboards/host-metrics.ndjson`](dashboards/host-metrics.ndjson)**
Select **overwrite conflicts** when updating an earlier import. This is OpenSearch's saved-object import format: each line is a JSON object. The equivalent pretty-printed [`host-metrics.json`](dashboards/host-metrics.json) is supplied for inspection and automation; upload the **NDJSON** file to the UI.
Or import from the Debian VM:
```bash
./scripts/import-dashboards.sh
```
Open **Dashboards → Host Metrics / Fleet overview**. The import contains:
| Dashboard | Coverage |
|---|---|
| Fleet overview | Hosts seen, CPU/RAM/filesystem peaks, normalized load, latest sample per host, inventory |
| CPU & memory | Total/per-core CPU, user/system/iowait/steal, 1/5/15-minute load, RAM headroom, swap I/O, process states, context switches, interrupts |
| Storage | Per-mount used/free capacity, inode use, capacity table, disk throughput, IOPS, device busy time, requests in progress |
| Network | Per-interface RX/TX, packets, errors, drops and interface inventory |
Dark mode, a consistent grid, prominent summary cards, legends, readable units and navigation links are configured. Use **Add filter → tags.host** to select a server; `tags.environment` and `tags.role` select groups. Links navigate between dashboards but do not promise to retain unsaved filters; check the filter bar after navigation.
Dashboards start at **Last 1 hour**, refreshing every 15 seconds. Cards explicitly report peaks/minima or hosts seen over that range; they are not instantaneous health indicators. The freshness table shows the newest sample **inside the selected range**. A missing host is not proof that it is healthy. See [metric semantics](docs/METRICS.md).
## Optional: Observability Metric Analytics
The primary dashboards above query OpenSearch's metric documents. They work without Prometheus.
The dedicated **Observability → Metrics / Metric Analytics** interface supports Prometheus and a specific OpenTelemetry metrics schema. This project's Telegraf index is a custom schema, so enable the supplied Prometheus adapter to use that interface:
```bash
./scripts/enable-metric-analytics.sh
# Wait around 60 seconds after enabling it, then:
python3 ./scripts/check-metric-analytics.py
```
This starts two optional containers and registers the `host_prometheus` SQL data source. The bridge consumes **the same Kafka topic in its own consumer group**, reconstructs Telegraf metrics and exposes them for Prometheus. It does not collect the container/host itself and it does not bypass Kafka. The main Data Prepper → OpenSearch ingestion path continues independently.
Open **Observability → Metrics**, select **host_prometheus**, and select metrics such as `cpu_usage_active`, `mem_used_percent`, `disk_used_percent` and `net_bytes_recv_per_sec`. Save selected visualizations to an operational dashboard if desired. The four supplied NDJSON dashboards are regular OpenSearch dashboards; they are not imports for the separate operational-panel format.
Example PPL query in the observability query interface:
```text
source = host_prometheus.cpu_usage_active
| where cpu = 'cpu-total'
| stats avg(@value) by span(@timestamp, 1m), host
```
For native Prometheus exploration, also forward port 9090 through SSH. Example PromQL:
```promql
cpu_usage_active{cpu="cpu-total"}
mem_used_percent
disk_used_percent{path="/"}
net_bytes_recv_per_sec
```
The `*_per_sec` metrics are already rates: **do not apply `rate()` to them**. The bridge starts at the newest offset for a new group. It exposes the most recent value per series between scrapes; it is for live metric analytics, not lossless historical replay into Prometheus. Original sample timestamps are exported, so an old Kafka sample does not masquerade as a newly collected sample. OpenSearch remains the historical store for the requested pipeline.
## Files and operations
```text
compose.yaml All core services + optional metrics-analytics profile
agent/ Native configuration, enrichment/rates and systemd unit
configs/kafka/ KRaft broker and topic initialization
configs/data-prepper/ Kafka → JSON → dated OpenSearch indices
configs/opensearch/ Node settings, mappings, ISM retention, initialization
configs/dashboards/ Dashboards settings
configs/metrics-bridge/ Optional Kafka-to-Prometheus protocol adapter
configs/prometheus/ Optional scrape configuration
dashboards/ Importable NDJSON + equivalent JSON
scripts/ Installation, startup, import and end-to-end checks
tests/ Offline contracts and rate calculation checks
docs/ Metric semantics, operations, scaling and references
```
See [operations and troubleshooting](docs/OPERATIONS.md), [scaling to many servers](docs/SCALING.md), and [validation status](docs/VALIDATION.md).
## Validation status
The JSON/TOML contracts, dashboard reference graph and layout, rate mathematics, YAML syntax and shell syntax were checked locally. Image tags and configuration references were checked against upstream sources. **The full stack has not been executed here: this development workspace has no Docker runtime and is not your Debian VM.** The native agent and dashboard rendering still require the included Debian smoke tests and a browser check. Treat this as a complete POC implementation with explicit runtime validation steps, not a claim of a completed deployment or production certification.

35
agent/enrich.star Normal file
View File

@ -0,0 +1,35 @@
# Rate calculation happens BEFORE Kafka: replay cannot corrupt counter deltas.
# State is per measurement + complete tag set; counters never mix across devices.
state = {}
COUNTERS = {
"diskio": ["read_bytes", "write_bytes", "reads", "writes", "read_time", "write_time", "io_time"],
"net": ["bytes_recv", "bytes_sent", "packets_recv", "packets_sent", "err_in", "err_out", "drop_in", "drop_out"],
"kernel": ["context_switches", "interrupts", "processes_forked"],
"swap": ["in", "out"],
}
def apply(metric):
identity = metric.name + "|" + str(sorted(metric.tags.items()))
if metric.name in COUNTERS:
previous = state.get(identity)
current = {}
for field in COUNTERS[metric.name]:
if field in metric.fields:
current[field] = metric.fields[field]
if previous != None and metric.time > previous[0]:
elapsed = (metric.time - previous[0]) / 1000000000.0
# Suppress gaps and resets: never turn them into false rate spikes.
if elapsed <= 120.0:
for field in current:
if field in previous[1] and current[field] >= previous[1][field]:
metric.fields[field + "_per_sec"] = (current[field] - previous[1][field]) / elapsed
if metric.name == "diskio" and "io_time_per_sec" in metric.fields:
metric.fields["io_busy_percent"] = min(100.0, metric.fields["io_time_per_sec"] / 10.0)
if previous == None or metric.time > previous[0]:
state[identity] = (metric.time, current)
# Human-readable series distinguish same-named devices on different hosts.
device = metric.tags.get("path", metric.tags.get("interface", metric.tags.get("name", metric.tags.get("cpu", "host"))))
metric.tags["series"] = metric.tags.get("host", "unknown") + " / " + device
if metric.name == "system" and metric.fields.get("n_cpus", 0) > 0:
metric.fields["load1_per_cpu"] = metric.fields.get("load1", 0.0) / metric.fields["n_cpus"]
return metric

28
agent/telegraf-host-metrics.service Normal file
View File

@ -0,0 +1,28 @@
[Unit]
Description=Host metrics to Kafka (Telegraf)
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=0
[Service]
Type=simple
User=telegraf
Group=telegraf
EnvironmentFile=/etc/telegraf/host-metrics/environment
ExecStart=/usr/bin/telegraf --config /etc/telegraf/host-metrics/telegraf.conf
Restart=always
RestartSec=10
TimeoutStopSec=30
StateDirectory=telegraf-host-metrics
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=read-only
PrivateTmp=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
# No PrivateDevices/ProtectProc/PrivateNetwork: this service measures the host.
# No CAP_SYS_ADMIN or Docker socket is needed.
[Install]
WantedBy=multi-user.target

53
agent/telegraf.conf Normal file
View File

@ -0,0 +1,53 @@
[global_tags]
environment = "${METRICS_ENVIRONMENT}"
role = "${METRICS_ROLE}"
[agent]
interval = "15s"
round_interval = true
collection_jitter = "2s"
flush_interval = "5s"
flush_jitter = "1s"
metric_batch_size = 500
metric_buffer_limit = 100000
hostname = "${METRICS_HOST}"
omit_hostname = false
# Stable memory buffer. Restart loses unsent data. See docs/OPERATIONS.md.
buffer_strategy = "memory"
skip_processors_after_aggregators = true
[[inputs.cpu]]
percpu = true
totalcpu = true
collect_cpu_time = false
report_active = true
[[inputs.mem]]
[[inputs.swap]]
[[inputs.system]]
include = ["load", "cpus", "uptime"]
fieldexclude = ["uptime_format"]
[[inputs.disk]]
ignore_fs = ["tmpfs", "devtmpfs", "devfs", "overlay", "aufs", "squashfs", "nsfs", "proc", "sysfs", "cgroup", "cgroup2", "tracefs", "debugfs", "securityfs", "pstore", "fusectl", "configfs", "mqueue", "hugetlbfs"]
[[inputs.diskio]]
skip_serial_number = true
[[inputs.net]]
[inputs.net.tagdrop]
interface = ["all", "lo", "veth*", "docker*", "br-*"]
[[inputs.kernel]]
[[inputs.processes]]
[[processors.starlark]]
script = "/etc/telegraf/host-metrics/enrich.star"
[[outputs.kafka]]
brokers = ["${KAFKA_BOOTSTRAP}"]
topic = "host-metrics-v1"
client_id = "host-metrics-${METRICS_HOST}"
version = "3.6.0"
routing_tag = "host"
compression_codec = 1
required_acks = -1
idempotent_writes = true
max_retry = 10
data_format = "json"
json_timestamp_units = "1ms"

175
compose.yaml Normal file
View File

@ -0,0 +1,175 @@
name: host-metrics
x-logging: &logging
driver: json-file
options:
max-size: "10m"
max-file: "3"
services:
volume-init:
image: apache/kafka:${KAFKA_VERSION:-4.3.1}
user: "0:0"
entrypoint: ["/bin/sh", "-ec"]
command: ["chown 1000:1000 /data/kafka /data/dlq"]
volumes:
- kafka-data:/data/kafka
- prepper-dlq:/data/dlq
logging: *logging
kafka:
image: apache/kafka:${KAFKA_VERSION:-4.3.1}
user: "1000:1000"
restart: unless-stopped
environment:
KAFKA_HEAP_OPTS: -Xms512m -Xmx1g
LOG_DIR: /tmp/kafka-logs
KAFKA_ADVERTISED_HOST: ${KAFKA_ADVERTISED_HOST:?Run scripts/configure-access.sh with the VM LAN IP first}
entrypoint: ["/bin/bash", "/etc/kafka/start.sh"]
volumes:
- ./configs/kafka/server.properties:/etc/kafka/server.properties:ro
- ./configs/kafka/start.sh:/etc/kafka/start.sh:ro
- kafka-data:/var/lib/kafka/data
ports:
- "127.0.0.1:9092:9092"
- "${BIND_ADDRESS:?Run scripts/configure-access.sh with VM and workstation addresses}:9092:9092"
depends_on:
volume-init:
condition: service_completed_successfully
healthcheck:
test: ["CMD", "/opt/kafka/bin/kafka-topics.sh", "--bootstrap-server", "kafka:29092", "--list"]
interval: 15s
timeout: 10s
retries: 12
start_period: 30s
logging: *logging
kafka-init:
image: apache/kafka:${KAFKA_VERSION:-4.3.1}
entrypoint: ["/bin/bash", "/etc/kafka/create-topic.sh"]
volumes:
- ./configs/kafka/create-topic.sh:/etc/kafka/create-topic.sh:ro
depends_on:
kafka:
condition: service_healthy
logging: *logging
opensearch:
image: opensearchproject/opensearch:${OPENSEARCH_VERSION:-3.8.0}
restart: unless-stopped
environment:
DISABLE_INSTALL_DEMO_CONFIG: "true"
DISABLE_SECURITY_PLUGIN: "true"
OPENSEARCH_JAVA_OPTS: ${OPENSEARCH_JAVA_OPTS:--Xms2g -Xmx2g}
DATASOURCE_MASTER_KEY: ${DATASOURCE_MASTER_KEY:?Run scripts/start.sh to generate the datasource encryption key}
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- ./configs/opensearch/opensearch.yml:/usr/share/opensearch/config/opensearch.yml:ro
- opensearch-data:/usr/share/opensearch/data
ports:
- "127.0.0.1:9200:9200"
- "${BIND_ADDRESS:?Run scripts/configure-access.sh with VM and workstation addresses}:9200:9200"
healthcheck:
test: ["CMD-SHELL", "curl -fsS 'http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s' | grep -q '\"timed_out\":false'"]
interval: 15s
timeout: 10s
retries: 20
start_period: 60s
logging: *logging
opensearch-init:
image: opensearchproject/opensearch:${OPENSEARCH_VERSION:-3.8.0}
entrypoint: ["/bin/bash", "/bootstrap/bootstrap.sh"]
volumes:
- ./configs/opensearch:/bootstrap:ro
depends_on:
opensearch:
condition: service_healthy
logging: *logging
data-prepper:
image: opensearchproject/data-prepper:${DATA_PREPPER_VERSION:-2.16.0}
restart: unless-stopped
environment:
JAVA_TOOL_OPTIONS: -Xms256m -Xmx1g
volumes:
- ./configs/data-prepper/pipelines.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml:ro
- ./configs/data-prepper/data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml:ro
- prepper-dlq:/usr/share/data-prepper/dlq
ports:
- "127.0.0.1:4900:4900"
- "${BIND_ADDRESS:?Run scripts/configure-access.sh with VM and workstation addresses}:4900:4900"
depends_on:
kafka-init:
condition: service_completed_successfully
opensearch-init:
condition: service_completed_successfully
logging: *logging
dashboards:
image: opensearchproject/opensearch-dashboards:${OPENSEARCH_VERSION:-3.8.0}
restart: unless-stopped
environment:
DISABLE_SECURITY_DASHBOARDS_PLUGIN: "true"
volumes:
- ./configs/dashboards/opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml:ro
ports:
- "127.0.0.1:5601:5601"
- "${BIND_ADDRESS:?Run scripts/configure-access.sh with VM and workstation addresses}:5601:5601"
depends_on:
opensearch-init:
condition: service_completed_successfully
healthcheck:
test: ["CMD-SHELL", "curl -fsS http://localhost:5601/api/status >/dev/null"]
interval: 15s
timeout: 10s
retries: 20
start_period: 60s
logging: *logging
# Optional independent Kafka consumer. No host collection happens here.
metrics-bridge:
image: telegraf:${TELEGRAF_VERSION:-1.40.0}
profiles: [metrics-analytics]
restart: unless-stopped
command: ["telegraf", "--config", "/etc/telegraf/bridge.conf"]
volumes:
- ./configs/metrics-bridge/telegraf.conf:/etc/telegraf/bridge.conf:ro
ports:
- "127.0.0.1:9273:9273"
- "${BIND_ADDRESS:?Run scripts/configure-access.sh with VM and workstation addresses}:9273:9273"
depends_on:
kafka-init:
condition: service_completed_successfully
logging: *logging
prometheus:
image: prom/prometheus:${PROMETHEUS_VERSION:-v3.14.0}
profiles: [metrics-analytics]
restart: unless-stopped
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention.time=7d
- --storage.tsdb.retention.size=5GB
volumes:
- ./configs/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus-data:/prometheus
ports:
- "127.0.0.1:9090:9090"
- "${BIND_ADDRESS:?Run scripts/configure-access.sh with VM and workstation addresses}:9090:9090"
depends_on:
- metrics-bridge
logging: *logging
volumes:
kafka-data:
opensearch-data:
prepper-dlq:
prometheus-data:

9
configs/dashboards/opensearch_dashboards.yml Normal file
View File

@ -0,0 +1,9 @@
server.name: host-metrics
server.host: 0.0.0.0
opensearch.hosts: ["http://opensearch:9200"]
opensearch.requestTimeout: 60000
opensearch.shardTimeout: 60000
workspace.enabled: false
uiSettings.overrides:
"theme:darkMode": true
"dateFormat:tz": UTC

2
configs/data-prepper/data-prepper-config.yaml Normal file
View File

@ -0,0 +1,2 @@
ssl: false
serverPort: 4900

43
configs/data-prepper/pipelines.yaml Normal file
View File

@ -0,0 +1,43 @@
host-metrics:
workers: 2
delay: 1000
source:
kafka:
bootstrap_servers: ["kafka:29092"]
client_dns_lookup: use_all_dns_ips
encryption:
type: none
acknowledgments: true
topics:
- name: host-metrics-v1
group_id: opensearch-host-metrics-v1
workers: 2
# Decode the Kafka VALUE as JSON; plaintext uses the producer key as
# the field name (e.g. hostname), so parse_json(source=message) fails.
serde_format: json
key_mode: discard
auto_offset_reset: earliest
auto_commit: false
buffer:
bounded_blocking:
buffer_size: 16384
batch_size: 512
processor:
- date:
match:
- key: timestamp
patterns: [epoch_milli]
destination: "@timestamp"
to_origination_metadata: true
- date:
from_time_received: true
destination: ingested_at
sink:
- opensearch:
hosts: ["http://opensearch:9200"]
index_type: custom
index: "host-metrics-v1-%{yyyy.MM.dd}"
bulk_size: 4
flush_timeout: 5000
# Unlimited retry for transient indexing failures; permanent failures go to DLQ.
dlq_file: /usr/share/data-prepper/dlq/failed-events.json

9
configs/kafka/create-topic.sh Executable file
View File

@ -0,0 +1,9 @@
#!/usr/bin/env bash
set -euo pipefail
/opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:29092 \
--create --if-not-exists --topic host-metrics-v1 \
--partitions 6 --replication-factor 1 \
--config cleanup.policy=delete --config retention.ms=259200000 \
--config retention.bytes=1073741824 --config min.insync.replicas=1
/opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:29092 \
--describe --topic host-metrics-v1

21
configs/kafka/server.properties Normal file
View File

@ -0,0 +1,21 @@
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@kafka:29093
controller.listener.names=CONTROLLER
listeners=INTERNAL://:29092,EXTERNAL://:9092,CONTROLLER://:29093
# start.sh renders this marker into a writable runtime configuration.
advertised.listeners=INTERNAL://kafka:29092,EXTERNAL://__KAFKA_ADVERTISED_HOST__:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
inter.broker.listener.name=INTERNAL
log.dirs=/var/lib/kafka/data
num.partitions=6
default.replication.factor=1
min.insync.replicas=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
group.initial.rebalance.delay.ms=0
auto.create.topics.enable=false
log.retention.hours=72
log.retention.check.interval.ms=300000
log.segment.bytes=268435456

12
configs/kafka/start.sh Executable file
View File

@ -0,0 +1,12 @@
#!/usr/bin/env bash
set -euo pipefail
advertised_host=${KAFKA_ADVERTISED_HOST:?Set the VM LAN IP or DNS name}
[[ $advertised_host =~ ^[a-zA-Z0-9][a-zA-Z0-9.-]*$ ]] || {
echo 'KAFKA_ADVERTISED_HOST must be an IPv4 address or DNS name, without protocol/port.' >&2
exit 1
}
sed "s/__KAFKA_ADVERTISED_HOST__/$advertised_host/g" \
/etc/kafka/server.properties > /tmp/host-metrics-server.properties
/opt/kafka/bin/kafka-storage.sh format --ignore-formatted \
--cluster-id 'MkU3OEVBNTcwNTJENDM2Qk' --config /tmp/host-metrics-server.properties
exec /opt/kafka/bin/kafka-server-start.sh /tmp/host-metrics-server.properties

73
configs/metrics-bridge/telegraf.conf Normal file
View File

@ -0,0 +1,73 @@
# Kafka-to-Prometheus adapter only; no local host inputs.
[agent]
flush_interval = "5s"
metric_batch_size = 500
metric_buffer_limit = 10000
omit_hostname = true
[[inputs.kafka_consumer]]
brokers = ["kafka:29092"]
topics = ["host-metrics-v1"]
consumer_group = "prometheus-host-metrics-v1"
kafka_version = "3.6.0"
offset = "newest"
max_undelivered_messages = 1000
data_format = "json_v2"
[[inputs.kafka_consumer.json_v2]]
measurement_name_path = "name"
timestamp_path = "timestamp"
timestamp_format = "unix_ms"
timestamp_timezone = "UTC"
[[inputs.kafka_consumer.json_v2.object]]
path = "fields"
[[inputs.kafka_consumer.json_v2.tag]]
path = "tags.host"
rename = "host"
optional = true
[[inputs.kafka_consumer.json_v2.tag]]
path = "tags.environment"
rename = "environment"
optional = true
[[inputs.kafka_consumer.json_v2.tag]]
path = "tags.role"
rename = "role"
optional = true
[[inputs.kafka_consumer.json_v2.tag]]
path = "tags.series"
rename = "series"
optional = true
[[inputs.kafka_consumer.json_v2.tag]]
path = "tags.cpu"
rename = "cpu"
optional = true
[[inputs.kafka_consumer.json_v2.tag]]
path = "tags.path"
rename = "path"
optional = true
[[inputs.kafka_consumer.json_v2.tag]]
path = "tags.device"
rename = "device"
optional = true
[[inputs.kafka_consumer.json_v2.tag]]
path = "tags.fstype"
rename = "fstype"
optional = true
[[inputs.kafka_consumer.json_v2.tag]]
path = "tags.mode"
rename = "mode"
optional = true
[[inputs.kafka_consumer.json_v2.tag]]
path = "tags.name"
rename = "name"
optional = true
[[inputs.kafka_consumer.json_v2.tag]]
path = "tags.interface"
rename = "interface"
optional = true
[[outputs.prometheus_client]]
listen = ":9273"
metric_version = 1
expiration_interval = "60s"
string_as_label = false
export_timestamp = true
collectors_exclude = ["gocollector", "process"]

16
configs/network/access-rules.sh Executable file
View File

@ -0,0 +1,16 @@
#!/usr/bin/env bash
# Restrict only this stack's published LAN ports, not SSH or other containers.
set -euo pipefail
source /etc/host-metrics-access.conf
iptables -w 5 -N DOCKER-USER 2>/dev/null || iptables -w 5 -S DOCKER-USER >/dev/null
iptables -w 5 -N HOST-METRICS-ACCESS 2>/dev/null || iptables -w 5 -F HOST-METRICS-ACCESS
for port in 4900 5601 9090 9092 9200 9273; do
iptables -w 5 -A HOST-METRICS-ACCESS -p tcp -s "$WORKSTATION_IP" \
-m conntrack --ctdir ORIGINAL --ctorigdst "$VM_LAN_IP" --ctorigdstport "$port" -j RETURN
iptables -w 5 -A HOST-METRICS-ACCESS -p tcp \
-m conntrack --ctdir ORIGINAL --ctorigdst "$VM_LAN_IP" --ctorigdstport "$port" -j DROP
done
iptables -w 5 -A HOST-METRICS-ACCESS -j RETURN
if ! iptables -w 5 -C DOCKER-USER -j HOST-METRICS-ACCESS 2>/dev/null; then
iptables -w 5 -I DOCKER-USER 1 -j HOST-METRICS-ACCESS
fi

12
configs/network/host-metrics-access.service Normal file
View File

@ -0,0 +1,12 @@
[Unit]
Description=Workstation access restriction for host metrics Docker ports
Before=docker.service
After=network-pre.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/host-metrics-access
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target

14
configs/opensearch/bootstrap.sh Executable file
View File

@ -0,0 +1,14 @@
#!/usr/bin/env bash
set -euo pipefail
BASE=http://opensearch:9200
code=$(curl -sS -o /tmp/host-metrics-policy.json -w '%{http_code}' \
"$BASE/_plugins/_ism/policies/host-metrics-retention")
case "$code" in
404) curl -fsS -X PUT "$BASE/_plugins/_ism/policies/host-metrics-retention" \
-H 'Content-Type: application/json' --data-binary @/bootstrap/retention-policy.json ;;
200) echo 'Preserving existing retention policy.' ;;
*) cat /tmp/host-metrics-policy.json; exit 1 ;;
esac
curl -fsS -X PUT "$BASE/_index_template/host-metrics-v1" \
-H 'Content-Type: application/json' --data-binary @/bootstrap/index-template.json
echo 'OpenSearch template and retention ready.'

178
configs/opensearch/index-template.json Normal file
View File

@ -0,0 +1,178 @@
{
"index_patterns": [
"host-metrics-v1-*"
],
"priority": 200,
"version": 1,
"template": {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"refresh_interval": "5s",
"index.mapping.total_fields.limit": 500
},
"mappings": {
"dynamic": true,
"properties": {
"@timestamp": {
"type": "date"
},
"ingested_at": {
"type": "date"
},
"timestamp": {
"type": "long"
},
"name": {
"type": "keyword"
},
"tags": {
"type": "object",
"dynamic": true
},
"fields": {
"type": "object",
"dynamic": true,
"properties": {
"usage_active": {
"type": "double"
},
"used_percent": {
"type": "double"
},
"load1_per_cpu": {
"type": "double"
},
"uptime": {
"type": "double"
},
"n_cpus": {
"type": "double"
},
"load1": {
"type": "double"
},
"usage_iowait": {
"type": "double"
},
"usage_user": {
"type": "double"
},
"usage_system": {
"type": "double"
},
"usage_steal": {
"type": "double"
},
"load5": {
"type": "double"
},
"load15": {
"type": "double"
},
"available": {
"type": "double"
},
"cached": {
"type": "double"
},
"buffered": {
"type": "double"
},
"in_per_sec": {
"type": "double"
},
"out_per_sec": {
"type": "double"
},
"running": {
"type": "double"
},
"blocked": {
"type": "double"
},
"zombie": {
"type": "double"
},
"context_switches_per_sec": {
"type": "double"
},
"interrupts_per_sec": {
"type": "double"
},
"inodes_used_percent": {
"type": "double"
},
"io_busy_percent": {
"type": "double"
},
"free": {
"type": "double"
},
"total": {
"type": "double"
},
"read_bytes_per_sec": {
"type": "double"
},
"write_bytes_per_sec": {
"type": "double"
},
"reads_per_sec": {
"type": "double"
},
"writes_per_sec": {
"type": "double"
},
"iops_in_progress": {
"type": "double"
},
"bytes_recv_per_sec": {
"type": "double"
},
"bytes_sent_per_sec": {
"type": "double"
},
"err_in_per_sec": {
"type": "double"
},
"drop_in_per_sec": {
"type": "double"
},
"packets_recv_per_sec": {
"type": "double"
},
"packets_sent_per_sec": {
"type": "double"
},
"err_out_per_sec": {
"type": "double"
},
"drop_out_per_sec": {
"type": "double"
}
}
}
},
"dynamic_templates": [
{
"tags": {
"path_match": "tags.*",
"mapping": {
"type": "keyword",
"ignore_above": 1024
}
}
},
{
"numeric_fields": {
"path_match": "fields.*",
"mapping": {
"type": "double"
}
}
}
]
}
}
}

7
configs/opensearch/opensearch.yml Normal file
View File

@ -0,0 +1,7 @@
cluster.name: host-metrics-poc
node.name: opensearch-1
network.host: 0.0.0.0
discovery.type: single-node
bootstrap.memory_lock: true
plugins.security.disabled: true
plugins.query.datasources.encryption.masterkey: "${DATASOURCE_MASTER_KEY}"

37
configs/opensearch/retention-policy.json Normal file
View File

@ -0,0 +1,37 @@
{
"policy": {
"description": "Delete host metric daily indices after 14 days; applies only to host-metrics-v1-*.",
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "14d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"delete": {}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"host-metrics-v1-*"
],
"priority": 200
}
]
}
}

8
configs/prometheus/prometheus.yml Normal file
View File

@ -0,0 +1,8 @@
global:
scrape_interval: 15s
scrape_timeout: 10s
scrape_configs:
- job_name: host-metrics-kafka
honor_labels: true
static_configs:
- targets: ["metrics-bridge:9273"]

1234
dashboards/host-metrics.json Normal file
View File

File diff suppressed because one or more lines are too long

48
dashboards/host-metrics.ndjson Normal file
View File

File diff suppressed because one or more lines are too long

59
docs/METRICS.md Normal file
View File

@ -0,0 +1,59 @@
# Metric schema and dashboard semantics
Every Kafka message is one Telegraf JSON measurement. It is **not** an OTLP envelope and it is not a JSON array/batch. Telegraf's Kafka output serializes each measurement separately. The Kafka message key is the `host` tag; the topic is `host-metrics-v1`.
Example (values illustrative):
```json
{
"name": "cpu",
"tags": {
"host": "debian-monitor-01",
"environment": "homelab",
"role": "monitoring",
"cpu": "cpu-total",
"series": "debian-monitor-01 / cpu-total"
},
"fields": {
"usage_active": 18.5,
"usage_user": 12.1,
"usage_system": 4.2,
"usage_idle": 81.5
},
"timestamp": 1789689600000
}
```
Data Prepper decodes the `message` field, converts the **millisecond epoch** into `@timestamp`, records `ingested_at`, and removes the redundant encoded message. The date processor also sets the origination metadata used for time-based sink index naming. Normal records land in `host-metrics-v1-YYYY.MM.dd`. Times and daily boundaries are UTC.
`name` and `tags.*` are keywords. `@timestamp` and `ingested_at` are dates. Numeric `fields.*` use doubles; this prevents a shared field such as `used_percent` becoming an integer mapping because the first observed value was zero. Exact very large integer counters above 2^53 are not preserved by double mappings; the dashboards primarily use percentages, current capacities, and derived rates. The original JSON remains in `_source`.
| Measurement | Interpretation |
|---|---|
| cpu | Percentages on a 0–100 scale. Fleet cards use `cpu-total`; per-core charts exclude it. `usage_active` is Telegraf's non-idle measure. I/O wait and steal are shown separately for diagnosis. |
| mem | Telegraf/Linux memory semantics. `available` includes reclaimable memory; cached memory is not necessarily pressure. `used_percent` is not a kernel OOM prediction. |
| system | 1/5/15-minute load, logical CPU count, uptime seconds. `load1_per_cpu` is load divided by logical CPUs. Load includes uninterruptible tasks, not just CPU demand. |
| disk | Capacity and inode usage per mounted real filesystem. Docker overlay and pseudo filesystems are excluded to avoid duplicates. Separate bind mounts may still describe the same underlying storage. |
| diskio | Kernel block-device counters. Derived read/write bytes/s, operations/s, and `io_busy_percent`. Layered devices (LVM/device-mapper/physical disks) may represent the same I/O; do not add them blindly. Busy percentage is not a reliable saturation limit for highly parallel NVMe/RAID. |
| net | Interface byte/packet/error/drop counters and derived rates. Loopback, veth and conventional Docker bridges are excluded. Adjust the tag filter if you want those interfaces. Rates are bytes/s, not bits/s. |
| swap | Capacity and derived swap-in/out bytes/s. On hosts with no swap, capacity metrics can be zero and activity charts may be empty. |
| processes | Host process counts by state; no per-process command-line collection. Permission restrictions such as `hidepid` may limit visibility. |
| kernel | Context switches, interrupts and fork counts; derived per-second rates. |
## Rate handling
The Starlark processor keeps previous counter values per **measurement + complete tag set**. A rate is `(current - previous) / actual_elapsed_seconds`, not divided by an assumed fixed polling interval. Counters on different hosts/devices never share a baseline. On the first sample, counter reset, repeated/out-of-order timestamp, or gap over 120 seconds, the affected rate is omitted. The next valid interval recovers automatically. Negative rates are never fabricated or silently presented as zero.
The state is local to the agent and resets after restart. Raw counters are kept alongside rates. Kafka replay preserves precomputed rates. For disk busy time, a millisecond/second rate is divided by 10 and capped at 100%. Processor state grows with distinct tag combinations; the fixed host-input set has bounded practical cardinality, but endlessly changing interfaces/mount names could grow it during a long-running process.
## Reading the dashboards
- Every time chart averages values within its automatically chosen bucket, separately for each selected series. Zoom in for detail; wider ranges smooth short spikes.
- Summary cards explicitly say **peak**, **minimum**, or **hosts seen in selected range**. They are time-range statistics. They do not promise a latest-value fleet total.
- `tags.series` combines host and mount/interface/device/CPU to keep two hosts' `eth0` or `/` separate.
- Charts and tables select the top 30 terms by document count. Use a host filter for large fleets or increase the limit deliberately.
- Latest-sample tables only search the selected time window. A host that stopped reporting before that window will be absent. Inventory-backed absent-host alerting is a separate production concern.
- No-data is not zero. A blank rate chart during the first two collection cycles is normal; prolonged blanks need investigation.
- The single-node pipeline is at least once after Kafka: restarts/retries can create duplicate indexed documents. There is no end-to-end exactly-once guarantee. Min/max are insensitive to identical duplicates; averages and counts can be biased. Do not treat document counts as exact sample counts.
The optional Prometheus exporter preserves metric sample timestamps. It presents a live snapshot per series, so its history can differ from OpenSearch when Kafka is replayed or when several records arrive between scrapes. It does not reproduce every Kafka sample in Prometheus.

96
docs/OPERATIONS.md Normal file
View File

@ -0,0 +1,96 @@
# Operations and troubleshooting
Run these commands from the deployment directory on Debian.
## Health and data
```bash
docker compose ps -a
docker compose logs --tail=100 kafka data-prepper opensearch dashboards
sudo journalctl -u telegraf-host-metrics -n 100 --no-pager
./scripts/smoke-test.sh
```
The init containers should exit with code 0. A running Data Prepper container alone does not prove delivery; use the ingestion check. For host-specific validation:
```bash
METRICS_HOST=debian-monitor-01 python3 scripts/check-ingestion.py
```
Inspect the topic and lag:
```bash
docker compose exec -T kafka /opt/kafka/bin/kafka-consumer-groups.sh \
--bootstrap-server kafka:29092 --describe --group opensearch-host-metrics-v1
curl -fsS 'http://127.0.0.1:9200/_cat/indices/host-metrics-v1-*?v'
curl -fsS 'http://127.0.0.1:9200/host-metrics-v1-*/_search?size=1&sort=@timestamp:desc'
curl -fsS 'http://127.0.0.1:9200/_plugins/_ism/explain/host-metrics-v1-*?show_policy=true'
```
Lag may oscillate while a batch is being acknowledged. Sustained growth means the consumer cannot keep up or the sink is unhealthy. `auto_offset_reset=earliest` only applies when there is no valid committed offset; it does not rewind a running group.
## Storage, retention and delivery limits
- Kafka records persist in `kafka-data`. Retention is **72 hours or approximately 1 GiB per partition**, whichever deletes old data first (about 6 GiB for this topic, plus segment and broker overhead). Retention is segment-based and not a hard immediate disk quota. It applies even when a consumer is behind.
- OpenSearch daily indices persist in `opensearch-data`. ISM deletes matching indices after **14 days of index age**, checked periodically. It does not delete individual documents by their exact event age. Very old replayed records may create an old-named index that is newly created and therefore remains for another 14 days. The template has one primary shard and zero replicas.
- Data Prepper uses an in-memory processing buffer, with end-to-end Kafka acknowledgments and disabled periodic offset auto-commit. Kafka is the durable queue. Permanent indexing failures can be written to `prepper-dlq`; check logs and this volume. Transient sink failures retry with backpressure. There is no distributed transaction between Kafka and OpenSearch.
- Native Telegraf has a **100,000-measurement memory output buffer**, with acknowledged/idempotent Kafka writes. While the agent stays alive it retries failed output writes. When full, data is dropped; a process/VM restart loses unsent buffered data. Idempotent Kafka writes do not make the whole pipeline exactly once.
- Telegraf 1.40 offers an **experimental** disk buffer. To evaluate it, set `buffer_strategy = "disk"`, `buffer_directory = "/var/lib/telegraf-host-metrics/buffer"` and `buffer_disk_sync = true` in the native config, then restart the unit and test outage/restart behavior. The systemd StateDirectory is writable. This POC defaults to the stable memory buffer and does not claim durable collection during agent outages.
- Optional Prometheus keeps 7 days or 5 GB of TSDB blocks; additional WAL/head space is needed. Its adapter is a live snapshot, not an archive importer.
- Container logs rotate at 10 MB × 3 files per container. The Data Prepper DLQ file is not automatically rotated. Inspect/export it regularly and monitor disk usage.
Inspect the DLQ without modifying it:
```bash
docker compose exec -T data-prepper sh -c \
'ls -lah /usr/share/data-prepper/dlq; tail -n 10 /usr/share/data-prepper/dlq/failed-events.json'
```
The file may not exist if no failures occurred. DLQ contents include failed event/context records, not necessarily original Kafka JSON. Do not blindly pipe the file into Kafka; inspect it, correct the cause and extract original metric events for deliberate replay.
## Common failures
| Symptom | Checks / remedy |
|---|---|
| OpenSearch exits during startup | Inspect logs for `vm.max_map_count`, memory or ulimit errors. Run `prepare-host.sh`; verify available RAM and Docker resources. |
| Kafka healthy, native agent cannot send | Confirm the published listener is `127.0.0.1:9092`, the topic exists and the agent runs on the same VM. `kafka:29092` is only resolvable inside Docker. |
| Agent can bootstrap remotely but then fails | Kafka returns `advertised.listeners` to clients. The advertised address must be reachable from that client; changing only the bootstrap address is insufficient. |
| Data Prepper reports SSL errors connecting to Kafka | Keep `encryption.type: none` for this plaintext POC. Data Prepper otherwise defaults to TLS. |
| Dashboards says “not ready” | Check OpenSearch health and `opensearch-init`. Server/client versions must match; security must be disabled consistently on both sides here. |
| Import succeeded but no charts | Check Last 1 hour, clear accidental filters, run the smoke test, inspect `@timestamp`, host UTC time and `tags.host`. Do not create a different index pattern with the same title; the supplied visuals reference `hm-index-v1`. |
| Mapping failures after editing agent fields | Custom `fields.*` are numeric. Do not add string fields to that object. A changed field type requires a new schema/index version, not just a template edit. |
| Root FS pressure / read-only index | Free disk space, inspect OpenSearch disk watermarks, then clear the affected read-only block after resolving capacity. Do not disable disk watermarks to hide the problem. |
| Prometheus has no host metrics | Check `docker compose --profile metrics-analytics logs metrics-bridge prometheus`. A new bridge starts from newest Kafka offsets; wait for new agent samples. Inspect `http://127.0.0.1:9090/api/v1/targets` on the VM. |
| Metric Analytics cannot list the source | Run `register-prometheus.py`; inspect its HTTP error and OpenSearch logs. Confirm the encryption key is configured and the URI is `http://prometheus:9090`, not localhost. Use `check-metric-analytics.py` to test PPL separately from the UI. |
## Restart, update and remove
```bash
# Apply edited container configurations.
docker compose up -d --force-recreate data-prepper dashboards
# Apply an edited native collector config.
sudo systemctl restart telegraf-host-metrics
# Stop containers while retaining named volumes.
docker compose --profile metrics-analytics down
# Stop the dedicated native service.
sudo systemctl disable --now telegraf-host-metrics
```
Restart core services with `./scripts/start.sh`; re-enable the optional profile with `./scripts/enable-metric-analytics.sh`. Running the enable script also rechecks the data-source registration. Already-created Kafka topic settings and the existing ISM policy are deliberately preserved by the initializers; changing a source JSON/property file does not update those existing resources automatically. Apply updates through their respective management APIs after reviewing the intended change.
To modify the ISM policy, GET the policy with its `_seq_no` and `_primary_term`, then PUT the edited body with optimistic concurrency parameters. Verify the managed-index policy state afterward. To change Kafka topic retention, use `kafka-configs.sh --alter --entity-type topics --entity-name host-metrics-v1 --add-config ...`. Changing index templates affects future indices, not existing ones.
Before version upgrades, save OpenSearch snapshots and your configuration, `.env` and dashboards; test the new versions in another deployment. Do not downgrade OpenSearch in place after a newer version has opened its data directory. Update the native package pin together with the bridge image when changing Telegraf. Regenerate dashboard/mapping assets with `python3 scripts/generate-assets.py`, then reimport the NDJSON.
**Destructive teardown:** `docker compose --profile metrics-analytics down -v` removes the deployment's named volumes and their data. It does not uninstall the native agent. This is never run by the setup scripts.
## Suggested POC acceptance exercise
1. Run the smoke test and import the dashboards. Verify CPU/RAM/`/` filesystem values against `top`, `free` and `df` on the VM, allowing for different memory definitions and averaging windows.
2. Briefly create CPU/network/disk activity using your own disposable test workload. Verify the corresponding charts and host filter.
3. Stop Data Prepper for one minute. Verify the agent still sends, Kafka offsets advance, and consumption catches up after restarting Data Prepper.
4. Stop Kafka briefly, then restart it. Inspect the agent retry logs and recovery. Keep the outage shorter than the memory buffer capacity; do not claim a measured recovery limit without testing at your load.
5. Restart the native agent. Confirm counters warm up without negative or enormous rate spikes.
6. Enable optional Metric Analytics and run its separate checker. Verify at least one chart in the UI.
These are manual acceptance steps, not tests already performed in the development workspace.

56
docs/SCALING.md Normal file
View File

@ -0,0 +1,56 @@
# Scaling beyond the single-host POC
The schema and host-keyed Kafka topic already support multiple agents. The supplied deployment is **single-node and not highly available**: one VM failure interrupts collection, queueing, indexing and dashboards. Moving to many servers requires the following infrastructure changes, not just a larger heap.
## 1. Add a secured remote Kafka listener
Keep separate internal and external listeners. Replace the EXTERNAL advertised address with a DNS name reachable from each agent and bind the host port to the intended private/VPN interface. For example, a future remote listener could advertise `metrics-kafka.example.internal:9094`; containers should continue to use the internal listener.
Configure TLS with trusted server certificates and SASL/SCRAM or mutual TLS. Add Kafka ACLs so agents can produce only to the metric topic and consumer identities can read only their assigned topics/groups. Mount client CA/certificate/key files on each native host and configure Telegraf's Kafka TLS/auth settings. Keep private keys readable only by the agent. The supplied plaintext listener is a local POC configuration, not a ready remote deployment.
Install the same agent bundle on each Debian host with a unique `METRICS_HOST`, environment/role tags, and the reachable bootstrap endpoint. Update the output's broker list for multiple bootstrap brokers when a cluster exists. The installer exposes one bootstrap string for the POC; edit the TOML broker array or distribute it through configuration management for several brokers.
## 2. Replicate Kafka
- Use at least three brokers and a production KRaft controller quorum. Avoid colocating all replicas on one failure domain. Use separate controllers where justified by scale/availability requirements.
- Set topic replication factor to 3 and `min.insync.replicas=2`; retain producer `required_acks=-1` and idempotence. Increase the internal offset and transaction-state topic replication settings too.
- Migrate existing topic replicas with Kafka's reassignment tooling. Changing defaults does not replicate existing data automatically.
- Begin with the existing six partitions and measure load and skew. Host-keyed routing preserves locality; an unusually busy host can dominate a partition. Increasing partitions changes future key placement, so plan ordering-sensitive transitions carefully.
- Size retention for realistic sink outages and Kafka consumption lag, with headroom for segments and broker recovery. A full topic partition loses its oldest data regardless of consumer progress.
## 3. Scale Data Prepper consumers
Run additional instances of the same pipeline using **the same group ID**. The group distributes partitions among consumers; different group IDs would independently consume and index the full stream, duplicating data. The POC has two consumers per instance and six partitions, so up to three instances can provide six active consumers before further instances become idle (other pipeline worker threads are distinct from Kafka consumers).
There is no fixed `container_name` or published Data Prepper port, so a local capacity experiment can use:
```bash
docker compose up -d --scale data-prepper=3
```
**First give each instance a separate DLQ file/volume or an appropriately configured external DLQ sink.** The POC's one shared local DLQ file is not safe for concurrent writers. A multi-host orchestrator should provide separate instance identity, logs and failure storage. The single-host scale command is an illustration; the supplied shared-DLQ configuration must be adjusted before using it.
The OpenSearch sink remains a bottleneck if indexing cannot keep up. Monitor consumer lag, sink retries, DLQ growth, CPU, heap, processing throughput and the OpenSearch write thread pool. End-to-end acknowledgments protect the offset boundary but allow duplicates after failures. If exact accounting matters, introduce a stable per-observation document ID and verify deduplication under replay before relying on counts.
## 4. Scale and secure OpenSearch
Build a multi-node cluster with appropriately sized cluster-manager and data roles; enable security and trusted TLS for node transport and REST. Replace `discovery.type: single-node` with proper discovery/bootstrap configuration. Use least-privilege ingestion and dashboard users. Enable the matching Dashboards security plugin and HTTPS access via a reverse proxy or the application's TLS settings.
Change the metric template to use replicas once enough data nodes exist. For example, one replica needs another data node; setting one replica on this single-node POC only makes indices yellow without adding availability. Choose shard counts from actual volume, shard sizes and recovery objectives. Daily one-shard indices suit the POC; size-based rollover/write aliases may fit larger or variable fleets. Avoid one tiny index/shard per host.
Take tested OpenSearch snapshots to external storage. Kafka retention is not an OpenSearch backup; it may not retain the full indexed history, metadata, dashboards or security configuration. Use ISM hot/warm/delete transitions and tested rollup/downsampling workflows for long metric retention. Keep schema versions explicit when fields change.
## 5. Measure capacity and service health
A measurement here contains multiple fields, so a measurement is not the same as one scalar time series. As an example, **40 measurements per 15-second interval** produce **230,400 documents/day/host**. At 100 hosts that is 23.04 million documents/day. At an illustrative 1 KB serialized JSON per measurement, that is roughly 23 GB/day of input before index compression, replicas, retained `_source` and storage overhead. Measure actual `pri.store.size`, input byte rate and cardinality rather than using that example as a promise.
For larger fleets:
- Increase collection intervals for low-change capacity signals and control device/tag cardinality. Disable per-core collection if total CPU is sufficient.
- Keep raw high-resolution retention short and use appropriate aggregate retention where needed.
- Maintain an inventory of expected hosts and alert on absent hosts, not just threshold crossings in received data.
- Add independent monitoring of Kafka, Data Prepper, OpenSearch and the native agents. The POC monitors host resources, not every component's internal metrics. Monitoring the monitoring system on the same VM cannot report when that VM is down.
- Add tested CPU/memory/disk/lag/freshness alerting with agreed durations and destinations. No alert messages or external notification channels are configured here.
- Treat the optional one-instance Prometheus bridge as a separate scaling problem: multiple consumers need unique scrape targets and consistent series ownership. For high-availability, long-term metric analytics, evaluate a supported Prometheus-compatible backend and a deliberately designed metrics delivery path.
Do not expose the POC's unauthenticated ports as a shortcut to remote deployment.

41
docs/SOURCES.md Normal file
View File

@ -0,0 +1,41 @@
# Upstream references
Checked on 2026-09-17. Configuration is pinned to the versions in `.env.example`; upstream `latest` pages may change after this date.
## Release and image verification
- [Kafka downloads](https://kafka.apache.org/community/downloads/) — stable 4.3.1; newer release candidates were excluded.
- [Kafka official container image](https://hub.docker.com/r/apache/kafka/tags?name=4.3.1).
- [OpenSearch 3.8.0 release](https://github.com/opensearch-project/OpenSearch/releases/tag/3.8.0).
- [OpenSearch Dashboards 3.8.0 release](https://github.com/opensearch-project/OpenSearch-Dashboards/releases/tag/3.8.0).
- [Data Prepper 2.16.0 release](https://github.com/opensearch-project/data-prepper/releases/tag/2.16.0).
- [Telegraf 1.40.0 release](https://github.com/influxdata/telegraf/releases/tag/v1.40.0).
- [Prometheus 3.14.0 release](https://github.com/prometheus/prometheus/releases/tag/v3.14.0).
## Collection and wire format
- [Telegraf Debian installation and repository signing key](https://docs.influxdata.com/telegraf/v1/install/).
- [Pinned Kafka output](https://github.com/influxdata/telegraf/blob/v1.40.0/plugins/outputs/kafka/README.md).
- [Pinned Kafka output serialization implementation](https://github.com/influxdata/telegraf/blob/v1.40.0/plugins/outputs/kafka/kafka.go) — one serialized metric per message.
- [Pinned JSON serializer](https://github.com/influxdata/telegraf/blob/v1.40.0/plugins/serializers/json/README.md).
- [Pinned host input plugins](https://github.com/influxdata/telegraf/tree/v1.40.0/plugins/inputs).
- [Pinned Starlark processor](https://github.com/influxdata/telegraf/blob/v1.40.0/plugins/processors/starlark/README.md).
- [Agent buffering configuration](https://github.com/influxdata/telegraf/blob/v1.40.0/docs/CONFIGURATION.md) — disk mode is experimental.
## Processing and visualization
- [Data Prepper Kafka source](https://docs.opensearch.org/latest/data-prepper/pipelines/configuration/sources/kafka/) — plaintext parsing, TLS default and end-to-end acknowledgments.
- [Data Prepper date processor](https://docs.opensearch.org/latest/data-prepper/pipelines/configuration/processors/date/) — epoch milliseconds and origination timestamps.
- [OpenSearch sink](https://docs.opensearch.org/latest/data-prepper/pipelines/configuration/sinks/opensearch/) — bulk, retry and DLQ options.
- [Data Prepper pinned source](https://github.com/opensearch-project/data-prepper/tree/2.16.0) — Kafka source, sink configuration and UID 1000 container user inspected.
- [OpenSearch Docker installation](https://docs.opensearch.org/latest/install-and-configure/install-opensearch/docker/).
- [Index State Management](https://docs.opensearch.org/latest/im-plugin/ism/index/).
- [Dashboards saved objects](https://docs.opensearch.org/latest/dashboards/management/saved-objects/).
## Optional Metric Analytics
- [OpenSearch Metric Analytics](https://docs.opensearch.org/latest/observing-your-data/prometheusmetrics/) — Prometheus SQL connector API, OTel-specific index schema and UI workflow.
- [SQL data-source API](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.md) — encryption master key and data-source registration.
- [Pinned Kafka consumer](https://github.com/influxdata/telegraf/blob/v1.40.0/plugins/inputs/kafka_consumer/README.md).
- [Pinned JSON v2 parser](https://github.com/influxdata/telegraf/blob/v1.40.0/plugins/parsers/json_v2/README.md).
- [Pinned Prometheus exporter](https://github.com/influxdata/telegraf/blob/v1.40.0/plugins/outputs/prometheus_client/README.md).

41
docs/VALIDATION.md Normal file
View File

@ -0,0 +1,41 @@
# Validation record
Date: 2026-09-17. Development environment: macOS arm64; no Docker daemon, no Debian VM connection supplied.
## Completed locally
- Confirmed stable release metadata and published Docker image tags. Kafka release candidates were excluded.
- Compared Kafka JSON serialization, host input options, JSON v2 parsing, Prometheus output, Starlark metric state and Data Prepper configuration against upstream documentation/source. Used `inputs.system.include` to emit a single numeric system measurement; removed an obsolete network option and used a tag filter for protocol totals. Corrected the process-state field to the documented singular `zombie`.
- Validated the complete Compose model, including the optional profile, with the official standalone **Docker Compose v5.5.1** CLI using a temporary test encryption key. This runs `config --quiet` and does not require/start Docker.
- Parsed all mounted YAML configurations with Ruby's YAML parser; parsed both Telegraf configurations with Python's TOML parser.
- Checked nine shell scripts with `bash -n`.
- Ran four Python test cases covering rate isolation/reset/gap handling, disk busy/load calculations, the saved-object reference graph and non-overlapping dashboard layouts, and the metric configuration/mapping contract.
- Generated 48 saved objects: 4 dashboards, 43 visualizations/navigation panels and one index pattern. NDJSON and pretty JSON representations are equivalent.
The rate test executes the deliberately Python-compatible subset of the Starlark script in Python. It checks mathematics/state behavior; it is **not** validation by the real Starlark interpreter. An official native Telegraf binary download for local runtime validation returned HTTP 403, so that runtime check was not completed.
## Not completed in this environment
- Image startup and full Kafka/Data Prepper/OpenSearch delivery.
- Debian APT installation, systemd sandbox behavior and collection under the real `telegraf` account.
- Real Telegraf configuration parsing, Starlark execution and optional JSON-to-Prometheus conversion.
- Import into a running OpenSearch Dashboards instance and visual browser inspection.
- SQL/Prometheus connector execution against the running pinned stack.
- Throughput, outage durability, recovery and memory/disk capacity benchmarks.
## Run on Debian
```bash
sudo ./scripts/prepare-host.sh
./scripts/start.sh
sudo ./scripts/install-agent.sh
./scripts/smoke-test.sh
./scripts/import-dashboards.sh
./scripts/enable-metric-analytics.sh
# After about one minute:
python3 ./scripts/check-metric-analytics.py
```
The installer performs a real Telegraf `--test` as the service user before starting its dedicated service. The ingestion smoke test requires recent CPU-total, memory, filesystem and disk/network rate data, identity tags, timestamps and expected index field types. Dashboard import checks the API's `success` flag, not just its HTTP status. The optional checker requires both Prometheus samples and a successful federated PPL query.
Finally inspect all four dashboards in the browser and perform the manual acceptance exercise in [OPERATIONS.md](OPERATIONS.md). A complete POC implementation is supplied, but deployment/runtime compatibility is not represented as already proven.

60
scripts/check-ingestion.py Normal file
View File

@ -0,0 +1,60 @@
#!/usr/bin/env python3
"""Assert end-to-end native host ingestion; bounded wait, nonzero exit on failure."""
import json
import os
import sys
import time
import urllib.error
import urllib.request
BASE = os.environ.get('OPENSEARCH_URL', 'http://127.0.0.1:9200')
REQUIRED = {
'cpu': 'usage_active', 'mem': 'used_percent', 'disk': 'used_percent',
'diskio': 'read_bytes_per_sec', 'net': 'bytes_recv_per_sec',
}
def request(path, payload=None):
r = urllib.request.Request(BASE + path, data=None if payload is None else json.dumps(payload).encode(),
headers={'Content-Type': 'application/json'})
with urllib.request.urlopen(r, timeout=20) as response:
return json.load(response)
failures = []
for attempt in range(36):
failures = []
for name, field in REQUIRED.items():
filters = [{'term': {'name': name}}, {'range': {'@timestamp': {'gte': 'now-2m'}}},
{'exists': {'field': 'fields.' + field}}]
if os.environ.get('METRICS_HOST'):
filters.append({'term': {'tags.host': os.environ['METRICS_HOST']}})
if name == 'cpu':
filters.append({'term': {'tags.cpu': 'cpu-total'}})
try:
data = request('/host-metrics-v1-*/_search', {
'size': 1, 'sort': [{'@timestamp': 'desc'}], 'query': {'bool': {'filter': filters}}})
hits = data['hits']['hits']
if not hits:
failures.append(f'{name}: no fresh {field}'); continue
doc = hits[0]['_source']
assert isinstance(doc['fields'][field], (int, float))
assert all(doc['tags'].get(k) for k in ('host', 'environment', 'role', 'series'))
assert doc['ingested_at'] and doc['@timestamp']
if field in ('usage_active', 'used_percent'):
assert 0 <= doc['fields'][field] <= 100.01
else:
assert doc['fields'][field] >= 0
except (urllib.error.URLError, KeyError, AssertionError) as e:
failures.append(f'{name}: {e}')
if not failures:
break
if attempt == 0 or attempt % 6 == 0:
print('Waiting:', '; '.join(failures), flush=True)
time.sleep(5)
else:
sys.exit('FAIL: ' + '; '.join(failures) + '\nCheck journalctl -u telegraf-host-metrics and docker compose logs data-prepper.')
caps = request('/host-metrics-v1-*/_field_caps?fields=@timestamp,tags.host,fields.used_percent')
assert 'date' in caps['fields']['@timestamp'], caps
assert 'keyword' in caps['fields']['tags.host'], caps
assert 'double' in caps['fields']['fields.used_percent'], caps
print('PASS: recent native host samples, CPU-total, derived rates, identity tags, timestamps and numeric mappings.')

21
scripts/check-metric-analytics.py Normal file
View File

@ -0,0 +1,21 @@
#!/usr/bin/env python3
"""Verify optional Prometheus scrape and federated OpenSearch PPL query."""
import json
import sys
import urllib.parse
import urllib.request
query = 'cpu_usage_active{cpu="cpu-total"}'
with urllib.request.urlopen('http://127.0.0.1:9090/api/v1/query?' + urllib.parse.urlencode({'query': query}), timeout=20) as r:
result = json.load(r)
if result.get('status') != 'success' or not result['data']['result']:
sys.exit('No Prometheus CPU samples yet. Wait 60 seconds and check metrics-bridge logs.')
print('PASS: Prometheus has host CPU samples.')
payload = {'query': 'source = host_prometheus.cpu_usage_active | head 5'}
req = urllib.request.Request('http://127.0.0.1:9200/_plugins/_ppl', data=json.dumps(payload).encode(),
headers={'Content-Type': 'application/json'})
with urllib.request.urlopen(req, timeout=60) as r:
result = json.load(r)
if not result.get('datarows'):
sys.exit('PPL returned no rows: ' + json.dumps(result))
print('PASS: OpenSearch queries the Prometheus data source through PPL.')

28
scripts/configure-access.sh Executable file
View File

@ -0,0 +1,28 @@
#!/usr/bin/env bash
set -euo pipefail
cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.."
[[ $# == 2 ]] || { echo "Usage: $0 VM_LAN_IP_OR_DNS_NAME WORKSTATION_IP" >&2; exit 1; }
python3 - "$1" "$2" <<'PY'
from pathlib import Path
import ipaddress, re, socket, sys
host = sys.argv[1]
if not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9.-]*', host) or host in ('0.0.0.0', 'localhost', '127.0.0.1'):
sys.exit('Use the VM LAN IPv4 address or a DNS name reachable from both the VM and workstation, without http:// or a port.')
bind = ipaddress.IPv4Address(socket.gethostbyname(host))
workstation = ipaddress.IPv4Address(sys.argv[2])
private_networks = [ipaddress.ip_network(x) for x in ['10.0.0.0/8','172.16.0.0/12','192.168.0.0/16']]
if not any(bind in n for n in private_networks) or not any(workstation in n for n in private_networks):
sys.exit('This helper requires private LAN IPv4 addresses for the VM and workstation.')
p = Path('.env')
old = p.read_text() if p.exists() else Path('.env.example').read_text()
updates = {'BIND_ADDRESS': str(bind), 'WORKSTATION_IP': str(workstation), 'KAFKA_ADVERTISED_HOST': host}
lines = [line for line in old.splitlines() if line.split('=', 1)[0].strip() not in updates]
lines.extend(f'{key}={value}' for key, value in updates.items())
p.write_text('\n'.join(lines) + '\n')
p.chmod(0o600)
print(f'Configured direct access. Dashboards: http://{host}:5601 ; OpenSearch: http://{host}:9200')
print(f'Kafka advertises {host}:9092 to external clients; Docker clients continue to use kafka:29092.')
print(f'LAN binding: {bind}; allowed workstation: {workstation}. Loopback access is retained.')
print('Install the source restriction first: sudo ./scripts/install-access-rules.sh')
print('Then apply with ./scripts/start.sh ; optional services: ./scripts/enable-metric-analytics.sh')
PY

67
scripts/diagnose.sh Executable file
View File

@ -0,0 +1,67 @@
#!/usr/bin/env bash
# Read-only diagnostics, except --probe explicitly sends one synthetic Kafka record.
set -uo pipefail
cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.."
[[ $# == 0 || ($# == 1 && $1 == --probe) ]] || { echo "Usage: $0 [--probe]" >&2; exit 1; }
mkdir -p diagnostics
report="diagnostics/report-$(date -u +%Y%m%dT%H%M%SZ).txt"
exec > >(tee "$report") 2>&1
section() { printf '\n### %s\n' "$1"; }
section 'Host time and native service'
date -u
systemctl --no-pager --full status telegraf-host-metrics.service
journalctl -u telegraf-host-metrics -n 100 --no-pager
section 'Native agent binary'
telegraf --version
section 'Native collection test (does not export to Kafka)'
# Parse only the four non-secret settings created by the installer.
if [[ $EUID == 0 && -f /etc/telegraf/host-metrics/environment ]]; then
python3 - <<'PY'
import os, shlex, subprocess
from pathlib import Path
env = os.environ.copy()
allowed = {'METRICS_HOST','METRICS_ENVIRONMENT','METRICS_ROLE','KAFKA_BOOTSTRAP'}
for line in Path('/etc/telegraf/host-metrics/environment').read_text().splitlines():
key, sep, raw = line.partition('=')
if sep and key in allowed:
values = shlex.split(raw)
if len(values) == 1:
env[key] = values[0]
print('Agent host:', env.get('METRICS_HOST'), 'Kafka bootstrap:', env.get('KAFKA_BOOTSTRAP'), flush=True)
try:
result = subprocess.run(['runuser','-u','telegraf','--','/usr/bin/telegraf','--config',
'/etc/telegraf/host-metrics/telegraf.conf','--test','--test-wait','3'],
env=env, timeout=35)
print('Native collection test exit:', result.returncode)
except subprocess.TimeoutExpired:
print('Native collection test timed out.')
PY
else
echo 'Run with sudo to include the native collection test and complete journal.'
fi
section 'Compose container states'
docker compose ps -a
section 'Initializer and ingestion logs'
docker compose logs --no-color --tail=100 kafka-init opensearch-init data-prepper
section 'Kafka broker logs'
docker compose logs --no-color --tail=50 kafka
section 'Kafka topic and latest offsets'
docker compose exec -T kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:29092 --describe --topic host-metrics-v1
docker compose exec -T kafka /opt/kafka/bin/kafka-get-offsets.sh --bootstrap-server kafka:29092 --topic host-metrics-v1 --time -1
section 'Data Prepper consumer group'
docker compose exec -T kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server kafka:29092 --describe --group opensearch-host-metrics-v1
section 'OpenSearch health, indices and newest document'
curl --max-time 10 -fsS 'http://127.0.0.1:9200/_cluster/health?pretty'
curl --max-time 10 -fsS 'http://127.0.0.1:9200/_cat/indices/host-metrics-v1-*?v'
curl --max-time 10 -fsS 'http://127.0.0.1:9200/host-metrics-v1-*/_search?size=1&sort=@timestamp:desc&pretty'
section 'Data Prepper DLQ'
docker compose exec -T data-prepper sh -c 'ls -lah /usr/share/data-prepper/dlq; if [ -f /usr/share/data-prepper/dlq/failed-events.json ]; then tail -n 5 /usr/share/data-prepper/dlq/failed-events.json; fi'
if [[ ${1:-} == --probe ]]; then
section 'End-to-end Kafka probe'
python3 scripts/probe-pipeline.py
probe_status=$?
printf 'Probe exit status: %s\n' "$probe_status"
fi
section 'Report saved'
echo "$report"
echo 'Review the report before sharing. It includes host labels, metrics and service logs; it does not read .env or print the datasource encryption key.'

10
scripts/enable-metric-analytics.sh Executable file
View File

@ -0,0 +1,10 @@
#!/usr/bin/env bash
set -euo pipefail
cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.."
docker compose --profile metrics-analytics up -d metrics-bridge prometheus
for ((i=0; i<60; i++)); do
if curl -fsS http://127.0.0.1:9090/-/ready >/dev/null; then break; fi
sleep 2
done
curl -fsS http://127.0.0.1:9090/-/ready
python3 scripts/register-prometheus.py

224
scripts/generate-assets.py Normal file
View File

@ -0,0 +1,224 @@
#!/usr/bin/env python3
"""Generate versioned index mappings and portable OpenSearch saved objects."""
import json
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
INDEX = 'hm-index-v1'
OBJECTS = []
FIELDS = {'@timestamp': 'date', 'ingested_at': 'date', 'timestamp': 'number', 'name': 'string'}
TAGS = ['host', 'environment', 'role', 'series', 'cpu', 'path', 'device', 'fstype', 'mode', 'name', 'interface']
FIELDS.update({'tags.' + t: 'string' for t in TAGS})
FORMATS = {}
def dumps(value):
return json.dumps(value, separators=(',', ':'))
def write_json(path, value):
(ROOT / path).write_text(json.dumps(value, indent=2) + '\n')
def agg(field, label=None, op='avg'):
FIELDS.setdefault(field, 'number')
if any(s in field for s in ['bytes', 'fields.available', 'fields.used', 'fields.free', 'fields.total', 'fields.cached', 'fields.buffered']):
if 'percent' not in field:
FORMATS[field] = {'id': 'bytes', 'params': {'pattern': '0.0 b'}}
return {'type': op, 'schema': 'metric', 'params': {'field': field, 'customLabel': label or field.split('.')[-1]}}
def vis(key, title, query, metrics, kind='line', split='tags.series', description=''):
aggs = [dict(a, id=str(i + 1), enabled=True) for i, a in enumerate(metrics)]
if kind == 'line':
aggs.append({'id': 'time', 'enabled': True, 'type': 'date_histogram', 'schema': 'segment',
'params': {'field': '@timestamp', 'interval': 'auto', 'min_doc_count': 1, 'extended_bounds': {}}})
if split and kind in ('line', 'table'):
aggs.append({'id': 'series', 'enabled': True, 'type': 'terms',
'schema': 'group' if kind == 'line' else 'bucket',
'params': {'field': split, 'size': 30, 'order': 'desc', 'orderBy': '_count',
'otherBucket': False, 'missingBucket': False}})
if kind == 'metric':
params = {'addTooltip': True, 'addLegend': False, 'type': 'metric',
'metric': {'percentageMode': False, 'useRanges': False, 'colorSchema': 'Green to Red',
'metricColorMode': 'None', 'colorsRange': [{'from': 0, 'to': 100}],
'labels': {'show': True}, 'invertColors': False,
'style': {'bgFill': '#000', 'bgColor': False, 'labelColor': False, 'subText': '', 'fontSize': 42}}}
elif kind == 'table':
params = {'perPage': 15, 'showPartialRows': False, 'showMetricsAtAllLevels': False,
'sort': {'columnIndex': None, 'direction': None}, 'showTotal': False, 'totalFunc': 'sum'}
else:
params = {'type': 'line', 'addTooltip': True, 'addLegend': True, 'legendPosition': 'bottom',
'grid': {'categoryLines': False}, 'times': [], 'addTimeMarker': False,
'categoryAxes': [{'id': 'CategoryAxis-1', 'type': 'category', 'position': 'bottom',
'show': True, 'style': {}, 'scale': {'type': 'linear'},
'labels': {'show': True, 'truncate': 100}, 'title': {}}],
'valueAxes': [{'id': 'ValueAxis-1', 'type': 'value', 'position': 'left', 'show': True,
'style': {}, 'scale': {'type': 'linear', 'mode': 'normal'},
'labels': {'show': True}, 'title': {'text': ''}}],
'seriesParams': [{'show': True, 'type': 'line', 'mode': 'normal', 'data': {'id': a['id'], 'label': a['params'].get('customLabel', '')},
'valueAxis': 'ValueAxis-1', 'drawLinesBetweenPoints': True,
'showCircles': False, 'lineWidth': 2, 'interpolate': 'linear'} for a in aggs if a['schema'] == 'metric']}
OBJECTS.append({'type': 'visualization', 'id': key, 'attributes': {
'title': title, 'description': description, 'version': 1,
'visState': dumps({'title': title, 'type': kind, 'params': params, 'aggs': aggs}),
'uiStateJSON': '{}',
'kibanaSavedObjectMeta': {'searchSourceJSON': dumps({'query': {'language': 'lucene', 'query': query},
'filter': [], 'indexRefName': 'kibanaSavedObjectMeta.searchSourceJSON.index'})}},
'references': [{'name': 'kibanaSavedObjectMeta.searchSourceJSON.index', 'type': 'index-pattern', 'id': INDEX}]})
return key
def markdown(key, title, body):
OBJECTS.append({'type': 'visualization', 'id': key, 'attributes': {
'title': title, 'description': '', 'version': 1, 'uiStateJSON': '{}',
'visState': dumps({'title': title, 'type': 'markdown', 'params': {'markdown': body, 'openLinksInNewTab': False}, 'aggs': []}),
'kibanaSavedObjectMeta': {'searchSourceJSON': dumps({'query': {'query': '', 'language': 'lucene'}, 'filter': []})}}, 'references': []})
return key
def dashboard(key, title, subtitle, cards, charts):
nav = ' • '.join(f'[{label}](/app/dashboards#/view/hm-{target})' for target, label in
[('fleet', 'Fleet'), ('compute', 'CPU & memory'), ('storage', 'Storage'), ('network', 'Network')])
banner = markdown(key + '-intro', title + ' / guide', f'# {title}\n{subtitle}\n\n{nav}\n\n'
'**Scope:** use **Add filter → tags.host** to isolate a server; `tags.environment` and `tags.role` select a fleet. '
'Cards summarize the selected time range. Charts show the top 30 series; narrow the filter for larger fleets. '
'Empty charts indicate missing data, never zero. Times are UTC.')
placements = [(banner, 0, 0, 48, 8)]
for i, card in enumerate(cards):
placements.append((card, i * 12, 8, 12, 8))
for i, chart in enumerate(charts):
placements.append((chart, (i % 2) * 24, 16 + (i // 2) * 15, 24, 15))
panels, refs = [], []
for i, (obj, x, y, w, h) in enumerate(placements):
ref = f'panel_{i}'
panels.append({'version': '3.8.0', 'type': 'visualization', 'panelIndex': str(i + 1), 'panelRefName': ref,
'embeddableConfig': {}, 'gridData': {'x': x, 'y': y, 'w': w, 'h': h, 'i': str(i + 1)}})
refs.append({'name': ref, 'type': 'visualization', 'id': obj})
OBJECTS.append({'type': 'dashboard', 'id': key, 'attributes': {
'title': 'Host Metrics / ' + title, 'description': subtitle, 'version': 1,
'panelsJSON': dumps(panels), 'optionsJSON': dumps({'useMargins': True, 'hidePanelTitles': False}),
'timeRestore': True, 'timeFrom': 'now-1h', 'timeTo': 'now', 'refreshInterval': {'pause': False, 'value': 15000},
'kibanaSavedObjectMeta': {'searchSourceJSON': dumps({'query': {'language': 'lucene', 'query': ''}, 'filter': []})}}, 'references': refs})
def m(key, title, query, field, op='avg'):
return vis('hm-' + key, title, query, [agg(field, title, op)], 'metric', None)
def line(key, title, query, *fields, split='tags.series'):
return vis('hm-' + key, title, query, [agg(f, label) for f, label in fields], split=split)
CPU = 'name:cpu AND tags.cpu:"cpu-total"'
active = vis('hm-hosts', 'Hosts seen in selected range', 'name:system', [agg('tags.host', 'Hosts', 'cardinality')], 'metric', None)
cpu_peak = m('cpu-peak', 'Peak CPU busy (%)', CPU, 'fields.usage_active', 'max')
ram_peak = m('ram-peak', 'Peak RAM used (%)', 'name:mem', 'fields.used_percent', 'max')
disk_peak = m('disk-peak', 'Peak filesystem used (%)', 'name:disk', 'fields.used_percent', 'max')
cpu = line('cpu', 'CPU busy (%) · by host', CPU, ('fields.usage_active', 'Busy %'), split='tags.host')
ram = line('ram', 'RAM used (%) · by host', 'name:mem', ('fields.used_percent', 'Used %'), split='tags.host')
load = line('load-normalized', 'Load / logical CPU · 1.0 means one task per CPU', 'name:system', ('fields.load1_per_cpu', 'Load / CPU'), split='tags.host')
fs = line('fs-used', 'Filesystem used (%) · by mount', 'name:disk', ('fields.used_percent', 'Used %'))
fresh = vis('hm-freshness', 'Latest sample per host · compare with current UTC time', 'name:system',
[dict(type='top_hits', schema='metric', params={'field': '@timestamp', 'aggregate': 'concat', 'size': 1,
'sortField': '@timestamp', 'sortOrder': 'desc', 'customLabel': 'Latest sample (UTC)'})], 'table', 'tags.host')
uptime = vis('hm-uptime', 'Host inventory · range statistics', 'name:system',
[agg('fields.uptime', 'Max uptime (s)', 'max'), agg('fields.n_cpus', 'Logical CPUs', 'max'), agg('fields.load1', 'Peak load', 'max')], 'table', 'tags.host')
dashboard('hm-fleet', 'Fleet overview', 'Health, capacity and collection freshness across your servers.',
[active, cpu_peak, ram_peak, disk_peak], [cpu, ram, load, fs, fresh, uptime])
dashboard('hm-compute', 'CPU & memory', 'CPU saturation, scheduler pressure, memory headroom and process health.',
[cpu_peak, ram_peak, m('iowait-peak', 'Peak I/O wait (%)', CPU, 'fields.usage_iowait', 'max'),
m('swap-peak', 'Peak swap used (%)', 'name:swap', 'fields.used_percent', 'max')],
[cpu, line('cpu-modes', 'CPU modes (%) · user / system / I/O wait / steal', CPU,
('fields.usage_user', 'User'), ('fields.usage_system', 'System'), ('fields.usage_iowait', 'I/O wait'), ('fields.usage_steal', 'Steal'), split='tags.host'),
line('cpu-core', 'Per-core CPU busy (%)', 'name:cpu AND NOT tags.cpu:"cpu-total"', ('fields.usage_active', 'Busy %')),
line('load', 'Load average · 1 / 5 / 15 minutes', 'name:system', ('fields.load1', '1m'), ('fields.load5', '5m'), ('fields.load15', '15m'), split='tags.host'),
ram, line('memory', 'RAM headroom · available / cached / buffered', 'name:mem', ('fields.available', 'Available'), ('fields.cached', 'Cached'), ('fields.buffered', 'Buffered'), split='tags.host'),
line('swap-io', 'Swap activity · bytes/s', 'name:swap', ('fields.in_per_sec', 'In'), ('fields.out_per_sec', 'Out'), split='tags.host'),
line('processes', 'Processes · running / blocked / zombie', 'name:processes', ('fields.running', 'Running'), ('fields.blocked', 'Blocked'), ('fields.zombie', 'Zombie'), split='tags.host'),
line('context', 'Context switches / second', 'name:kernel', ('fields.context_switches_per_sec', 'Switches/s'), split='tags.host'),
line('interrupts', 'Interrupts / second', 'name:kernel', ('fields.interrupts_per_sec', 'Interrupts/s'), split='tags.host')])
dashboard('hm-storage', 'Storage', 'Filesystem and inode capacity, per-device throughput, operations and busy time.',
[disk_peak, m('inode-peak', 'Peak inode use (%)', 'name:disk', 'fields.inodes_used_percent', 'max'),
m('disk-busy-peak', 'Peak device busy (%)', 'name:diskio', 'fields.io_busy_percent', 'max'),
m('disk-free-min', 'Minimum mount free bytes', 'name:disk', 'fields.free', 'min')],
[fs, line('fs-free', 'Free bytes · by mount', 'name:disk', ('fields.free', 'Free bytes')),
line('inodes', 'Inode use (%) · by mount', 'name:disk', ('fields.inodes_used_percent', 'Inodes used %')),
vis('hm-fs-table', 'Mount capacity · extrema over selected range', 'name:disk',
[agg('fields.total', 'Capacity', 'max'), agg('fields.free', 'Minimum free', 'min'), agg('fields.used_percent', 'Peak used %', 'max'), agg('fields.inodes_used_percent', 'Peak inode %', 'max')], 'table'),
line('disk-throughput', 'Disk throughput · bytes/s', 'name:diskio', ('fields.read_bytes_per_sec', 'Read'), ('fields.write_bytes_per_sec', 'Write')),
line('disk-iops', 'Disk operations / second', 'name:diskio', ('fields.reads_per_sec', 'Read IOPS'), ('fields.writes_per_sec', 'Write IOPS')),
line('disk-busy', 'Device busy (%) · not an NVMe saturation measurement', 'name:diskio', ('fields.io_busy_percent', 'Busy %')),
line('disk-queue', 'I/O requests in progress', 'name:diskio', ('fields.iops_in_progress', 'In progress'))])
dashboard('hm-network', 'Network', 'Per-interface throughput, packet rates, errors and drops. Loopback and Docker bridges are excluded.',
[m('rx-peak', 'Peak interface RX bytes/s', 'name:net', 'fields.bytes_recv_per_sec', 'max'),
m('tx-peak', 'Peak interface TX bytes/s', 'name:net', 'fields.bytes_sent_per_sec', 'max'),
m('rx-errors', 'Peak RX errors/s', 'name:net', 'fields.err_in_per_sec', 'max'),
m('rx-drops', 'Peak RX drops/s', 'name:net', 'fields.drop_in_per_sec', 'max')],
[line('network-bytes', 'Network throughput · bytes/s', 'name:net', ('fields.bytes_recv_per_sec', 'RX'), ('fields.bytes_sent_per_sec', 'TX')),
line('network-packets', 'Packets / second', 'name:net', ('fields.packets_recv_per_sec', 'RX'), ('fields.packets_sent_per_sec', 'TX')),
line('network-errors', 'Errors / second', 'name:net', ('fields.err_in_per_sec', 'RX errors'), ('fields.err_out_per_sec', 'TX errors')),
line('network-drops', 'Drops / second', 'name:net', ('fields.drop_in_per_sec', 'RX drops'), ('fields.drop_out_per_sec', 'TX drops')),
vis('hm-net-table', 'Interface inventory · peaks over selected range', 'name:net',
[agg('fields.bytes_recv_per_sec', 'Peak RX', 'max'), agg('fields.bytes_sent_per_sec', 'Peak TX', 'max'), agg('fields.err_in_per_sec', 'Peak errors/s', 'max')], 'table'),
fresh])
field_list = [{'name': k, 'type': v, 'count': 0, 'scripted': False, 'searchable': True,
'aggregatable': True, 'readFromDocValues': True} for k, v in sorted(FIELDS.items())]
OBJECTS.insert(0, {'type': 'index-pattern', 'id': INDEX, 'attributes': {
'title': 'host-metrics-v1-*', 'timeFieldName': '@timestamp', 'fields': dumps(field_list),
'fieldFormatMap': dumps(FORMATS)}, 'references': []})
(ROOT / 'dashboards/host-metrics.ndjson').write_text('\n'.join(dumps(o) for o in OBJECTS) + '\n')
write_json('dashboards/host-metrics.json', OBJECTS)
properties = {'@timestamp': {'type': 'date'}, 'ingested_at': {'type': 'date'}, 'timestamp': {'type': 'long'},
'name': {'type': 'keyword'}, 'tags': {'type': 'object', 'dynamic': True},
'fields': {'type': 'object', 'dynamic': True,
'properties': {k.removeprefix('fields.'): {'type': 'double'} for k in FIELDS if k.startswith('fields.')}}}
write_json('configs/opensearch/index-template.json', {
'index_patterns': ['host-metrics-v1-*'], 'priority': 200, 'version': 1,
'template': {'settings': {'number_of_shards': 1, 'number_of_replicas': 0, 'refresh_interval': '5s',
'index.mapping.total_fields.limit': 500},
'mappings': {'dynamic': True, 'properties': properties, 'dynamic_templates': [
{'tags': {'path_match': 'tags.*', 'mapping': {'type': 'keyword', 'ignore_above': 1024}}},
{'numeric_fields': {'path_match': 'fields.*', 'mapping': {'type': 'double'}}}]}}})
write_json('configs/opensearch/retention-policy.json', {'policy': {
'description': 'Delete host metric daily indices after 14 days; applies only to host-metrics-v1-*.',
'default_state': 'hot', 'states': [
{'name': 'hot', 'actions': [], 'transitions': [{'state_name': 'delete', 'conditions': {'min_index_age': '14d'}}]},
{'name': 'delete', 'actions': [{'delete': {}}], 'transitions': []}],
'ism_template': [{'index_patterns': ['host-metrics-v1-*'], 'priority': 200}]}})
bridge = '''# Kafka-to-Prometheus adapter only; no local host inputs.
[agent]
flush_interval = "5s"
metric_batch_size = 500
metric_buffer_limit = 10000
omit_hostname = true
[[inputs.kafka_consumer]]
brokers = ["kafka:29092"]
topics = ["host-metrics-v1"]
consumer_group = "prometheus-host-metrics-v1"
kafka_version = "3.6.0"
offset = "newest"
max_undelivered_messages = 1000
data_format = "json_v2"
[[inputs.kafka_consumer.json_v2]]
measurement_name_path = "name"
timestamp_path = "timestamp"
timestamp_format = "unix_ms"
timestamp_timezone = "UTC"
[[inputs.kafka_consumer.json_v2.object]]
path = "fields"
'''
for tag in TAGS:
bridge += f''' [[inputs.kafka_consumer.json_v2.tag]]
path = "tags.{tag}"
rename = "{tag}"
optional = true
'''
bridge += '''
[[outputs.prometheus_client]]
listen = ":9273"
metric_version = 1
expiration_interval = "60s"
string_as_label = false
export_timestamp = true
collectors_exclude = ["gocollector", "process"]
'''
(ROOT / 'configs/metrics-bridge/telegraf.conf').write_text(bridge)
print(f'Generated {len(OBJECTS)} saved objects, 4 dashboards, mappings, retention policy and metrics bridge.')

14
scripts/import-dashboards.sh Executable file
View File

@ -0,0 +1,14 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)
response=$(mktemp)
trap 'rm -f "$response"' EXIT
curl -fsS -X POST "${DASHBOARDS_URL:-http://127.0.0.1:5601}/api/saved_objects/_import?overwrite=true" \
-H 'osd-xsrf: true' --form "file=@$ROOT/dashboards/host-metrics.ndjson" > "$response"
python3 - "$response" <<'PY'
import json,sys
r=json.load(open(sys.argv[1]))
if not r.get('success'):
print(json.dumps(r,indent=2));sys.exit(1)
print(f"Imported {r['successCount']} saved objects. Open Dashboards → Host Metrics / Fleet overview.")
PY

29
scripts/install-access-rules.sh Executable file
View File

@ -0,0 +1,29 @@
#!/usr/bin/env bash
set -euo pipefail
[[ $EUID == 0 ]] || { echo 'Run with sudo.' >&2; exit 1; }
cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.."
command -v iptables >/dev/null
python3 - <<'PY'
import ipaddress,json,subprocess
from pathlib import Path
values = {}
for line in Path('.env').read_text().splitlines():
key, sep, value = line.partition('=')
if sep and key in ('BIND_ADDRESS','WORKSTATION_IP'):
values[key] = str(ipaddress.IPv4Address(value))
lan, workstation = values['BIND_ADDRESS'], values['WORKSTATION_IP']
interfaces = json.loads(subprocess.check_output(['ip','-j','-4','address','show']))
if lan not in [a.get('local') for i in interfaces for a in i.get('addr_info',[])]:
raise SystemExit('BIND_ADDRESS is not assigned to this VM.')
if ipaddress.ip_address(lan).is_unspecified or ipaddress.ip_address(lan).is_loopback:
raise SystemExit('Use the actual VM LAN address.')
p=Path('/etc/host-metrics-access.conf')
p.write_text(f'VM_LAN_IP={lan}\nWORKSTATION_IP={workstation}\n');p.chmod(0o600)
print(f'Restricting the metrics service ports on {lan} to workstation {workstation}.')
PY
install -m 0755 configs/network/access-rules.sh /usr/local/sbin/host-metrics-access
install -m 0644 configs/network/host-metrics-access.service /etc/systemd/system/host-metrics-access.service
systemctl daemon-reload
systemctl enable host-metrics-access.service
systemctl restart host-metrics-access.service
iptables -w 5 -S HOST-METRICS-ACCESS

58
scripts/install-agent.sh Executable file
View File

@ -0,0 +1,58 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)
[[ $EUID -eq 0 ]] || { echo 'Run with sudo.' >&2; exit 1; }
[[ -f /etc/debian_version ]] || { echo 'This installer targets Debian.' >&2; exit 1; }
METRICS_HOST=${METRICS_HOST:-$(hostname -f 2>/dev/null || hostname)}
METRICS_ENVIRONMENT=${METRICS_ENVIRONMENT:-lab}
METRICS_ROLE=${METRICS_ROLE:-docker-host}
KAFKA_BOOTSTRAP=${KAFKA_BOOTSTRAP:-127.0.0.1:9092}
TELEGRAF_PACKAGE_VERSION=${TELEGRAF_PACKAGE_VERSION:-1.40.0-1}
# These values are interpolated into TOML through a systemd EnvironmentFile.
for value in "$METRICS_HOST" "$METRICS_ENVIRONMENT" "$METRICS_ROLE" "$KAFKA_BOOTSTRAP"; do
[[ $value =~ ^[a-zA-Z0-9_.:/-]+$ ]] || { echo "Invalid label/endpoint: $value" >&2; exit 1; }
done
apt-get update
apt-get install -y ca-certificates curl gnupg
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
curl --fail --silent --show-error --location --retry 3 \
https://repos.influxdata.com/influxdata-archive.key -o "$tmp/influxdata.key"
gpg --batch --show-keys --with-colons "$tmp/influxdata.key" > "$tmp/key-info"
awk -F: '$1 == "fpr" {print $10}' "$tmp/key-info" | \
grep -qx '24C975CBA61A024EE1B631787C3D57159FC2F927'
install -d -m 0755 /etc/apt/keyrings
gpg --batch --yes --dearmor --output "$tmp/influxdata.gpg" "$tmp/influxdata.key"
if ! grep -Rqs 'repos.influxdata.com' /etc/apt/sources.list /etc/apt/sources.list.d 2>/dev/null; then
install -m 0644 "$tmp/influxdata.gpg" /etc/apt/keyrings/influxdata-host-metrics.gpg
printf '%s\n' 'deb [signed-by=/etc/apt/keyrings/influxdata-host-metrics.gpg] https://repos.influxdata.com/debian stable main' \
> /etc/apt/sources.list.d/influxdata-host-metrics.list
else
echo 'Using the existing InfluxData APT repository and its configured signing key.'
fi
apt-get update
had_package=false
if dpkg-query -W -f='${Status}' telegraf 2>/dev/null | grep -qx 'install ok installed'; then had_package=true; fi
apt-get install -y "telegraf=$TELEGRAF_PACKAGE_VERSION"
# On a fresh install, use our dedicated unit instead of the package default unit.
if [[ $had_package == false ]]; then systemctl disable --now telegraf.service || true; fi
install -d -m 0755 /etc/telegraf/host-metrics
if [[ -f /etc/telegraf/host-metrics/telegraf.conf ]]; then
cp -a /etc/telegraf/host-metrics "/etc/telegraf/host-metrics.backup.$(date +%s)"
fi
install -m 0644 "$ROOT/agent/telegraf.conf" /etc/telegraf/host-metrics/telegraf.conf
install -m 0644 "$ROOT/agent/enrich.star" /etc/telegraf/host-metrics/enrich.star
printf 'METRICS_HOST=%s\nMETRICS_ENVIRONMENT=%s\nMETRICS_ROLE=%s\nKAFKA_BOOTSTRAP=%s\n' \
"$METRICS_HOST" "$METRICS_ENVIRONMENT" "$METRICS_ROLE" "$KAFKA_BOOTSTRAP" \
> /etc/telegraf/host-metrics/environment
chmod 0644 /etc/telegraf/host-metrics/environment
install -m 0644 "$ROOT/agent/telegraf-host-metrics.service" /etc/systemd/system/telegraf-host-metrics.service
# Validate configuration and gathering as the actual service user.
runuser -u telegraf -- env METRICS_HOST="$METRICS_HOST" METRICS_ENVIRONMENT="$METRICS_ENVIRONMENT" \
METRICS_ROLE="$METRICS_ROLE" KAFKA_BOOTSTRAP="$KAFKA_BOOTSTRAP" \
telegraf --config /etc/telegraf/host-metrics/telegraf.conf --test --test-wait 3
systemctl daemon-reload
systemctl enable telegraf-host-metrics.service
systemctl restart telegraf-host-metrics.service
systemctl --no-pager --full status telegraf-host-metrics.service
echo 'Agent installed. Check ingestion with: ./scripts/smoke-test.sh'

15
scripts/prepare-host.sh Executable file
View File

@ -0,0 +1,15 @@
#!/usr/bin/env bash
set -euo pipefail
[[ $EUID -eq 0 ]] || { echo 'Run with sudo.' >&2; exit 1; }
[[ -f /etc/debian_version ]] || { echo 'This script targets Debian.' >&2; exit 1; }
apt-get update
apt-get install -y ca-certificates curl gnupg python3
command -v docker >/dev/null || { echo 'Install Docker Engine and the Compose plugin first: https://docs.docker.com/engine/install/debian/' >&2; exit 1; }
docker compose version
current=$(sysctl -n vm.max_map_count)
if (( current < 262144 )); then
printf 'vm.max_map_count=262144\n' > /etc/sysctl.d/90-host-metrics.conf
sysctl -p /etc/sysctl.d/90-host-metrics.conf
fi
docker info >/dev/null
echo 'Host ready. Recommended: 4 vCPUs, 12 GiB RAM, 60+ GiB free SSD space.'

63
scripts/probe-pipeline.py Normal file
View File

@ -0,0 +1,63 @@
#!/usr/bin/env python3
"""Send ONE uniquely tagged probe to Kafka, then verify its indexed representation.
Does not reset consumer offsets or write directly to OpenSearch. The measurement
name 'pipeline_probe' is excluded by every supplied metric visualization filter.
"""
import json
from pathlib import Path
import subprocess
import sys
import time
import urllib.error
import urllib.request
import uuid
ROOT = Path(__file__).resolve().parents[1]
def probe():
probe_id = uuid.uuid4().hex
sample = {'name': 'pipeline_probe', 'timestamp': int(time.time() * 1000),
'tags': {'host': 'pipeline-probe', 'environment': 'diagnostic',
'role': 'diagnostic', 'series': 'pipeline-probe / diagnostic', 'probe_id': probe_id},
'fields': {'value': 1.0}}
print('Publishing one diagnostic measurement to Kafka:', probe_id, flush=True)
command = ['docker', 'compose', 'exec', '-T', 'kafka', '/opt/kafka/bin/kafka-console-producer.sh',
'--bootstrap-server', 'kafka:29092', '--topic', 'host-metrics-v1',
'--producer-property', 'acks=all', '--producer-property', 'delivery.timeout.ms=15000',
'--producer-property', 'request.timeout.ms=10000', '--producer-property', 'max.block.ms=15000']
try:
sent = subprocess.run(command, cwd=ROOT, input=json.dumps(sample) + '\n', text=True,
capture_output=True, timeout=40)
except subprocess.TimeoutExpired:
print('FAIL: Kafka producer timed out. Check broker/topic/listeners.'); return 2
if sent.returncode or 'ERROR' in sent.stderr:
print(sent.stdout, sent.stderr)
print('FAIL: Kafka did not accept the probe.'); return 2
print('Producer completed; checking Data Prepper → OpenSearch for up to 60 seconds.', flush=True)
last_error = ''
for _ in range(12):
query = {'size': 1, 'query': {'term': {'tags.probe_id': probe_id}}}
request = urllib.request.Request('http://127.0.0.1:9200/host-metrics-v1-*/_search',
data=json.dumps(query).encode(), headers={'Content-Type': 'application/json'})
try:
with urllib.request.urlopen(request, timeout=5) as response:
hits = json.load(response)['hits']['hits']
if hits:
doc = hits[0]['_source']
if not doc.get('@timestamp') or doc.get('fields', {}).get('value') != 1:
print('FAIL: probe indexed with incorrect transformation:', json.dumps(doc)); return 3
print('PASS: Kafka → Data Prepper → OpenSearch is working.')
print('Index:', hits[0]['_index'], 'timestamp:', doc['@timestamp'])
print('If native host metrics are absent, inspect the Telegraf service/configuration and its Kafka connection.')
return 0
except (urllib.error.URLError, KeyError, ValueError) as error:
last_error = str(error)
time.sleep(5)
print('FAIL: probe was not found in OpenSearch within the wait window.', last_error)
print('Inspect Data Prepper startup errors, consumer lag/backlog, OpenSearch write failures and the DLQ.')
print('Probe ID (for searching again after a backlog drains):', probe_id)
return 3
if __name__ == '__main__':
sys.exit(probe())

26
scripts/register-prometheus.py Normal file
View File

@ -0,0 +1,26 @@
#!/usr/bin/env python3
"""Register the SQL plugin's Prometheus connector, preserving existing definitions."""
import json
import urllib.request
BASE = 'http://127.0.0.1:9200/_plugins/_query/_datasources'
with urllib.request.urlopen(BASE, timeout=30) as response:
existing = json.load(response)
def has_name(value):
if isinstance(value, dict):
return value.get('name') == 'host_prometheus' or any(has_name(v) for v in value.values())
if isinstance(value, list):
return any(has_name(v) for v in value)
return False
if has_name(existing):
print('Preserving existing host_prometheus data source.')
else:
payload = {'name': 'host_prometheus', 'connector': 'prometheus',
'properties': {'prometheus.uri': 'http://prometheus:9090'}}
request = urllib.request.Request(BASE, data=json.dumps(payload).encode(),
headers={'Content-Type': 'application/json'}, method='POST')
with urllib.request.urlopen(request, timeout=30) as response:
print(response.read().decode())
print('Open Observability → Metrics and select host_prometheus. Allow 30–60 seconds for samples.')

88
scripts/repair-malformed-documents.py Normal file
View File

@ -0,0 +1,88 @@
#!/usr/bin/env python3
"""Recover the POC's former hostname-keyed JSON documents in place.
Default: read-only report. --apply requires an exclusive backup JSONL path.
Only documents missing @timestamp and containing a valid Telegraf JSON payload
are candidates. Existing IDs are retained; writes use optimistic concurrency.
"""
import argparse
from datetime import datetime, timezone
import json
from pathlib import Path
import urllib.request
BASE = 'http://127.0.0.1:9200'
def request(path, data, method='POST', content_type='application/json'):
body = data if isinstance(data, bytes) else json.dumps(data).encode()
req = urllib.request.Request(BASE + path, data=body, method=method, headers={'Content-Type': content_type})
with urllib.request.urlopen(req, timeout=30) as response:
return json.load(response)
def recover(source):
for value in source.values():
if not isinstance(value, str) or not value.lstrip().startswith('{'):
continue
try:
parsed = json.loads(value)
if not (isinstance(parsed.get('name'), str) and isinstance(parsed.get('fields'), dict)
and isinstance(parsed.get('tags'), dict) and parsed['tags'].get('host')):
continue
parsed['@timestamp'] = datetime.fromtimestamp(parsed['timestamp'] / 1000, timezone.utc).isoformat(timespec='milliseconds').replace('+00:00', 'Z')
if source.get('ingested_at'):
parsed['ingested_at'] = source['ingested_at']
return parsed
except (ValueError, TypeError, KeyError, OverflowError):
continue
return None
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--apply', action='store_true')
parser.add_argument('--backup', type=Path)
args = parser.parse_args()
if args.apply and not args.backup:
parser.error('--apply requires --backup PATH')
backup = args.backup.open('x') if args.apply else None
found = changed = skipped = 0
scroll = None
try:
page = request('/host-metrics-v1-*/_search?scroll=2m', {
'size': 500, 'seq_no_primary_term': True,
'query': {'bool': {'must_not': [{'exists': {'field': '@timestamp'}}]}}})
while True:
scroll = page.get('_scroll_id', scroll)
hits = page['hits']['hits']
if not hits:
break
operations = []
for hit in hits:
parsed = recover(hit['_source'])
if parsed is None:
skipped += 1; continue
found += 1
if args.apply:
backup.write(json.dumps(hit) + '\n')
action = {'index': {'_index': hit['_index'], '_id': hit['_id'],
'if_seq_no': hit['_seq_no'], 'if_primary_term': hit['_primary_term']}}
operations.extend([json.dumps(action), json.dumps(parsed)])
if operations:
backup.flush()
import os
os.fsync(backup.fileno())
result = request('/_bulk?refresh=wait_for', ('\n'.join(operations) + '\n').encode(), content_type='application/x-ndjson')
if result.get('errors'):
raise RuntimeError('Bulk errors; originals are backed up: ' + json.dumps(result))
changed += len(result['items'])
page = request('/_search/scroll', {'scroll': '2m', 'scroll_id': scroll})
finally:
if backup:
backup.close()
if scroll:
request('/_search/scroll', {'scroll_id': [scroll]}, method='DELETE')
print(f'Recoverable: {found}; repaired: {changed}; skipped unrecognized documents: {skipped}')
if not args.apply:
print('Read-only report. Use --apply --backup /path/to/original-documents.jsonl to repair.')
if __name__ == '__main__':
main()

12
scripts/smoke-test.sh Executable file
View File

@ -0,0 +1,12 @@
#!/usr/bin/env bash
set -euo pipefail
cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.."
docker compose ps -a
curl -fsS http://127.0.0.1:5601/api/status >/dev/null
docker compose exec -T kafka /opt/kafka/bin/kafka-topics.sh \
--bootstrap-server kafka:29092 --describe --topic host-metrics-v1
# Wait for real recent agent data and reset-aware rates, not merely container health.
python3 scripts/check-ingestion.py
docker compose exec -T kafka /opt/kafka/bin/kafka-consumer-groups.sh \
--bootstrap-server kafka:29092 --describe --group opensearch-host-metrics-v1
echo 'PASS: fresh CPU, RAM, filesystem and I/O rate documents are queryable. Review consumer lag above.'

27
scripts/start.sh Executable file
View File

@ -0,0 +1,27 @@
#!/usr/bin/env bash
set -euo pipefail
cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.."
[[ -f .env ]] || cp .env.example .env
python3 - <<'PY'
from pathlib import Path
import secrets
p = Path('.env')
if not any(line.startswith('DATASOURCE_MASTER_KEY=') and line.split('=', 1)[1].strip() for line in p.read_text().splitlines()):
with p.open('a') as f:
f.write('\nDATASOURCE_MASTER_KEY=' + secrets.token_hex(16) + '\n')
p.chmod(0o600)
PY
docker compose config --quiet
docker compose pull
docker compose up -d
echo 'Waiting for Dashboards...'
for ((i=0; i<120; i++)); do
if curl -fsS http://127.0.0.1:5601/api/status >/dev/null; then
echo 'Stack ready. Install the native agent, then import dashboards/host-metrics.ndjson.'
exit 0
fi
sleep 5
done
docker compose ps -a
echo 'Startup did not finish in 10 minutes. Run docker compose logs --tail=100.' >&2
exit 1

82
tests/test_assets.py Normal file
View File

@ -0,0 +1,82 @@
"""Contract checks; no Docker required. Does not replace the Debian smoke test."""
import json
from pathlib import Path
import tomllib
import unittest
ROOT = Path(__file__).resolve().parents[1]
class Metric:
def __init__(self, measurement, fields, seconds, **tags):
self.name, self.fields, self.time = measurement, fields, int(seconds * 1e9)
self.tags = dict(host='node-a', **tags)
class Contracts(unittest.TestCase):
def setUp(self):
# enrich.star deliberately uses Python-compatible Starlark constructs.
# This exercises rate mathematics; it is not a Starlark-runtime validation.
self.scope = {}
exec((ROOT / 'agent/enrich.star').read_text(), self.scope)
self.apply = self.scope['apply']
def test_rates_isolate_devices_and_handle_resets_gaps(self):
self.apply(Metric('net', {'bytes_recv': 100}, 10, interface='eth0'))
self.apply(Metric('net', {'bytes_recv': 8000}, 10, interface='eth1'))
got = self.apply(Metric('net', {'bytes_recv': 400}, 25, interface='eth0'))
self.assertEqual(got.fields['bytes_recv_per_sec'], 20)
self.assertEqual(got.tags['series'], 'node-a / eth0')
reset = self.apply(Metric('net', {'bytes_recv': 1}, 40, interface='eth0'))
self.assertNotIn('bytes_recv_per_sec', reset.fields)
same = self.apply(Metric('net', {'bytes_recv': 100}, 40, interface='eth0'))
self.assertNotIn('bytes_recv_per_sec', same.fields)
recover = self.apply(Metric('net', {'bytes_recv': 31}, 55, interface='eth0'))
self.assertEqual(recover.fields['bytes_recv_per_sec'], 2)
gap = self.apply(Metric('net', {'bytes_recv': 99999}, 500, interface='eth0'))
self.assertNotIn('bytes_recv_per_sec', gap.fields)
def test_busy_percent_and_normalized_load(self):
self.apply(Metric('diskio', {'io_time': 100}, 1, name='sda'))
m = self.apply(Metric('diskio', {'io_time': 7600}, 16, name='sda'))
self.assertEqual(m.fields['io_busy_percent'], 50)
m = self.apply(Metric('system', {'load1': 4, 'n_cpus': 8}, 16))
self.assertEqual(m.fields['load1_per_cpu'], 0.5)
def test_saved_object_graph_fields_and_layouts(self):
objects = [json.loads(x) for x in (ROOT / 'dashboards/host-metrics.ndjson').read_text().splitlines()]
self.assertEqual(objects, json.loads((ROOT / 'dashboards/host-metrics.json').read_text()))
ids = {(o['type'], o['id']) for o in objects}
self.assertEqual(len(ids), len(objects))
fields = {f['name'] for f in json.loads(objects[0]['attributes']['fields'])}
for o in objects:
for r in o['references']:
self.assertIn((r['type'], r['id']), ids)
a = o['attributes']
if o['type'] == 'visualization':
for agg in json.loads(a['visState'])['aggs']:
if 'field' in agg['params']:
self.assertIn(agg['params']['field'], fields)
if o['type'] == 'dashboard':
panels = json.loads(a['panelsJSON'])
for i, panel in enumerate(panels):
p = panel['gridData']
self.assertLessEqual(p['x'] + p['w'], 48)
for other in panels[i + 1:]:
q = other['gridData']
overlaps = p['x'] < q['x'] + q['w'] and q['x'] < p['x'] + p['w'] and p['y'] < q['y'] + q['h'] and q['y'] < p['y'] + p['h']
self.assertFalse(overlaps)
self.assertEqual(sum(o['type'] == 'dashboard' for o in objects), 4)
def test_configs_and_index_contract(self):
agent = tomllib.loads((ROOT / 'agent/telegraf.conf').read_text())
bridge = tomllib.loads((ROOT / 'configs/metrics-bridge/telegraf.conf').read_text())
self.assertEqual(agent['outputs']['kafka'][0]['topic'], bridge['inputs']['kafka_consumer'][0]['topics'][0])
self.assertEqual(agent['outputs']['kafka'][0]['json_timestamp_units'], '1ms')
self.assertEqual(bridge['inputs']['kafka_consumer'][0]['json_v2'][0]['timestamp_format'], 'unix_ms')
self.assertEqual(set(bridge['inputs']), {'kafka_consumer'})
template = json.loads((ROOT / 'configs/opensearch/index-template.json').read_text())
props = template['template']['mappings']['properties']
self.assertEqual(props['@timestamp']['type'], 'date')
self.assertEqual(props['fields']['properties']['used_percent']['type'], 'double')
if __name__ == '__main__':
unittest.main()