flavien/host-metrics
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
main
host-metrics/docs/OPERATIONS.md
flavien 0ae0d515ad first commit
2026-09-17 22:31:35 +02:00

8.8 KiB
Raw Permalink Blame History

Operations and troubleshooting

Run these commands from the deployment directory on Debian.

Health and data

docker compose ps -a
docker compose logs --tail=100 kafka data-prepper opensearch dashboards
sudo journalctl -u telegraf-host-metrics -n 100 --no-pager
./scripts/smoke-test.sh

The init containers should exit with code 0. A running Data Prepper container alone does not prove delivery; use the ingestion check. For host-specific validation:

METRICS_HOST=debian-monitor-01 python3 scripts/check-ingestion.py

Inspect the topic and lag:

docker compose exec -T kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server kafka:29092 --describe --group opensearch-host-metrics-v1
curl -fsS 'http://127.0.0.1:9200/_cat/indices/host-metrics-v1-*?v'
curl -fsS 'http://127.0.0.1:9200/host-metrics-v1-*/_search?size=1&sort=@timestamp:desc'
curl -fsS 'http://127.0.0.1:9200/_plugins/_ism/explain/host-metrics-v1-*?show_policy=true'

Lag may oscillate while a batch is being acknowledged. Sustained growth means the consumer cannot keep up or the sink is unhealthy. auto_offset_reset=earliest only applies when there is no valid committed offset; it does not rewind a running group.

Storage, retention and delivery limits

  • Kafka records persist in kafka-data. Retention is 72 hours or approximately 1 GiB per partition, whichever deletes old data first (about 6 GiB for this topic, plus segment and broker overhead). Retention is segment-based and not a hard immediate disk quota. It applies even when a consumer is behind.
  • OpenSearch daily indices persist in opensearch-data. ISM deletes matching indices after 14 days of index age, checked periodically. It does not delete individual documents by their exact event age. Very old replayed records may create an old-named index that is newly created and therefore remains for another 14 days. The template has one primary shard and zero replicas.
  • Data Prepper uses an in-memory processing buffer, with end-to-end Kafka acknowledgments and disabled periodic offset auto-commit. Kafka is the durable queue. Permanent indexing failures can be written to prepper-dlq; check logs and this volume. Transient sink failures retry with backpressure. There is no distributed transaction between Kafka and OpenSearch.
  • Native Telegraf has a 100,000-measurement memory output buffer, with acknowledged/idempotent Kafka writes. While the agent stays alive it retries failed output writes. When full, data is dropped; a process/VM restart loses unsent buffered data. Idempotent Kafka writes do not make the whole pipeline exactly once.
  • Telegraf 1.40 offers an experimental disk buffer. To evaluate it, set buffer_strategy = "disk", buffer_directory = "/var/lib/telegraf-host-metrics/buffer" and buffer_disk_sync = true in the native config, then restart the unit and test outage/restart behavior. The systemd StateDirectory is writable. This POC defaults to the stable memory buffer and does not claim durable collection during agent outages.
  • Optional Prometheus keeps 7 days or 5 GB of TSDB blocks; additional WAL/head space is needed. Its adapter is a live snapshot, not an archive importer.
  • Container logs rotate at 10 MB × 3 files per container. The Data Prepper DLQ file is not automatically rotated. Inspect/export it regularly and monitor disk usage.

Inspect the DLQ without modifying it:

docker compose exec -T data-prepper sh -c \
  'ls -lah /usr/share/data-prepper/dlq; tail -n 10 /usr/share/data-prepper/dlq/failed-events.json'

The file may not exist if no failures occurred. DLQ contents include failed event/context records, not necessarily original Kafka JSON. Do not blindly pipe the file into Kafka; inspect it, correct the cause and extract original metric events for deliberate replay.

Common failures

Symptom Checks / remedy
OpenSearch exits during startup Inspect logs for vm.max_map_count, memory or ulimit errors. Run prepare-host.sh; verify available RAM and Docker resources.
Kafka healthy, native agent cannot send Confirm the published listener is 127.0.0.1:9092, the topic exists and the agent runs on the same VM. kafka:29092 is only resolvable inside Docker.
Agent can bootstrap remotely but then fails Kafka returns advertised.listeners to clients. The advertised address must be reachable from that client; changing only the bootstrap address is insufficient.
Data Prepper reports SSL errors connecting to Kafka Keep encryption.type: none for this plaintext POC. Data Prepper otherwise defaults to TLS.
Dashboards says “not ready” Check OpenSearch health and opensearch-init. Server/client versions must match; security must be disabled consistently on both sides here.
Import succeeded but no charts Check Last 1 hour, clear accidental filters, run the smoke test, inspect @timestamp, host UTC time and tags.host. Do not create a different index pattern with the same title; the supplied visuals reference hm-index-v1.
Mapping failures after editing agent fields Custom fields.* are numeric. Do not add string fields to that object. A changed field type requires a new schema/index version, not just a template edit.
Root FS pressure / read-only index Free disk space, inspect OpenSearch disk watermarks, then clear the affected read-only block after resolving capacity. Do not disable disk watermarks to hide the problem.
Prometheus has no host metrics Check docker compose --profile metrics-analytics logs metrics-bridge prometheus. A new bridge starts from newest Kafka offsets; wait for new agent samples. Inspect http://127.0.0.1:9090/api/v1/targets on the VM.
Metric Analytics cannot list the source Run register-prometheus.py; inspect its HTTP error and OpenSearch logs. Confirm the encryption key is configured and the URI is http://prometheus:9090, not localhost. Use check-metric-analytics.py to test PPL separately from the UI.

Restart, update and remove

# Apply edited container configurations.
docker compose up -d --force-recreate data-prepper dashboards
# Apply an edited native collector config.
sudo systemctl restart telegraf-host-metrics
# Stop containers while retaining named volumes.
docker compose --profile metrics-analytics down
# Stop the dedicated native service.
sudo systemctl disable --now telegraf-host-metrics

Restart core services with ./scripts/start.sh; re-enable the optional profile with ./scripts/enable-metric-analytics.sh. Running the enable script also rechecks the data-source registration. Already-created Kafka topic settings and the existing ISM policy are deliberately preserved by the initializers; changing a source JSON/property file does not update those existing resources automatically. Apply updates through their respective management APIs after reviewing the intended change.

To modify the ISM policy, GET the policy with its _seq_no and _primary_term, then PUT the edited body with optimistic concurrency parameters. Verify the managed-index policy state afterward. To change Kafka topic retention, use kafka-configs.sh --alter --entity-type topics --entity-name host-metrics-v1 --add-config .... Changing index templates affects future indices, not existing ones.

Before version upgrades, save OpenSearch snapshots and your configuration, .env and dashboards; test the new versions in another deployment. Do not downgrade OpenSearch in place after a newer version has opened its data directory. Update the native package pin together with the bridge image when changing Telegraf. Regenerate dashboard/mapping assets with python3 scripts/generate-assets.py, then reimport the NDJSON.

Destructive teardown: docker compose --profile metrics-analytics down -v removes the deployment's named volumes and their data. It does not uninstall the native agent. This is never run by the setup scripts.

Suggested POC acceptance exercise

  1. Run the smoke test and import the dashboards. Verify CPU/RAM// filesystem values against top, free and df on the VM, allowing for different memory definitions and averaging windows.
  2. Briefly create CPU/network/disk activity using your own disposable test workload. Verify the corresponding charts and host filter.
  3. Stop Data Prepper for one minute. Verify the agent still sends, Kafka offsets advance, and consumption catches up after restarting Data Prepper.
  4. Stop Kafka briefly, then restart it. Inspect the agent retry logs and recovery. Keep the outage shorter than the memory buffer capacity; do not claim a measured recovery limit without testing at your load.
  5. Restart the native agent. Confirm counters warm up without negative or enormous rate spikes.
  6. Enable optional Metric Analytics and run its separate checker. Verify at least one chart in the UI.

These are manual acceptance steps, not tests already performed in the development workspace.