97 lines
8.8 KiB
Markdown
97 lines
8.8 KiB
Markdown
# Operations and troubleshooting
|
||
|
||
Run these commands from the deployment directory on Debian.
|
||
|
||
## Health and data
|
||
|
||
```bash
|
||
docker compose ps -a
|
||
docker compose logs --tail=100 kafka data-prepper opensearch dashboards
|
||
sudo journalctl -u telegraf-host-metrics -n 100 --no-pager
|
||
./scripts/smoke-test.sh
|
||
```
|
||
|
||
The init containers should exit with code 0. A running Data Prepper container alone does not prove delivery; use the ingestion check. For host-specific validation:
|
||
|
||
```bash
|
||
METRICS_HOST=debian-monitor-01 python3 scripts/check-ingestion.py
|
||
```
|
||
|
||
Inspect the topic and lag:
|
||
|
||
```bash
|
||
docker compose exec -T kafka /opt/kafka/bin/kafka-consumer-groups.sh \
|
||
--bootstrap-server kafka:29092 --describe --group opensearch-host-metrics-v1
|
||
curl -fsS 'http://127.0.0.1:9200/_cat/indices/host-metrics-v1-*?v'
|
||
curl -fsS 'http://127.0.0.1:9200/host-metrics-v1-*/_search?size=1&sort=@timestamp:desc'
|
||
curl -fsS 'http://127.0.0.1:9200/_plugins/_ism/explain/host-metrics-v1-*?show_policy=true'
|
||
```
|
||
|
||
Lag may oscillate while a batch is being acknowledged. Sustained growth means the consumer cannot keep up or the sink is unhealthy. `auto_offset_reset=earliest` only applies when there is no valid committed offset; it does not rewind a running group.
|
||
|
||
## Storage, retention and delivery limits
|
||
|
||
- Kafka records persist in `kafka-data`. Retention is **72 hours or approximately 1 GiB per partition**, whichever deletes old data first (about 6 GiB for this topic, plus segment and broker overhead). Retention is segment-based and not a hard immediate disk quota. It applies even when a consumer is behind.
|
||
- OpenSearch daily indices persist in `opensearch-data`. ISM deletes matching indices after **14 days of index age**, checked periodically. It does not delete individual documents by their exact event age. Very old replayed records may create an old-named index that is newly created and therefore remains for another 14 days. The template has one primary shard and zero replicas.
|
||
- Data Prepper uses an in-memory processing buffer, with end-to-end Kafka acknowledgments and disabled periodic offset auto-commit. Kafka is the durable queue. Permanent indexing failures can be written to `prepper-dlq`; check logs and this volume. Transient sink failures retry with backpressure. There is no distributed transaction between Kafka and OpenSearch.
|
||
- Native Telegraf has a **100,000-measurement memory output buffer**, with acknowledged/idempotent Kafka writes. While the agent stays alive it retries failed output writes. When full, data is dropped; a process/VM restart loses unsent buffered data. Idempotent Kafka writes do not make the whole pipeline exactly once.
|
||
- Telegraf 1.40 offers an **experimental** disk buffer. To evaluate it, set `buffer_strategy = "disk"`, `buffer_directory = "/var/lib/telegraf-host-metrics/buffer"` and `buffer_disk_sync = true` in the native config, then restart the unit and test outage/restart behavior. The systemd StateDirectory is writable. This POC defaults to the stable memory buffer and does not claim durable collection during agent outages.
|
||
- Optional Prometheus keeps 7 days or 5 GB of TSDB blocks; additional WAL/head space is needed. Its adapter is a live snapshot, not an archive importer.
|
||
- Container logs rotate at 10 MB × 3 files per container. The Data Prepper DLQ file is not automatically rotated. Inspect/export it regularly and monitor disk usage.
|
||
|
||
Inspect the DLQ without modifying it:
|
||
|
||
```bash
|
||
docker compose exec -T data-prepper sh -c \
|
||
'ls -lah /usr/share/data-prepper/dlq; tail -n 10 /usr/share/data-prepper/dlq/failed-events.json'
|
||
```
|
||
|
||
The file may not exist if no failures occurred. DLQ contents include failed event/context records, not necessarily original Kafka JSON. Do not blindly pipe the file into Kafka; inspect it, correct the cause and extract original metric events for deliberate replay.
|
||
|
||
## Common failures
|
||
|
||
| Symptom | Checks / remedy |
|
||
|---|---|
|
||
| OpenSearch exits during startup | Inspect logs for `vm.max_map_count`, memory or ulimit errors. Run `prepare-host.sh`; verify available RAM and Docker resources. |
|
||
| Kafka healthy, native agent cannot send | Confirm the published listener is `127.0.0.1:9092`, the topic exists and the agent runs on the same VM. `kafka:29092` is only resolvable inside Docker. |
|
||
| Agent can bootstrap remotely but then fails | Kafka returns `advertised.listeners` to clients. The advertised address must be reachable from that client; changing only the bootstrap address is insufficient. |
|
||
| Data Prepper reports SSL errors connecting to Kafka | Keep `encryption.type: none` for this plaintext POC. Data Prepper otherwise defaults to TLS. |
|
||
| Dashboards says “not ready” | Check OpenSearch health and `opensearch-init`. Server/client versions must match; security must be disabled consistently on both sides here. |
|
||
| Import succeeded but no charts | Check Last 1 hour, clear accidental filters, run the smoke test, inspect `@timestamp`, host UTC time and `tags.host`. Do not create a different index pattern with the same title; the supplied visuals reference `hm-index-v1`. |
|
||
| Mapping failures after editing agent fields | Custom `fields.*` are numeric. Do not add string fields to that object. A changed field type requires a new schema/index version, not just a template edit. |
|
||
| Root FS pressure / read-only index | Free disk space, inspect OpenSearch disk watermarks, then clear the affected read-only block after resolving capacity. Do not disable disk watermarks to hide the problem. |
|
||
| Prometheus has no host metrics | Check `docker compose --profile metrics-analytics logs metrics-bridge prometheus`. A new bridge starts from newest Kafka offsets; wait for new agent samples. Inspect `http://127.0.0.1:9090/api/v1/targets` on the VM. |
|
||
| Metric Analytics cannot list the source | Run `register-prometheus.py`; inspect its HTTP error and OpenSearch logs. Confirm the encryption key is configured and the URI is `http://prometheus:9090`, not localhost. Use `check-metric-analytics.py` to test PPL separately from the UI. |
|
||
|
||
## Restart, update and remove
|
||
|
||
```bash
|
||
# Apply edited container configurations.
|
||
docker compose up -d --force-recreate data-prepper dashboards
|
||
# Apply an edited native collector config.
|
||
sudo systemctl restart telegraf-host-metrics
|
||
# Stop containers while retaining named volumes.
|
||
docker compose --profile metrics-analytics down
|
||
# Stop the dedicated native service.
|
||
sudo systemctl disable --now telegraf-host-metrics
|
||
```
|
||
|
||
Restart core services with `./scripts/start.sh`; re-enable the optional profile with `./scripts/enable-metric-analytics.sh`. Running the enable script also rechecks the data-source registration. Already-created Kafka topic settings and the existing ISM policy are deliberately preserved by the initializers; changing a source JSON/property file does not update those existing resources automatically. Apply updates through their respective management APIs after reviewing the intended change.
|
||
|
||
To modify the ISM policy, GET the policy with its `_seq_no` and `_primary_term`, then PUT the edited body with optimistic concurrency parameters. Verify the managed-index policy state afterward. To change Kafka topic retention, use `kafka-configs.sh --alter --entity-type topics --entity-name host-metrics-v1 --add-config ...`. Changing index templates affects future indices, not existing ones.
|
||
|
||
Before version upgrades, save OpenSearch snapshots and your configuration, `.env` and dashboards; test the new versions in another deployment. Do not downgrade OpenSearch in place after a newer version has opened its data directory. Update the native package pin together with the bridge image when changing Telegraf. Regenerate dashboard/mapping assets with `python3 scripts/generate-assets.py`, then reimport the NDJSON.
|
||
|
||
**Destructive teardown:** `docker compose --profile metrics-analytics down -v` removes the deployment's named volumes and their data. It does not uninstall the native agent. This is never run by the setup scripts.
|
||
|
||
## Suggested POC acceptance exercise
|
||
|
||
1. Run the smoke test and import the dashboards. Verify CPU/RAM/`/` filesystem values against `top`, `free` and `df` on the VM, allowing for different memory definitions and averaging windows.
|
||
2. Briefly create CPU/network/disk activity using your own disposable test workload. Verify the corresponding charts and host filter.
|
||
3. Stop Data Prepper for one minute. Verify the agent still sends, Kafka offsets advance, and consumption catches up after restarting Data Prepper.
|
||
4. Stop Kafka briefly, then restart it. Inspect the agent retry logs and recovery. Keep the outage shorter than the memory buffer capacity; do not claim a measured recovery limit without testing at your load.
|
||
5. Restart the native agent. Confirm counters warm up without negative or enormous rate spikes.
|
||
6. Enable optional Metric Analytics and run its separate checker. Verify at least one chart in the UI.
|
||
|
||
These are manual acceptance steps, not tests already performed in the development workspace.
|