# Scaling beyond the single-host POC The schema and host-keyed Kafka topic already support multiple agents. The supplied deployment is **single-node and not highly available**: one VM failure interrupts collection, queueing, indexing and dashboards. Moving to many servers requires the following infrastructure changes, not just a larger heap. ## 1. Add a secured remote Kafka listener Keep separate internal and external listeners. Replace the EXTERNAL advertised address with a DNS name reachable from each agent and bind the host port to the intended private/VPN interface. For example, a future remote listener could advertise `metrics-kafka.example.internal:9094`; containers should continue to use the internal listener. Configure TLS with trusted server certificates and SASL/SCRAM or mutual TLS. Add Kafka ACLs so agents can produce only to the metric topic and consumer identities can read only their assigned topics/groups. Mount client CA/certificate/key files on each native host and configure Telegraf's Kafka TLS/auth settings. Keep private keys readable only by the agent. The supplied plaintext listener is a local POC configuration, not a ready remote deployment. Install the same agent bundle on each Debian host with a unique `METRICS_HOST`, environment/role tags, and the reachable bootstrap endpoint. Update the output's broker list for multiple bootstrap brokers when a cluster exists. The installer exposes one bootstrap string for the POC; edit the TOML broker array or distribute it through configuration management for several brokers. ## 2. Replicate Kafka - Use at least three brokers and a production KRaft controller quorum. Avoid colocating all replicas on one failure domain. Use separate controllers where justified by scale/availability requirements. - Set topic replication factor to 3 and `min.insync.replicas=2`; retain producer `required_acks=-1` and idempotence. Increase the internal offset and transaction-state topic replication settings too. - Migrate existing topic replicas with Kafka's reassignment tooling. Changing defaults does not replicate existing data automatically. - Begin with the existing six partitions and measure load and skew. Host-keyed routing preserves locality; an unusually busy host can dominate a partition. Increasing partitions changes future key placement, so plan ordering-sensitive transitions carefully. - Size retention for realistic sink outages and Kafka consumption lag, with headroom for segments and broker recovery. A full topic partition loses its oldest data regardless of consumer progress. ## 3. Scale Data Prepper consumers Run additional instances of the same pipeline using **the same group ID**. The group distributes partitions among consumers; different group IDs would independently consume and index the full stream, duplicating data. The POC has two consumers per instance and six partitions, so up to three instances can provide six active consumers before further instances become idle (other pipeline worker threads are distinct from Kafka consumers). There is no fixed `container_name` or published Data Prepper port, so a local capacity experiment can use: ```bash docker compose up -d --scale data-prepper=3 ``` **First give each instance a separate DLQ file/volume or an appropriately configured external DLQ sink.** The POC's one shared local DLQ file is not safe for concurrent writers. A multi-host orchestrator should provide separate instance identity, logs and failure storage. The single-host scale command is an illustration; the supplied shared-DLQ configuration must be adjusted before using it. The OpenSearch sink remains a bottleneck if indexing cannot keep up. Monitor consumer lag, sink retries, DLQ growth, CPU, heap, processing throughput and the OpenSearch write thread pool. End-to-end acknowledgments protect the offset boundary but allow duplicates after failures. If exact accounting matters, introduce a stable per-observation document ID and verify deduplication under replay before relying on counts. ## 4. Scale and secure OpenSearch Build a multi-node cluster with appropriately sized cluster-manager and data roles; enable security and trusted TLS for node transport and REST. Replace `discovery.type: single-node` with proper discovery/bootstrap configuration. Use least-privilege ingestion and dashboard users. Enable the matching Dashboards security plugin and HTTPS access via a reverse proxy or the application's TLS settings. Change the metric template to use replicas once enough data nodes exist. For example, one replica needs another data node; setting one replica on this single-node POC only makes indices yellow without adding availability. Choose shard counts from actual volume, shard sizes and recovery objectives. Daily one-shard indices suit the POC; size-based rollover/write aliases may fit larger or variable fleets. Avoid one tiny index/shard per host. Take tested OpenSearch snapshots to external storage. Kafka retention is not an OpenSearch backup; it may not retain the full indexed history, metadata, dashboards or security configuration. Use ISM hot/warm/delete transitions and tested rollup/downsampling workflows for long metric retention. Keep schema versions explicit when fields change. ## 5. Measure capacity and service health A measurement here contains multiple fields, so a measurement is not the same as one scalar time series. As an example, **40 measurements per 15-second interval** produce **230,400 documents/day/host**. At 100 hosts that is 23.04 million documents/day. At an illustrative 1 KB serialized JSON per measurement, that is roughly 23 GB/day of input before index compression, replicas, retained `_source` and storage overhead. Measure actual `pri.store.size`, input byte rate and cardinality rather than using that example as a promise. For larger fleets: - Increase collection intervals for low-change capacity signals and control device/tag cardinality. Disable per-core collection if total CPU is sufficient. - Keep raw high-resolution retention short and use appropriate aggregate retention where needed. - Maintain an inventory of expected hosts and alert on absent hosts, not just threshold crossings in received data. - Add independent monitoring of Kafka, Data Prepper, OpenSearch and the native agents. The POC monitors host resources, not every component's internal metrics. Monitoring the monitoring system on the same VM cannot report when that VM is down. - Add tested CPU/memory/disk/lag/freshness alerting with agreed durations and destinations. No alert messages or external notification channels are configured here. - Treat the optional one-instance Prometheus bridge as a separate scaling problem: multiple consumers need unique scrape targets and consistent series ownership. For high-availability, long-term metric analytics, evaluate a supported Prometheus-compatible backend and a deliberately designed metrics delivery path. Do not expose the POC's unauthenticated ports as a shortcut to remote deployment.