How would you design a centralized log aggregation pipeline for a fleet of services, from collection through to searchable storage?

A typical pipeline has four distinct stages: a collection agent on each node/pod (reading stdout or log files and forwarding them), an optional buffering/streaming layer (to absorb load spikes and decouple producers from the storage backend), a processing stage (parsing, enriching, and structuring raw log lines), and a storage/indexing backend that supports fast search (commonly Elasticsearch/OpenSearch, or a log-specific system like Loki) — each stage exists to solve a specific problem the previous stage alone can't.

Each stage of this pipeline exists because the stage before it alone doesn't solve every requirement — collection alone doesn't survive load spikes, and even buffered raw logs aren't genuinely searchable without processing and indexed storage on top.

• Logs from every service across the fleet must be collected without requiring each service to individually manage its own shipping logic.
• The pipeline must handle load spikes without dropping logs or overwhelming the storage backend.
• Logs must be genuinely searchable (not just stored) within a reasonable time of being generated.
• The design should scale as the number of services and log volume grows, without requiring a redesign at each growth stage.

A collection agent runs per-node (or as a sidecar) and handles the actual reading/forwarding: agents like Fluent Bit, Fluentd, Vector, or a cloud-provider-specific equivalent run on every node (as a DaemonSet in Kubernetes, for instance), reading container stdout/stderr or log files directly and forwarding them onward — this centralizes the collection logic in one well-tested component rather than requiring every individual service to implement its own shipping code.

A buffering/streaming layer decouples log production from the storage backend's ingest rate: a message queue or streaming system (Kafka is common) sitting between collection and processing/storage absorbs bursts in log volume (a sudden spike in error logging during an incident, ironically often exactly when you most need logs to keep flowing) without those bursts directly overwhelming or backpressuring the storage backend — this is what keeps a log spike from becoming a second incident on top of the first.

A processing stage parses, enriches, and structures raw log lines before storage: this is where free-text or semi-structured log lines get parsed into genuinely structured fields, where common enrichment happens (adding a Kubernetes namespace/pod label, geo-IP lookup, or similar), and where routing decisions (which logs go to which storage index, or get dropped/sampled for cost reasons) are made — tools like Logstash, Fluentd's own processing capabilities, or Vector fill this role.

The storage/indexing backend is what actually makes logs searchable at query time: Elasticsearch/OpenSearch (full-text and structured search, more resource-intensive) or Loki (indexes only labels, not full log content, trading some search flexibility for meaningfully lower storage/indexing cost) are the common choices — the right choice depends on how much genuinely full-text search capability is needed versus how much cost/operational overhead is acceptable, since full-text-indexing systems are considerably more expensive to run at scale than label-indexed ones.

Retention and cost management need to be designed in from the start, not bolted on later: log volume grows continuously and indefinitely without a retention policy — tiered retention (recent logs in fast, expensive storage; older logs moved to cheaper, slower storage, or deleted entirely past a compliance-driven or cost-driven cutoff) needs to be a deliberate part of the architecture, since an unmanaged, ever-growing log volume becomes a real, escalating cost problem.

A full pipeline (agent, buffer, processor, indexed storage) is meaningfully more infrastructure than a simpler "agent ships directly to storage" design, but the buffering and processing stages are what provide resilience against load spikes and genuinely structured, enriched data — skipping them is a reasonable simplification for a small-scale setup, but becomes a real reliability and usability gap as fleet size and log volume grow. Choosing a full-text-indexed backend (Elasticsearch) versus a label-indexed one (Loki) is a direct cost-versus-search-flexibility trade-off that should be made deliberately based on actual query patterns, not defaulted to whichever tool is most familiar.

• A typical pipeline has four stages: per-node collection agent, buffering/streaming layer, processing (parse/enrich/structure), and searchable storage — each solving a distinct problem.
• A buffering layer (like Kafka) is specifically what prevents a log-volume spike (often occurring during an incident) from overwhelming storage or causing log loss right when logs matter most.
• The storage backend choice (full-text-indexed vs. label-indexed) is a direct cost-versus-search-flexibility trade-off that should match actual query patterns.
• Retention/tiering needs to be designed in from the start, since unmanaged log volume growth becomes an escalating, unbounded cost.

• Fluent Bit: Documentation
• Grafana Loki: Documentation