Log Aggregation Pipeline: Mermaid Flowchart

Log Aggregation Pipeline flowchart diagram

About Source

A log aggregation pipeline collects log output from every service, host, and container in the infrastructure, parses and enriches it, and ships it to a centralized store where engineers can search and analyze it during incidents or routine debugging.

How the pipeline works

Log data originates from multiple sources simultaneously: application processes writing JSON or plaintext to stdout, web servers writing access logs, operating system daemons, and cloud platform services emitting audit events. Each source produces logs at its own rate and in its own format.

A log shipper (Fluentd, Fluent Bit, Filebeat, Promtail) runs as a DaemonSet or sidecar and tails log files or reads from the container runtime's log driver. The shipper parses each line, attempting to extract structured fields. JSON logs are parsed directly; plaintext logs are processed with regex or grok patterns to extract fields like timestamp, log level, service name, and request ID.

After parsing, the pipeline enriches each log record with contextual metadata: Kubernetes pod name, namespace, node, deployment version, and cloud region. This metadata is added from the environment the shipper is running in, not from the application itself, so enrichment works even for third-party services that cannot be instrumented.

Enriched log events are batched and forwarded to a log aggregation backend — Elasticsearch, OpenSearch, or Grafana Loki. The backend indexes the logs, making them searchable by any field. Engineers use a query interface (Kibana, Grafana) to filter logs by time range, service, log level, or correlation ID during incident investigations (see Incident Management Flow).

Retention policies control storage costs: debug and info logs may be retained for 30 days, while audit logs are retained for one year or longer per compliance requirements.

Frequently asked questions

A log aggregation pipeline collects log output from every service, host, and container, parses and enriches it with contextual metadata, and ships it to a centralised store where engineers can search and correlate it during incidents or debugging sessions.

A log shipper (Fluentd, Fluent Bit, Promtail) tails log files or reads from the container runtime. It parses each line to extract structured fields, enriches records with environment metadata (pod name, namespace, version), and forwards batches to a backend — Elasticsearch, OpenSearch, or Grafana Loki — for indexing and querying.

Elasticsearch indexes every field in every log record, enabling full-text search but consuming significant storage and memory. Loki indexes only labels (like service name and environment), stores log text compressed, and is optimised for cost-efficient storage at scale — querying by label first, then filtering by content.

The most common mistakes are logging without structured fields (making queries slow and fragile), not setting retention policies (unbounded storage costs), and failing to include a correlation ID in logs (making it impossible to trace a request across services).

mermaid

flowchart TD
    AppLogs[Application stdout and stderr] --> Shipper[Log shipper agent]
    SysLogs[OS and infrastructure logs] --> Shipper
    PlatformLogs[Cloud platform audit logs] --> Shipper
    Shipper --> Parse[Parse log lines into structured fields]
    Parse --> Enrich[Enrich with pod and environment metadata]
    Enrich --> Filter[Filter noisy or low-value entries]
    Filter --> Batch[Batch log events for transmission]
    Batch --> Backend[Ship to log aggregation backend]
    Backend --> Index[Index logs for full-text search]
    Index --> Search[Engineer queries logs via UI]
    Index --> RetentionPolicy{Apply retention policy}
    RetentionPolicy -->|Audit logs| LongRetain[Retain for 1 year or more]
    RetentionPolicy -->|Debug and info logs| ShortRetain[Retain for 30 days]
    ShortRetain --> Expire[Expire old log data]