TLDR: Fluentd is an open-source data collector that decouples log sources from destinations. It ingests logs from 100+ sources (Nginx, Docker, syslog), normalizes them to JSON, applies filters and transformations, and routes them to 100+ outputs (Elasticsearch, S3, Kafka). Tag-based routing is the core concept.

---

📖 TLDR: A Thousand Services, One Logging Chaos

Before unified logging, a typical microservices stack looks like this:

• Nginx writes to /var/log/nginx/access.log
• Java app writes to Log4j rotation files
• Kubernetes pods write to stdout
• Database writes to /var/lib/postgresql/log/

Each destination (Splunk, Elasticsearch, S3) requires custom scripts per source. Ten services × three destinations = 30 custom scripts, each with its own error handling and retry logic.

Fluentd solves this with a unified layer: any input goes through Fluentd, gets normalized to JSON, and routes to any output using a single config.

---

🔍 Understanding Log Events: Tag, Time, and Record

Every piece of data flowing through Fluentd is called an event. Each event has exactly three components:

Think of it like a letter in a postal system: the tag is the address, the time is the postmark, and the record is the contents. Fluentd's routing engine reads only the tag to decide where the letter goes — it never needs to open the envelope.

Fluentd itself runs as a long-lived daemon process. On startup it reads a single config file (typically /etc/fluent/fluent.conf), initialises the requested plugins, and then loops continuously: collecting events, running them through the filter chain, and flushing them to outputs. This daemon model means configuration changes require a reload, but operational overhead is minimal once running.

---

🔢 Tags and Routing: Fluentd's Core Concept

Every event in Fluentd has a tag — a dot-separated string that determines where the event is routed.

<source>
  @type tail
  path /var/log/nginx/access.log
  tag web.nginx
  format nginx
</source>

<source>
  @type tail
  path /var/log/app/app.log
  tag app.backend
  format json
</source>

<match web.**>
  @type elasticsearch
  host elastic.local
  port 9200
  index_name nginx-logs
</match>

<match app.**>
  @type s3
  s3_bucket my-log-archive
  path logs/%Y/%m/%d/
</match>

• web.nginx events match <match web.**> → go to Elasticsearch
• app.backend events match <match app.**> → go to S3

flowchart TD
    Nginx[Nginx access.log tag: web.nginx] --> Fluentd
    App[App log tag: app.backend] --> Fluentd
    Sys[Syslog tag: system.kernel] --> Fluentd

    Fluentd -->|match web.**| ES[Elasticsearch]
    Fluentd -->|match app.**| S3[Amazon S3]
    Fluentd -->|match system.**| Kafka[Kafka topic]

---

📊 End-to-End Pipeline: From Log Line to Storage

Understanding the tag concept is one thing — seeing data flow from a raw log line all the way to a storage backend makes it concrete. The diagram below traces a single Nginx request through every stage of a typical Fluentd pipeline:

flowchart LR
    A[ Nginx access.log] -->|tail plugin reads new lines| B[ Input: tail tag: web.nginx]
    B -->|raw text  JSON| C[ Parser: nginx {status, path, method}]
    C -->|add hostname + env fields| D[ Filter: record_transformer {status, path, hostname, env}]
    D -->|batch to disk first| E[ Buffer: file /var/log/fluentd-buffer]
    E -->|flush every 5 s| F[ Output: elasticsearch nginx-logs index]

Here is what happens at each step:

1. Input (tail) — Fluentd watches /var/log/nginx/access.log for new lines, stamping each with the tag web.nginx and a Unix timestamp.
2. Parser (nginx) — The raw text line (127.0.0.1 - [date] "GET /api" 200 512) is parsed into a structured JSON record using Nginx's known log format.
3. Filter (record_transformer) — Additional fields (hostname, environment, derived log_level) are injected before the record leaves the filter chain.
4. Buffer (file) — The enriched record is written to a local file buffer. If Elasticsearch is unavailable, no data is lost — Fluentd will retry with exponential backoff.
5. Output (elasticsearch) — On each flush cycle the buffer is read, and the batch of records is bulk-indexed into the nginx-logs index.

📊 Log Pipeline: App to Elasticsearch

sequenceDiagram
    participant A as App / Nginx
    participant I as Fluentd Input
    participant F as Filter Chain
    participant B as Buffer (file)
    participant O as Output Plugin
    participant E as Elasticsearch

    A->>I: write log line to file / stdout
    I->>I: tail plugin reads new lines
    I->>F: raw event {tag, time, record}
    F->>F: parser: text  JSON
    F->>F: record_transformer: add fields
    F->>B: write enriched event to disk
    Note over B: durable  survives restarts
    B->>O: flush batch every 5s
    O->>E: bulk index to nginx-logs
    E-->>O: 200 OK

This sequence diagram traces a single Nginx log line from the moment it lands on disk all the way to Elasticsearch indexing. Notice that the buffer step in the middle is what provides durability — the event is written to disk before the output plugin even attempts to send it, so a downstream outage only stalls flushing, never loses data. Reading this diagram from top to bottom maps directly to the five pipeline stages described in the text above.

📊 Fluentd Plugin Architecture

flowchart LR
    IN[Input Plugin (tail, http, syslog)] --> PA[Parser (nginx, json, csv)]
    PA --> FI[Filter Plugin (grep, transform, geoip)]
    FI --> BU[Buffer Plugin (file, memory)]
    BU --> OUT[Output Plugin (elasticsearch, s3, kafka)]

This diagram shows the five plugin types as a strict left-to-right pipeline: every event enters through an Input, gets parsed into structured JSON, passes through zero or more Filter transformations, is written to a Buffer for durability, and finally exits through an Output plugin. The modular design means you can swap any stage independently — for example, replacing the Elasticsearch output with an S3 output requires changing only the rightmost box, leaving all upstream plugins untouched.

---

⚙️ The Plugin Architecture: Input → Filter → Buffer → Output

Fluentd's power comes from its plugin model:

Buffer plugins are critical for reliability. Without buffering, a downstream outage (e.g., Elasticsearch restart) causes log loss. With a file buffer:

• Events are written to disk first.
• Flushed to the output on schedule (or when the buffer fills).
• Retried automatically on failure with exponential backoff.

<match **>
  @type elasticsearch
  host elastic.local

  <buffer>
    @type file
    path /var/log/fluentd-buffer
    flush_interval 5s
    retry_max_times 10
  </buffer>
</match>

---

🧠 Deep Dive: Filter Enriching and Transforming Events

Filters run in the pipeline between input and output:

<filter web.nginx>
  @type record_transformer
  enable_ruby true
  <record>
    environment "production"
    hostname "#{Socket.gethostname}"
    log_level ${record["status"].to_i >= 500 ? "ERROR" : "INFO"}
  </record>
</filter>

This adds environment, hostname, and a derived log_level to every Nginx event before sending to Elasticsearch.

---

🌍 Real-World Applications: Fluentd vs Logstash vs Fluent Bit

The common production pattern: Fluent Bit as a lightweight DaemonSet on every node, forwarding to a central Fluentd aggregation layer, then to Elasticsearch/S3.

---

🧪 Practical: Collecting Kubernetes Container Logs

In Kubernetes every container's stdout/stderr lands in /var/log/containers/ on the host node. A Fluentd DaemonSet reads these files and ships them to a central store. The ConfigMap below shows a minimal but production-ready pipeline:

# fluentd-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos   # ← resume after restart
      tag kubernetes.*
      read_from_head true
      <parse>
        @type json
      </parse>
    </source>

    <filter kubernetes.**>
      @type kubernetes_metadata   # ← enriches with pod name, namespace, labels
    </filter>

    <match kubernetes.**>
      @type elasticsearch
      host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
      port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
      logstash_format true        # ← creates logstash-YYYY.MM.DD indices for Kibana
      logstash_prefix k8s
      <buffer>
        @type file
        path /var/log/fluentd-buffer
        flush_interval 5s
        retry_max_times 17
      </buffer>
    </match>

Three design decisions are worth calling out:

• pos_file records the last-read byte offset for every log file. If the Fluentd pod restarts (node eviction, OOM, rolling upgrade), it resumes from exactly where it left off — no duplicate lines, no gaps.
• kubernetes_metadata filter makes a call to the Kubernetes API server to enrich each record with the pod name, namespace, deployment label, and container name. This is what lets you filter in Kibana by kubernetes.namespace_name: "payments" rather than hunting through raw file paths.
• logstash_format true tells the Elasticsearch output to create time-partitioned indices (k8s-2026.03.09) using the same naming convention Kibana's default index patterns expect, so dashboards work out of the box.

---

⚖️ Trade-offs & Failure Modes: Fluentd Configuration Pitfalls

---

🧭 Decision Guide: When to Choose Fluentd

---

🛠️ Logback Fluentd Appender: Shipping Structured Logs from Spring Boot to Fluentd

Spring Boot applications log via SLF4J + Logback by default. The fluent-logger-java Logback appender encodes each log event as a MessagePack record and forwards it over TCP to Fluentd's @type forward input — the same Tag, Time, Record model described in this post. Structured field arguments (via logstash-logback-encoder) become JSON keys in the Fluentd record, making them fully queryable in Kibana without any Fluentd filter plugin.

<!-- pom.xml -->
<dependency>
    <groupId>org.fluentd</groupId>
    <artifactId>fluent-logger</artifactId>
    <version>0.3.4</version>
</dependency>
<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>7.4</version>
</dependency>

<!-- src/main/resources/logback-spring.xml -->
<configuration>
    <!-- Console output for local development -->
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
    </appender>

    <!-- Fluentd forward appender for production -->
    <appender name="FLUENTD"
              class="org.fluentd.logger.logback.FluentLogbackAppender">
        <tag>app.payments</tag>           <!-- routes to <match app.**> in Fluentd -->
        <remoteHost>fluentd-aggregator</remoteHost>
        <port>24224</port>
        <maxQueueSize>500</maxQueueSize>  <!-- in-memory buffer before forward -->
    </appender>

    <springProfile name="prod">
        <root level="INFO">
            <appender-ref ref="FLUENTD"/>
        </root>
    </springProfile>
    <springProfile name="!prod">
        <root level="INFO">
            <appender-ref ref="CONSOLE"/>
        </root>
    </springProfile>
</configuration>

// PaymentService.java — structured fields appear as JSON keys in Fluentd records
import net.logstash.logback.argument.StructuredArguments;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@Service
public class PaymentService {
    private static final Logger log = LoggerFactory.getLogger(PaymentService.class);

    public void processPayment(String orderId, double amount) {
        log.info("Payment processed",
            StructuredArguments.kv("orderId",   orderId),  // → "orderId":"ORD-42"
            StructuredArguments.kv("amount",    amount),   // → "amount":49.99
            StructuredArguments.kv("currency",  "USD")
        );
        // Fluentd receives: tag=app.payments
        // Record: {"message":"Payment processed","orderId":"ORD-42","amount":49.99}
    }
}

The tag: app.payments in Logback maps directly to Fluentd's <match app.**> routing block — following the app.<service-name> naming convention means adding a new service requires no Fluentd configuration changes.

For a full deep-dive on structured logging pipelines with Fluentd, Fluent Bit, and Logstash from Spring Boot, a dedicated follow-up post is planned.

---

📚 Lessons from Fluentd in Production

Running Fluentd at scale surfaces patterns that do not appear in tutorials:

• Always buffer to disk for critical log streams, not memory. A memory buffer (@type memory) is faster but evaporates on process restart or OOM kill. File buffers survive pod restarts and node reboots. The performance difference is minimal for most log volumes (< 50 MB/s), so the durability trade-off is nearly always worth it.
• Establish a tag naming convention before you have ten teams. Tags like env.team.service (prod.payments.api) make wildcard routing predictable: <match prod.**> catches all production traffic; <match prod.payments.**> catches only the payments team. Without a convention, configs become a maze of overlapping ** rules within weeks.
• Monitor buffer utilisation via the built-in monitoring API. Fluentd exposes a Prometheus-compatible metrics endpoint on port 24220. Alert when buffer_total_queued_size exceeds 80 % of the configured total_limit_size — that is your early warning that a downstream output is degraded and backpressure is building.
• Run Fluent Bit at the edge and Fluentd at the centre for large clusters. In clusters with 50+ nodes, deploying full Fluentd as a DaemonSet burns ~60 MB per node just for the Ruby runtime. Fluent Bit uses ~1 MB per node and forwards raw events to a small Fluentd aggregation Deployment. Fluentd then handles the heavy work — routing, enrichment, multi-output fanout — on a handful of pods instead of every node.

---

📌 TLDR: Summary & Key Takeaways

• Fluentd collects logs from any source, normalizes to JSON, and routes to any destination via tag-based matching.
• Plugin types: Input → Parser → Filter → Buffer → Output.
• Buffer plugins provide durability: events survive output outages.
• Compared to Logstash (Java, heavy) and Fluent Bit (C, ultra-light), Fluentd sits in the middle as a reliable aggregation layer.
• Common pattern: Fluent Bit (edge) → Fluentd (aggregation) → Elasticsearch or Kafka.

---

🔗 Related Posts

• How Kubernetes Works
• How Kafka Works
• System Design Advanced: Security, Rate Limiting, and Reliability
• Elasticsearch vs Time-Series DB