Log Aggregation Guide
Stack Comparison
| Stack | Components | Best For | Cost |
|---|---|---|---|
| ELK | Elasticsearch + Logstash + Kibana | Full-text search, complex queries, large scale | High (resource hungry) |
| EFK | Elasticsearch + Fluentd + Kibana | Kubernetes native, structured logs | High |
| PLG (Loki) | Prometheus + Loki + Grafana | K8s, label-based, cost-efficient | Low |
| Vector + ClickHouse | Vector + ClickHouse + Grafana | High throughput, SQL analytics | Medium |
Fluent Bit Configuration (Kubernetes)
# fluent-bit.conf
[SERVICE]
Flush 1
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
[OUTPUT]
Name es
Match *
Host elasticsearch.logging
Port 9200
Logstash_Format On
Logstash_Prefix logs
Retry_Limit False
Structured Logging (Go)
// Using slog (Go 1.21+)
import "log/slog"
logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo,
}))
// Structured log entry
logger.Info("request processed",
slog.String("method", "GET"),
slog.String("path", "/api/users"),
slog.Int("status", 200),
slog.Duration("latency", 45*time.Millisecond),
slog.String("trace_id", traceID),
)
// Output:
// {"time":"2024-01-15T10:00:00Z","level":"INFO","msg":"request processed",
// "method":"GET","path":"/api/users","status":200,"latency":"45ms"}
Loki LogQL Examples
# Stream selector (labels)
{app="my-app", env="production"}
# Filter for error logs
{app="my-app"} |= "error"
# Regex filter
{app="my-app"} |~ "ERR|FATAL"
# Parse JSON logs
{app="my-app"} | json | status >= 500
# Rate of error lines per minute
rate({app="my-app"} |= "error" [1m])
# Top paths by request count (last 1h)
topk(10, sum by (path) (
count_over_time({app="api"} | json | __error__="" [1h])
))