Loki Log Collection Setup: Label Cardinality and Query Performance

Introduction

Grafana Loki is a log aggregation system that indexes metadata (labels) rather than full log content. This approach makes it incredibly storage-efficient compared to full-text search engines like Elasticsearch. But the trade-off is that label management becomes critical — high-cardinality labels can destroy performance and blow up your storage costs.

I set up Loki to collect logs from a Docker-based deployment with about 15 services. The initial setup worked fine with a small amount of log data, but as the system grew, Loki started consuming excessive memory and queries became painfully slow. The root cause was label cardinality — I was using request IDs and user IDs as labels, which created millions of unique label combinations.

Environment

• Loki version: 2.9.3
• Deployment: Docker Compose with Loki, Promtail, Grafana
• Log sources: 15 Docker containers (Go, Python, Node.js services)
• Storage: Local filesystem (planning migration to S3-compatible storage)
• Expected log volume: ~10GB per day

Problem

After running Loki for about two weeks with increasing log volume, I noticed the following symptoms:

1. Loki's memory usage climbed from 512MB to over 4GB
2. Log queries in Grafana timed out after 30 seconds
3. Promtail was falling behind, with a growing backlog of unprocessed logs
4. The Loki compactor was running continuously but not freeing up space

I checked the Loki metrics:

curl -s http://localhost:3100/metrics | grep "loki_ingester_memory_chunks"
# loki_ingester_memory_chunks 12847523

Over 12 million chunks in memory — far too many for a small deployment. The problem was clear when I examined the label values:

{job="api-server"} | json | line_format "{{.request_id}}"

The request_id label was generating a unique value for every single log line. Since Loki indexes labels, this meant every log line had a unique index entry, completely defeating the purpose of label-based indexing.

Analysis

I quantified the cardinality problem by counting unique label values:

# Using logcli to check label cardinality
logcli labels request_id --timezone=UTC
# 4582931 unique values

logcli labels user_id --timezone=UTC
# 289456 unique values

With 4.5 million unique request_id values and 290,000 unique user_id values, the combined cardinality was enormous. Each unique label combination created a separate index entry and a separate set of chunks in memory.

The Promtail configuration was:

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        target_label: 'container'
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: 'service'
      # WRONG: High-cardinality labels
      - source_labels: ['__meta_docker_container_label_request_id']
        target_label: 'request_id'
      - source_labels: ['__meta_docker_container_label_user_id']
        target_label: 'user_id'

The request_id and user_id labels should never have been added as stream labels. These are per-log-line values that belong in the log content, not in the label index.

Solution

I restructured the Promtail configuration to use only low-cardinality labels:

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      # Keep only low-cardinality labels
      - source_labels: ['__meta_docker_container_name']
        target_label: 'container'
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: 'service'
      - source_labels: ['__meta_docker_container_label_com_docker_compose_project']
        target_label: 'project'
      # Environment is typically low cardinality
      - source_labels: ['__meta_docker_container_label_environment']
        target_label: 'env'
    pipeline_stages:
      # Parse JSON logs to extract fields for searching
      - json:
          expressions:
            request_id: request_id
            user_id: user_id
            level: level
            method: method
            path: path
            status_code: status_code
            latency_ms: latency_ms
      # Add extracted fields as structured metadata (not labels)
      - structured_metadata:
          request_id: request_id
          user_id: user_id
      # Drop the raw labels we no longer need
      - labels:
          level:
          method:

Key changes:

1. Removed high-cardinality labels — request_id and user_id are no longer stream labels
2. Added structured metadata — Loki 2.9+ supports structured metadata, which allows indexing without the overhead of stream labels
3. Parsed level and method as labels — These are low cardinality (about 5 and 8 values respectively)

I also configured the Loki storage and retention properly:

# loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /loki/chunks

limits_config:
  max_query_length: 721h
  retention_period: 720h  # 30 days
  max_entries_limit_per_query: 10000
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
  per_stream_rate_limit: 5MB
  per_stream_rate_limit_burst: 15MB

compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

After redeploying with the new configuration, the results were dramatic:

# Before optimization
curl -s http://localhost:3100/metrics | grep "loki_ingester_memory_chunks"
# loki_ingester_memory_chunks 12847523

# After optimization (24 hours later)
curl -s http://localhost:3100/metrics | grep "loki_ingester_memory_chunks"
# loki_ingester_memory_chunks 342156

Memory usage dropped from 4GB to under 512MB, and query performance improved from 30+ second timeouts to sub-second responses.

For queries that need to search by request_id or user_id, I use LogQL line filters or JSON parsing:

# Search by request_id using line filter
{service="api-server"} |= "req-abc123"

# Search by user_id using JSON parsing
{service="api-server"} | json | user_id="user-456"

# Combine with structured metadata
{service="api-server"} | request_id="req-abc123"

Lessons Learned

1. Low cardinality is everything in Loki — Only use labels for fields with a limited number of unique values (services, environments, log levels). Never use request IDs, user IDs, or timestamps as labels.
2. Use structured metadata for per-log fields — Loki 2.9+ supports structured metadata that allows indexing without the stream label overhead.
3. Monitor label cardinality — Use logcli labels <label> to check the number of unique values. If it exceeds a few thousand, it is too high for a label.
4. Parse at query time, not at ingestion — Use LogQL json and line_format to extract fields from log content at query time instead of creating labels at ingestion time.
5. Set retention policies early — Configure retention_period to prevent unbounded storage growth. Loki's compactor handles cleanup automatically.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.