Elasticsearch Indexing Performance Optimization: Bulk API Tuning and Shard Strategy

Introduction

Elasticsearch is powerful, but it can be deceptively slow if you do not configure it properly for your workload. I discovered this firsthand when I was building a logging pipeline that needed to ingest approximately 50,000 documents per second. The initial setup was using default Elasticsearch settings, and indexing throughput was barely reaching 5,000 documents per second — a tenfold shortfall from what I needed.

After profiling the indexing pipeline and experimenting with various Elasticsearch configurations, I was able to achieve consistent throughput of over 60,000 documents per second on modest hardware. The improvements came from several areas: bulk API tuning, shard strategy, refresh interval optimization, and mapping adjustments. This post documents every change I made and the impact each one had.

Environment

• Elasticsearch version: 8.11.3
• Cluster: 3 nodes, each with 16GB RAM, 8 vCPUs, 500GB NVMe SSD
• Heap size: 8GB per node (50% of RAM)
• Index type: Time-based log data (JSON documents, ~2KB average)
• Ingestion method: Filebeat → Logstash → Elasticsearch
• Target throughput: 50,000 docs/sec sustained

Problem

The initial indexing configuration was:

PUT /logs-2025.01
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  }
}

The Bulk API was being used, but with default settings:

from elasticsearch import Elasticsearch

es = Elasticsearch(["http://localhost:9200"])

# Default bulk settings
def index_documents(documents):
    for doc in documents:
        es.index(index="logs-2025.01", document=doc)

Indexing 10,000 documents took approximately 45 seconds, which translates to roughly 222 documents per second. Even when I switched to bulk indexing, the throughput was only about 5,000 documents per second.

The Elasticsearch cluster was showing high indexing latency in the monitoring dashboard:

curl -s "localhost:9200/_nodes/stats/indices/indexing" | python3 -m json.tool

{
  "nodes": {
    "abc123": {
      "indices": {
        "indexing": {
          "index_total": 450000,
          "index_time_in_millis": 234000,
          "index_current": 12
        }
      }
    }
  }
}

The index_time_in_millis divided by index_total showed an average of 0.52ms per document — far too high for simple JSON documents on NVMe storage.

Analysis

I profiled the ingestion pipeline and identified several bottlenecks:

1. Individual index requests instead of bulk API:

The initial code was using es.index() for each document. This creates a separate HTTP request for every single document, which is extremely inefficient. Each request involves HTTP overhead, JSON parsing, and a separate index operation.

2. Default refresh interval:

Elasticsearch refreshes indices every 1 second by default, making new documents searchable. This refresh operation is expensive because it creates a new Lucene segment. During high-throughput indexing, this 1-second refresh cycle was causing significant overhead.

3. Excessive replicas during indexing:

The index had 1 replica, meaning every indexed document had to be written to two nodes. This doubles the write load without any benefit during the initial indexing phase.

4. No bulk size tuning:

The Bulk API has default limits on batch size and concurrent requests. Without tuning these, the pipeline was not utilizing the available network and disk bandwidth.

5. Suboptimal mapping:

Elasticsearch was automatically detecting field types, which involves sampling documents. This dynamic mapping adds overhead during indexing.

Solution

I made changes across five areas to optimize indexing performance:

1. Switch to Bulk API with proper sizing:

from elasticsearch import Elasticsearch, helpers
import time

es = Elasticsearch(
    ["http://localhost:9200"],
    max_retries=3,
    retry_on_timeout=True,
    request_timeout=60
)

def bulk_index(documents, batch_size=5000):
    actions = [
        {"_index": "logs-2025.01", "_source": doc}
        for doc in documents
    ]

    success, errors = helpers.bulk(
        es,
        actions,
        chunk_size=batch_size,
        max_retries=3,
        initial_backoff=2,
        max_backoff=30,
        queue_size=8
    )

    return success, errors

# Index in batches of 50,000
all_docs = load_documents()
for i in range(0, len(all_docs), 50000):
    batch = all_docs[i:i+50000]
    success, errors = bulk_index(batch)
    print(f"Indexed {success} documents in batch {i//50000 + 1}")
    if errors:
        print(f"Errors: {errors}")

2. Optimize index settings for write-heavy workload:

PUT /logs-2025.01-optimized
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 0,
    "refresh_interval": "30s",
    "translog.durability": "async",
    "translog.sync_interval": "30s",
    "merge.scheduler.max_thread_count": 2,
    "index.translog.flush_threshold_size": "1g"
  }
}

Key changes:

• refresh_interval: 30 seconds instead of 1 second — reduces segment creation overhead
• number_of_replicas: 0 during indexing — no replica write overhead
• translog.durability: async — trades slight durability for write speed
• flush_threshold_size: 1GB — flushes less frequently

3. Create mapping explicitly to avoid dynamic mapping overhead:

PUT /logs-2025.01-optimized
{
  "mappings": {
    "properties": {
      "timestamp": { "type": "date" },
      "level": { "type": "keyword" },
      "service": { "type": "keyword" },
      "message": { "type": "text" },
      "request_id": { "type": "keyword" },
      "response_time_ms": { "type": "integer" },
      "status_code": { "type": "short" },
      "host": { "type": "keyword" }
    }
  }
}

4. Index lifecycle management for shard rotation:

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "30gb",
            "max_age": "1d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {},
          "set_priority": { "priority": 0 }
        }
      }
    }
  }
}

5. Monitor indexing throughput:

#!/bin/bash
# monitor-indexing.sh — Run during indexing to track throughput

while true; do
    stats=$(curl -s "localhost:9200/_nodes/stats/indices/indexing")
    total=$(echo "$stats" | python3 -c "
import json, sys
data = json.load(sys.stdin)
for node_id, node in data['nodes'].items():
    print(node['indices']['indexing']['index_total'])
    break
")

    echo "$(date): Total documents indexed: $total"
    sleep 10
done

After implementing all changes, I ran the indexing pipeline again:

# Result:
# Indexed 5,000,000 documents in 78 seconds
# Throughput: ~64,100 documents per second
# Average latency per document: 0.015ms

The throughput increased from 5,000 to over 64,000 documents per second — a 12.8x improvement. After indexing was complete, I updated the index settings for search performance:

PUT /logs-2025.01-optimized/_settings
{
  "number_of_replicas": 1,
  "refresh_interval": "1s",
  "translog.durability": "request",
  "translog.sync_interval": "5s"
}

Lessons Learned

1. Always use Bulk API for batch ingestion — Individual index requests are 10-100x slower than bulk operations. There is no excuse for not using bulk in production ingestion pipelines.
2. Reduce refresh interval during heavy indexing — Setting refresh_interval to 30s or higher dramatically reduces segment creation overhead. Reset it to 1s after indexing completes.
3. Disable replicas during initial indexing — Replicas add write overhead. Add them back after the bulk indexing phase is complete.
4. Use async translog for write performance — This trades a small amount of durability for significant write speed improvement. Acceptable for log data that can tolerate minor data loss.
5. Pre-define your mappings — Dynamic mapping adds overhead. Explicit mappings are faster and give you more control over how data is stored and searched.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.