Redis Sentinel Failover Problems: Split-Brain Prevention and Recovery

Introduction

Redis Sentinel failover should be automatic and reliable, but in practice, several edge cases can cause problems. I experienced a split-brain scenario where two different nodes were promoted to master simultaneously after a network partition between my Sentinel nodes. This caused data inconsistency — writes to one master were not replicated to the other, and when the partition healed, one master was demoted and its unreplicated writes were lost.

This was a critical issue for my trading system because lost writes meant lost trade signals. Here is how I diagnosed the split-brain, recovered from it, and configured Sentinel to prevent it in the future.

Environment

• Redis: 7.2.3
• Sentinel: 7.2.3
• Nodes: 5 Sentinel instances across 3 data centers
• Replica set: 1 master, 2 replicas
• Quorum: 3 (majority of 5 Sentinels)

Problem

A network partition occurred between data center A (Sentinels 1, 2) and data centers B and C (Sentinels 3, 4, 5). During the partition:

1. Sentinels 3, 4, 5 detected the master as down and elected a new master from one of the replicas
2. Sentinels 1, 2 could not reach the majority, so they should have stopped monitoring
3. But the application connected to both masters during the partition

The result was two masters accepting writes simultaneously:

# On Sentinel 1 (data center A)
redis-cli -p 26379 sentinel get-master-addr-by-name mymaster
# 1) "10.0.0.1"    (original master)

# On Sentinel 3 (data center B)
redis-cli -p 26379 sentinel get-master-addr-by-name mymaster
# 1) "10.0.0.2"    (newly promoted replica)

When the network partition healed, Sentinel 1 and 2 detected that the master had changed and demoted the original master to a replica. But the original master had accepted writes during the partition that were not replicated to the new master. Those writes were lost.

Analysis

I examined the Sentinel logs to understand the timeline:

# Data center A (Sentinels 1, 2)
[2024-05-01 10:00:00] +sdown master mymaster 10.0.0.1 6379
[2024-05-01 10:00:01] +odown master mymaster 10.0.0.1 6379 #quorum 2/5
[2024-05-01 10:00:01] # Failover state: 2 Sentinels agree, need 3
[2024-05-01 10:00:01] # Cannot start failover — quorum not reached

# Data center B, C (Sentinels 3, 4, 5)
[2024-05-01 10:00:00] +sdown master mymaster 10.0.0.1 6379
[2024-05-01 10:00:01] +odown master mymaster 10.0.0.1 6379 #quorum 3/5
[2024-05-01 10:00:01] +failover-detected master mymaster 10.0.0.1 6379
[2024-05-01 10:00:02] +switch-master mymaster 10.0.0.2 6379

The quorum of 3 was reached by Sentinels 3, 4, 5 (in data centers B and C), and they promoted replica 10.0.0.2 to master. But Sentinels 1, 2 (in data center A) could not form a quorum because they were isolated.

The problem was that the application was not using Sentinel's master discovery correctly. Instead of querying Sentinel for the current master, the application had the master address hardcoded:

# BUG: Hardcoded master address
import redis
r = redis.Redis(host="10.0.0.1", port=6379)  # Original master

When the original master was demoted to replica, it became read-only, and writes started failing. But the application did not detect this because it was not using Sentinel's automatic failover support.

Additionally, I found that min-slaves-to-write was not configured on the original master. This setting prevents a master from accepting writes if it does not have enough replicas connected. Without it, the master continued accepting writes even after being isolated from its replicas.

Solution

I implemented a multi-layer fix:

1. Configure the application to use Sentinel for master discovery:

from redis.sentinel import Sentinel

sentinel = Sentinel(
    [
        ('10.0.0.1', 26379),
        ('10.0.0.2', 26379),
        ('10.0.0.3', 26379),
        ('10.0.0.4', 26379),
        ('10.0.0.5', 26379),
    ],
    socket_timeout=0.5
)

# Get the current master dynamically
master = sentinel.master_for('mymaster', socket_timeout=0.5)
replica = sentinel.replica_for('mymaster', socket_timeout=0.5)

# Write operations go to master
master.set('key', 'value')

# Read operations go to a replica (reduces master load)
value = replica.get('key')

This ensures the application always connects to the current master, even after a failover.

2. Configure min-slaves-to-write on Redis:

# /etc/redis/redis.conf
min-replicas-to-write 1
min-replicas-max-lag 10

This tells the master: "Do not accept writes if you have fewer than 1 replica connected, and that replica is not more than 10 seconds behind in replication."

When the master is isolated from its replicas, it stops accepting writes instead of continuing alone. This prevents the split-brain write loss.

3. Configure Sentinel for proper split-brain prevention:

# /etc/redis/sentinel.conf
sentinel monitor mymaster 10.0.0.1 6379 3
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 30000
sentinel parallel-syncs mymaster 1

# Require authorization for inter-sentinel communication
sentinel auth-pass mymaster 
sentinel down-after-milliseconds mymaster 5000

4. Add Sentinel connection retry logic in the application:

class RedisConnection:
    def __init__(self):
        self.sentinel = Sentinel(
            [('10.0.0.1', 26379), ('10.0.0.2', 26379), ('10.0.0.3', 26379)],
            socket_timeout=0.5,
            retry_on_timeout=True
        )
        self._master = None
        self._master_fail_time = 0

    def get_master(self):
        try:
            if self._master is None or time.time() - self._master_fail_time > 5:
                self._master = self.sentinel.master_for(
                    'mymaster',
                    socket_timeout=0.5,
                    retry_on_timeout=True
                )
            return self._master
        except Exception as e:
            self._master = None
            self._master_fail_time = time.time()
            raise

    def execute(self, command, *args, **kwargs):
        retries = 3
        for i in range(retries):
            try:
                master = self.get_master()
                return getattr(master, command)(*args, **kwargs)
            except redis.exceptions.ConnectionError:
                self._master = None
                time.sleep(0.1 * (i + 1))
        raise Exception("Failed to connect to Redis after 3 retries")

5. Create a post-failover consistency check:

def check_data_consistency():
    """After failover, verify data consistency between old and new master."""
    master_info = sentinel.discover_master('mymaster')
    print(f"Current master: {master_info}")

    # Check replication lag on the new master's replicas
    for replica_addr in sentinel.discover_replicas('mymaster'):
        replica = sentinel.replica_for('mymaster')
        info = replica.info('replication')
        lag = info.get('master_repl_offset', 0) - info.get('slave_repl_offset', 0)
        print(f"Replica {replica_addr}: lag = {lag} bytes")

After implementing all fixes, I simulated another network partition:

# Block network between data centers
sudo iptables -A INPUT -s 10.0.0.1 -j DROP
sudo iptables -A INPUT -s 10.0.0.2 -j DROP

# Wait 10 seconds for detection
sleep 10

# Check master status on both sides
redis-cli -h 10.0.0.3 -p 26379 sentinel get-master-addr-by-name mymaster
# 1) "10.0.0.3"  (new master in data center B)

redis-cli -h 10.0.0.1 -p 26379 ping
# Could not connect (master stopped accepting writes due to min-slaves-to-write)

# Remove the network block
sudo iptables -D INPUT -s 10.0.0.1 -j DROP
sudo iptables -D INPUT -s 10.0.0.2 -j DROP

# Wait for recovery
sleep 15

# Verify single master
redis-cli -h 10.0.0.1 -p 26379 sentinel get-master-addr-by-name mymaster
# 1) "10.0.0.3"  (single master, no split-brain)

The original master correctly refused writes when isolated, and after the partition healed, only one master existed. No data loss occurred.

Lessons Learned

1. Never hardcode Redis master addresses — Always use Sentinel's master discovery to get the current master address dynamically.
2. Configure min-slaves-to-write — This prevents a master from accepting writes when it is isolated from its replicas, preventing split-brain data loss.
3. Use a quorum of majority — Set the Sentinel quorum to a majority of Sentinels (3 out of 5, not 2 out of 3) to prevent false failovers during network partitions.
4. Add retry logic — Redis connections can fail during failover. Implement retry logic with exponential backoff in the application.
5. Test failover scenarios regularly — Simulate network partitions, node crashes, and split-brain scenarios to verify your configuration works correctly.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.