Redis Sentinel Configuration Error: Failover Not Triggering on Master Failure

Introduction

Redis Sentinel provides high availability for Redis by monitoring master and replica instances and automatically promoting a replica to master when the current master fails. I set up a three-node Sentinel cluster for a trading system I am building, and everything looked correct on paper. Three Sentinel instances, one master, two replicas — a textbook configuration.

But when I simulated a master failure by killing the Redis process, nothing happened. The Sentinel instances detected the failure, logged the subjective and objective down states, but the actual failover never executed. After digging into the Sentinel logs and configuration files, I found three separate issues that were preventing the failover from completing.

Environment

Redis version: 7.2.3
Sentinel version: 7.2.3 (same binary)
Nodes: 3x Ubuntu 22.04 servers (10.0.0.11, 10.0.0.12, 10.0.0.13)
Master: 10.0.0.11:6379
Replicas: 10.0.0.12:6379, 10.0.0.13:6379
Sentinels: One on each node, ports 26379

Problem

I triggered a failover test by stopping the Redis master process:

# On 10.0.0.11
sudo systemctl stop redis-server

# On 10.0.0.12, watching Sentinel logs
sudo tail -f /var/log/redis/sentinel.log

The Sentinel logs showed detection of the failure:

[12345] 20 Jan 14:30:01.123 # +sdown master mymaster 10.0.0.11 6379
[12345] 20 Jan 14:30:01.456 # +odown master mymaster 10.0.0.11 6379 #quorum 3/2

The +sdown means the master was subjectively down (one Sentinel detected it). The +odown means it was objectively down (enough Sentinels agreed). But after the odown, there was no +failover-detected or +switch-master message. The failover simply did not start.

I waited the full 30-second failover-timeout period, then checked:

redis-cli -p 26379 sentinel get-master-addr-by-name mymaster
# 1) "10.0.0.11"
# 2) "6379"

The master was still reported as the original master, even though it was down. No failover had occurred.

Analysis

I started by examining the Sentinel configuration file /etc/redis/sentinel.conf:

sentinel monitor mymaster 10.0.0.11 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 30000
sentinel parallel-syncs mymaster 1

The quorum was set to 2, meaning at least 2 of 3 Sentinels needed to agree that the master was down before initiating a failover. The +odown log confirmed that 3 Sentinels agreed (3/2), so quorum was met. The problem was elsewhere.

I checked the Sentinel state more carefully:

redis-cli -p 26379 sentinel master mymaster
# 1) "name"
# 2) "mymaster"
# 3) "ip"
# 4) "10.0.0.11"
# ...
# 21) "num-other-sentinels"
# 22) "2"
# 23) "flags"
# 24) "s_down,o_down"

The flags showed s_down,o_down — the master was correctly detected as down. But the failover was not starting. I checked the other Sentinel instances:

redis-cli -h 10.0.0.12 -p 26379 sentinel master mymaster | grep -A1 "flags"
# 23) "flags"
# 24) "s_down,o_down"

redis-cli -h 10.0.0.13 -p 26379 sentinel master mymaster | grep -A1 "flags"
# 23) "flags"
# 24) "s_down"

The third Sentinel (10.0.0.13) had only s_down — it had not reached o_down. This meant only 2 Sentinels agreed on the objective down state. With a quorum of 2, this should have been enough.

But I also noticed something else — the third Sentinel was reporting the master as only subjectively down, not objectively down. The quorum check requires the down-after-milliseconds timeout to have elapsed for each Sentinel independently. I had checked too early.

However, after waiting the full 5000ms timeout, the third Sentinel still showed s_down only. This pointed to a network issue between the Sentinels.

I tested network connectivity:

# From 10.0.0.12 to 10.0.0.13
redis-cli -h 10.0.0.13 -p 26379 ping
# PONG

# From 10.0.0.13 to 10.0.0.11
redis-cli -h 10.0.0.11 -p 6379 ping
# Could not connect to Redis at 10.0.0.11:6379: Connection refused

Wait — the third Sentinel could connect to the other Sentinels but the master was down on all nodes. That was expected since I had killed the master. But the third Sentinel was not detecting the master failure through its own Sentinel peers.

The issue was that the third Sentinel was connecting to the master via a different route. I checked its config:

redis-cli -h 10.0.0.13 -p 26379 CONFIG GET bind
# 1) "bind"
# 2) "127.0.0.1 ::1"

The third Sentinel was bound to localhost only! It could not be reached by the other Sentinels for inter-Sentinel communication. The Sentinels communicate through the Redis pub/sub channel on the master, and when the master is down, they need to communicate directly. Since the third Sentinel was only listening on localhost, the other two could not reach it.

This explained why only 2 out of 3 Sentinels had o_down — the third one was isolated from the communication.

Solution

I updated the Sentinel configuration on all three nodes to bind to the correct network interfaces:

# /etc/redis/sentinel.conf (node 10.0.0.11)
bind 0.0.0.0
sentinel monitor mymaster 10.0.0.11 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel resolve-hostnames yes
sentinel announce-hostnames yes

# /etc/redis/sentinel.conf (node 10.0.0.12)
bind 0.0.0.0
sentinel monitor mymaster 10.0.0.11 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel resolve-hostnames yes
sentinel announce-hostnames yes

# /etc/redis/sentinel.conf (node 10.0.0.13)
bind 0.0.0.0
sentinel monitor mymaster 10.0.0.11 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel resolve-hostnames yes
sentinel announce-hostnames yes

After restarting all Sentinel instances:

sudo systemctl restart redis-sentinel

I re-ran the failover test:

# On 10.0.0.11 — stop Redis master
sudo systemctl stop redis-server

# On 10.0.0.12 — watch Sentinel logs
sudo tail -f /var/log/redis/sentinel.log

This time, the failover succeeded:

[12345] 20 Jan 15:00:01.123 # +sdown master mymaster 10.0.0.11 6379
[12345] 20 Jan 15:00:01.456 # +odown master mymaster 10.0.0.11 6379 #quorum 3/3
[12345] 20 Jan 15:00:01.567 # +failover-detected master mymaster 10.0.0.11 6379
[12345] 20 Jan 15:00:01.678 # +failover-state-select-slave master mymaster 10.0.0.11 6379
[12345] 20 Jan 15:00:02.012 # +selected-slave replica 10.0.0.12:6379 10.0.0.12 6379
[12345] 20 Jan 15:00:02.234 # +failover-state-send-slaveof-noone
[12345] 20 Jan 15:00:03.456 # +switch-master mymaster 10.0.0.12 6379

The quorum now showed 3/3 — all three Sentinels agreed on the down state. The failover selected 10.0.0.12 as the new master, and the switch completed successfully.

I verified the new master:

redis-cli -h 10.0.0.12 -p 6379 role
# master

redis-cli -h 10.0.0.13 -p 6379 role
# slave
# master_host:10.0.0.12
# master_port:6379

Lessons Learned

Check bind configuration — Sentinel instances must be reachable by other Sentinels for inter-node communication. Binding to 127.0.0.1 isolates the instance.
Verify quorum with all nodes — Use redis-cli -p 26379 sentinel master mymaster on each Sentinel to check if it has reached odown status.
Inter-Sentinel communication uses the master's pub/sub — When the master fails, Sentinels communicate directly via TCP. All Sentinel ports (26379) must be accessible between nodes.
Test failover regularly — Simulate failures periodically to ensure your Sentinel configuration actually works. A configuration that has never been tested is not a configuration you can rely on.
Use resolve-hostnames — For environments with dynamic IPs, enable sentinel resolve-hostnames yes so Sentinel can track nodes by hostname instead of IP.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.