MongoDB Replica Set Configuration Error: Election and Heartbeat Failures

Introduction

A MongoDB replica set provides automatic failover and data redundancy by maintaining multiple copies of your data across different servers. I set up a three-member replica set for my trading platform's MongoDB deployment, and it worked perfectly until the primary node restarted for a scheduled OS update. The expectation was that one of the secondaries would automatically become the new primary. Instead, the entire cluster became read-only for about 90 seconds while MongoDB struggled to hold an election.

The cause was a combination of heartbeat configuration issues and member priority settings that I had overlooked. Here is how I diagnosed and fixed the replica set.

Environment

• MongoDB: 7.0.4
• Replica set: 3 members (1 primary, 2 secondaries)
• Nodes: 3 separate Ubuntu 22.04 servers
• Network: Private network, 10ms latency between nodes
• Priority: All members set to default priority (1)

Problem

When I performed a rolling restart of the primary node:

# On the primary node
sudo systemctl restart mongod

The replica set logs showed:

[rsSync] transition to PRIMARY
[rsHealthMonitor] member 10.0.0.1:27017 is now in state DOWN
[rsElect] Running election for replica set trading-rs
[rsElect] Candidate received vote from 10.0.0.2:27017
[rsElect] Not enough votes to proceed with election. Got 1, need 2
[rsElect] Failed election: not enough votes for victory

The election required 2 votes (majority of 3 members), but only 1 vote was received. The third member (10.0.0.3) was not participating in the election because its heartbeat connection to the other members was timing out.

I checked the heartbeat status on the third member:

mongo --host 10.0.0.3 --eval "rs.status().members.map(m => ({name: m.name, stateStr: m.stateStr, lastHeartbeat: m.lastHeartbeat, lastHeartbeatRecv: m.lastHeartbeatRecv}))"

[
    { "name": "10.0.0.1:27017", "stateStr": "DOWN", "lastHeartbeat": "never", "lastHeartbeatRecv": "2024-06-01T10:00:00Z" },
    { "name": "10.0.0.2:27017", "stateStr": "PRIMARY", "lastHeartbeat": "2024-06-01T10:00:05Z", "lastHeartbeatRecv": "2024-06-01T10:00:04Z" },
    { "name": "10.0.0.3:27017", "stateStr": "SECONDARY", "lastHeartbeat": "never", "lastHeartbeatRecv": "never" }
]

The third member had never received a heartbeat from the other members. This meant it was isolated from the replica set communication, and it could not participate in elections.

Analysis

I checked the MongoDB configuration on the third member:

# /etc/mongod.conf (member 3)
net:
  port: 27017
  bindIp: 127.0.0.1,10.0.0.3

replication:
  replSetName: trading-rs
  heartbeatTimeoutSecs: 10

The bindIp included both 127.0.0.1 and 10.0.0.3, which was correct. But I checked the network connectivity from the third member:

# From member 3
nc -zv 10.0.0.1 27017
# Connection to 10.0.0.1 27017 port [tcp/mongodb] succeeded!

nc -zv 10.0.0.2 27017
# Connection to 10.0.0.2 27017 port [tcp/mongodb] succeeded!

Network connectivity was fine. The issue was in the MongoDB configuration. The third member's bindIp did not include the wildcard 0.0.0.0 or all necessary interfaces. More importantly, the MongoDB replica set uses a separate port range for inter-member communication that was being blocked by the firewall.

I checked the firewall:

sudo ufw status | grep 27017
# 27017/tcp    ALLOW       10.0.0.0/24

Port 27017 was open, but MongoDB replica sets use an additional internal port for the replica set protocol. The heartbeat traffic uses a different port range that was not configured.

Actually, the real issue was simpler. The third member's MongoDB configuration had a replica set name mismatch:

# Member 1
replication:
  replSetName: trading-rs

# Member 2
replication:
  replSetName: trading-rs

# Member 3 (INCORRECT)
replication:
  replSetName: trading_RS  # Underscore instead of hyphen!

The replica set name on member 3 was trading_RS instead of trading-rs. MongoDB was running but could not communicate with the replica set because the names did not match. The member was accepting connections but not participating in replica set operations.

Solution

I fixed the configuration on member 3:

# /etc/mongod.conf (member 3 - fixed)
net:
  port: 27017
  bindIp: 127.0.0.1,10.0.0.3

replication:
  replSetName: trading-rs  # Fixed: hyphen instead of underscore
  heartbeatTimeoutSecs: 15
  electionTimeoutMillis: 10000

After restarting MongoDB on member 3:

sudo systemctl restart mongod

I verified the replica set status:

mongo --host 10.0.0.1 --eval "rs.status().members.map(m => ({name: m.name, stateStr: m.stateStr, health: m.health}))"

[
    { "name": "10.0.0.1:27017", "stateStr": "PRIMARY", "health": 1 },
    { "name": "10.0.0.2:27017", "stateStr": "SECONDARY", "health": 1 },
    { "name": "10.0.0.3:27017", "stateStr": "SECONDARY", "health": 1 }
]

All three members were healthy and communicating. I then re-tested the failover by restarting the primary:

# On the primary node
sudo systemctl restart mongod

[rsSync] transition to SECONDARY
[rsElect] Running election for replica set trading-rs
[rsElect] Candidate received vote from 10.0.0.3:27017
[rsElect] Won election with 2 votes
[rsSync] transition to PRIMARY

Failover completed in about 3 seconds instead of 90 seconds.

I also optimized the replica set configuration for faster failover:

// On the primary, update replica set configuration
rs.reconfig({
    _id: "trading-rs",
    members: [
        {
            _id: 0,
            host: "10.0.0.1:27017",
            priority: 10,  // Preferred primary
            votes: 1
        },
        {
            _id: 1,
            host: "10.0.0.2:27017",
            priority: 5,   // Can become primary
            votes: 1
        },
        {
            _id: 2,
            host: "10.0.0.3:27017",
            priority: 1,   // Less preferred primary
            votes: 1
        }
    ],
    settings: {
        heartbeatTimeoutSecs: 15,
        electionTimeoutMillis: 5000,
        chainingAllowed: false
    }
}, { force: true })

Key configuration changes:

• priority: 10 on member 1 makes it the preferred primary (it will win elections)
• electionTimeoutMillis: 5000 reduces the time before a secondary initiates an election
• chainingAllowed: false prevents secondaries from syncing from other secondaries (reduces sync lag)

After deploying the optimized configuration:

# Test failover time
time mongo --host 10.0.0.2 --eval "rs.status().myState"
# Primary elected in 2.8 seconds

Lessons Learned

1. Verify replica set names match — A subtle typo in the replica set name can cause a member to be silently excluded from the set. Always double-check configuration across all members.
2. Set appropriate priorities — Use priority to control which member becomes primary during failover. Higher priority members win elections.
3. Reduce election timeout — The default electionTimeoutMillis is 10 seconds. Reducing it to 5 seconds speeds up failover detection.
4. Monitor heartbeat status — Use rs.status() to verify that all members are receiving heartbeats. Missing heartbeats indicate connectivity or configuration issues.
5. Test failover regularly — Perform planned failover tests to verify your replica set configuration works correctly under real conditions.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.