tooolbox/blog/MinIO S3-Compatible Storage Setup: Distributed Mode and Bucket Lifecycle
devops2024-12-01Β·8Β·246/348

MinIO S3-Compatible Storage Setup: Distributed Mode and Bucket Lifecycle

Setting up MinIO in distributed mode with erasure coding, bucket lifecycle policies, and integration with S3-compatible applications.

Introduction

MinIO is an S3-compatible object storage server that you can self-host. I switched to MinIO from storing files on local disk because I needed a reliable, scalable storage backend for training data used by my quantitative trading models. The training datasets are large (hundreds of gigabytes), and I needed something that could handle concurrent reads from multiple training jobs without corruption.

Setting up MinIO in single-node mode is trivial β€” download the binary, run it, and you have an S3-compatible server. But distributed mode, which provides fault tolerance and high availability, requires careful planning. This post covers my experience setting up a four-node MinIO cluster with erasure coding and the issues I encountered along the way.

Environment

  • MinIO version: RELEASE.2024-01-18T22-51-28Z
  • Nodes: 4x Hetzner Cloud CX32 (8 vCPUs, 32GB RAM, 640GB NVMe each)
  • Operating system: Ubuntu 22.04
  • Network: Private network 10.0.0.0/24
  • Storage: 2x 320GB NVMe per node (8 drives total)
  • Use case: Training data storage for ML models

Problem

I initially tried setting up MinIO distributed mode with the following command:

minio server http://10.0.0.{1...4}/data{1...2}

This started successfully, but when I tried to create a bucket and upload a file, I got this error:

mc mb myminio/training-data
# Bucket created successfully

mc cp large-file.parquet myminio/training-data/
# mc:  Put operation can not proceed. Writing to one or more data drives failed.

The error message indicated that MinIO could not write to the required number of data drives. In distributed mode with erasure coding, MinIO needs a minimum number of healthy drives to write data. With 8 total drives (2 per node), the erasure coding scheme requires at least 6 drives to be writable.

I checked the MinIO server output:

 API: http://10.0.0.1:9000
  RootUser: minioadmin
  RootPass: minioadmin

Drive Capacity: 298 GiB Free, 320 GiB Total
Drives:
  10.0.0.1/data1    online
  10.0.0.1/data2    online
  10.0.0.2/data1    offline
  10.0.0.2/data2    offline
  10.0.0.3/data1    online
  10.0.0.3/data2    online
  10.0.0.4/data1    online
  10.0.0.4/data2    online

Two drives on node 10.0.0.2 were showing as offline. With only 6 out of 8 drives online, MinIO was on the minimum threshold for erasure coding, and write operations were failing intermittently.

Analysis

I investigated why the drives on node 10.0.0.2 were offline. The first thing I checked was network connectivity:

# From node 10.0.0.1
ping 10.0.0.2
# PING 10.0.0.2: 64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=0.5ms

# Check MinIO port
curl -s http://10.0.0.2:9000/minio/health/cluster
# {"status": "unavailable"}

The cluster health check returned "unavailable". MinIO uses port 9000 for API traffic and port 9001 for the console. It also uses dynamic ports for inter-node communication (MinIO uses its own RPC protocol).

I checked the firewall rules on node 10.0.0.2:

sudo ufw status
# Status: active
#
# To                         Action      From
# --                         ------      ----
# 22/tcp                     ALLOW       Anywhere
# 9000/tcp                   ALLOW       Anywhere
# 9001/tcp                   ALLOW       Anywhere

The firewall was missing the inter-node communication ports. MinIO uses port range 9000 for API and additional ports for internal communication. I needed to open the full range.

But there was a second issue. The drives on node 10.0.0.2 were formatted with ext4, and the mount options were missing the noatime flag, which MinIO recommends for performance. More importantly, the drives were not mounted at /data1 and /data2 as MinIO expected:

df -h | grep data
# /dev/nvme1n1  320G   60G  244G  20% /mnt/disk1
# /dev/nvme2n1  320G   60G  244G  20% /mnt/disk2

The drives were mounted at /mnt/disk1 and /mnt/disk2, not /data1 and /data2. The MinIO command expected the paths /data1 and /data2 to exist on each node.

Solution

I fixed three issues: firewall configuration, drive mounting, and MinIO startup:

1. Configure firewall on all nodes:

#!/bin/bash
# setup-firewall.sh β€” Run on each node

sudo ufw allow from 10.0.0.0/24 to any port 9000 proto tcp
sudo ufw allow from 10.0.0.0/24 to any port 9001 proto tcp
sudo ufw allow from 10.0.0.0/24 to any proto tcp  # MinIO inter-node RPC
sudo ufw allow from 10.0.0.0/24 to any proto udp  # MinIO inter-node discovery
sudo ufw reload

2. Set up drives with proper mounting:

#!/bin/bash
# setup-drives.sh β€” Run on each node

# Format drives (only if not already formatted)
sudo mkfs.ext4 -L data1 /dev/nvme1n1
sudo mkfs.ext4 -L data2 /dev/nvme2n1

# Create mount points
sudo mkdir -p /data1 /data2

# Mount with optimized options
sudo mount -o noatime,nodiratime /dev/nvme1n1 /data1
sudo mount -o noatime,nodiratime /dev/nvme2n1 /data2

# Add to fstab for persistence
echo "LABEL=data1 /data1 ext4 defaults,noatime,nodiratime 0 2" | sudo tee -a /etc/fstab
echo "LABEL=data2 /data2 ext4 defaults,noatime,nodiratime 0 2" | sudo tee -a /etc/fstab

# Set ownership for MinIO
sudo useradd -r minio-user -s /sbin/nologin
sudo chown -R minio-user:minio-user /data1 /data2

3. Create systemd service for MinIO:

# /etc/systemd/system/minio.service
[Unit]
Description=MinIO
Documentation=https://docs.min.io
After=network-online.target
Wants=network-online.target

[Service]
User=minio-user
Group=minio-user
Environment="MINIO_ROOT_USER=minioadmin"
Environment="MINIO_ROOT_PASSWORD=minioadmin"
ExecStart=/usr/local/bin/minio server http://10.0.0.{1...4}/data{1...2}
Restart=always
RestartSec=10
LimitNOFILE=65536
TasksMax=infinity
TimeoutStopSec=infinity
SendSIGkill=no

[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now minio

After starting MinIO on all four nodes:

# Check cluster health
curl -s http://10.0.0.1:9000/minio/health/cluster
# {"status": "healthy"}

# Check drive status
mc admin info myminio
# ...
# Drives:
#   10.0.0.1/data1    online  298 GiB
#   10.0.0.1/data2    online  298 GiB
#   10.0.0.2/data1    online  298 GiB
#   10.0.0.2/data2    online  298 GiB
#   10.0.0.3/data1    online  298 GiB
#   10.0.0.3/data2    online  298 GiB
#   10.0.0.4/data1    online  298 GiB
#   10.0.0.4/data2    online  298 GiB

All 8 drives were online. I verified write operations:

mc cp large-file.parquet myminio/training-data/
# large-file.parquet: 2.3 GiB / 2.3 GiB

I also set up bucket lifecycle policies for automatic cleanup of old training data:

{
  "Rules": [
    {
      "ID": "expire-old-datasets",
      "Status": "Enabled",
      "Expiration": {
        "Days": 90
      },
      "Filter": {
        "Prefix": "datasets/"
      }
    },
    {
      "ID": "transition-to-archive",
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "GLACIER"
        }
      ],
      "Filter": {
        "Prefix": "models/"
      }
    }
  ]
}
mc ilm import myminio/training-data < lifecycle-policy.json

Lessons Learned

  1. Open all required ports β€” MinIO uses port 9000 for API and additional ports for inter-node RPC. Check the MinIO documentation for the complete list of required ports.
  2. Match mount paths exactly β€” MinIO expects drives at the paths you specify in the server command. If drives are mounted elsewhere, create symlinks or update mount points.
  3. Use noatime for storage performance β€” This eliminates unnecessary metadata updates on every file read, significantly improving throughput.
  4. Plan for erasure coding β€” MinIO with erasure coding needs at least 4 drives (ideally 8+). Each drive reduces available storage but increases redundancy.
  5. Set lifecycle policies early β€” Without lifecycle rules, object storage grows unbounded. Set up expiration and transition policies from the start.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.